Every consultant in enterprise AI has an opinion on whether you need fine-tuning or retrieval-augmented generation. Most of those opinions are wrong — not because the technical details are incorrect, but because the framing is. The question is not "which one?" It is "which job needs doing?" Fine-tuning and RAG solve different problems. When you understand what each one actually does, the debate collapses. What remains is a sizing exercise.
Two forces make this concrete for regulated Indian enterprises. On one side, regulators — Schedule M (2024 revised), ALCOA+ Attributable, RBI IT Outsourcing MD — effectively mandate retrieval. Every AI output that will survive a USFDA inspection or an RBI audit must be traceable to a named source document, ideally with a version number and a section reference. That is a retrieval problem, not a training problem. On the other side, house style, refusal patterns, domain vocabulary, and the model's instinct to stay in-scope are training problems. Prompt engineering can approximate them for a month or two. It cannot enforce them at scale across a team of fifty people over three years. Every OwnAI production deployment uses both: a domain fine-tune and a Qdrant-backed retrieval layer. Not as redundancy. As different instruments solving different jobs.
What each technique actually does
Fine-tuning (specifically LoRA/QLoRA, which is what ships in practice) modifies a small subset of the model's weight matrices — typically 1–5% of total parameters. The adapter is generated from your training dataset and frozen at the end of the run. After fine-tuning, the model behaves differently. It uses your vocabulary. It formats outputs the way your templates require. It refuses questions that are out-of-scope. It applies your citation style. None of this requires any document to be in memory at inference time. The knowledge is baked in.
The constraint is also the definition: fine-tuning is frozen. If you update your Standard Operating Procedure tomorrow, the fine-tuned model does not know. If RBI issues a new circular next week, the weights say nothing about it. Fine-tuning shapes how the model thinks and writes. It does not keep the model current.
Retrieval-Augmented Generation runs a vector search at inference time — in our stack, against Qdrant — pulls the top-k most relevant document chunks, injects them as context, and asks the model to answer citing those chunks. The documents themselves live outside the model. That means they can be updated, versioned, and audited independently. A deviation report that cites "Per SOP-OPS-2026-04-12 §4.3 [retrieved]" is attributable, legible, and contemporaneous in ALCOA+ terms. You can pull the exact document from the audit trail and show the inspector what the model read. You cannot do that with fine-tuned knowledge — the model knows it, but you cannot show the inspector which document it came from.
The constraint is symmetric: RAG does not change how the model behaves. A model that has never seen GMP vocabulary will retrieve your SOPs and respond with generic language that an inspector will flag. A model with no refusal training will retrieve a customer's loan file and happily summarise PII in the output. Retrieval provides the facts. Fine-tuning provides the judgement about what to do with them.
Five axes to choose between (or combine) them
In practice, the sizing decision comes down to five variables. Here is how they map to technique preference:
| Axis | Lean fine-tune-heavy when… | Lean RAG-heavy when… | Notes for regulated deployments |
|---|---|---|---|
| Data churn rate | Compliance ontology is stable for years (regulatory frameworks, domain vocabulary) | Source documents change daily or weekly (live SOPs, open CAPAs, current RBI circulars) | Most regulated environments have both: stable vocabulary + changing documents. You need both techniques. |
| Audit obligation | Internal copilot; no per-response citation required; outputs are not GxP records | Every response must cite a named source; ALCOA+ Attributable applies; USFDA or RBI may inspect the evidence trail | Schedule M (2024) ALCOA+ Attributable effectively mandates RAG for any output that enters a regulated record. There is no workaround. |
| Latency budget | Response time under 200 ms hard requirement; synchronous UI with no loading states | Latency of 500–800 ms per query is acceptable; asynchronous workflows are already in use | RAG adds 200–500 ms on the Mac Mini M4 Pro stack (Qdrant + LiteLLM gateway). Most batch workflows tolerate this. Real-time patient-facing interfaces may not. |
| Token budget | Inference hardware is constrained; every token costs throughput; context window pressure is a real operational concern | Context window is large enough; you need the model to reason over the retrieved document, not just echo it | RAG injects 1 K–5 K tokens of context per query. On a Mac Mini M4 Pro (Qwen 3 32B, ~11 tok/s, 1–2 concurrent users), that is the difference between 18 tok/s usable and a saturated context queue. Hardware sizing must account for this. |
| Data sensitivity | You have enough training data to absorb the reconstruction risk of putting knowledge into weights (thousands of high-quality examples minimum) | You have rich source documents but few or no labelled training pairs; the documents are too sensitive to embed permanently | Both are sensitive. RAG embeddings are derived from your data — a Qdrant store leak allows partial source reconstruction. Fine-tuned weights are noisier but theoretically inversion-attackable. Neither is "safer by default". DPDPA §8 reasonable-security safeguards apply to both. |
The pharma example
A mid-size Indian pharmaceutical manufacturer deploying an OwnAI quality assistant uses the combination this way.
The fine-tune is trained on the company's historical deviation reports, CAPA records, and quality-review minutes — curated down to roughly 800 high-quality examples in GMP format. After two epochs on an A100 80 GB (cloud, not on-premise), the adapter encodes: the company's deviation classification taxonomy (critical / major / minor), the standard six-section structure of their deviation reports, the refusal pattern that prevents the model from making therapeutic efficacy claims, and the citation style that the company's QA head has used for fifteen years. This adapter ships to the on-premise Mac Mini and stays there.
The RAG layer indexes this week's SOPs (version-controlled in the company's document management system), the current open batch records, and the live CAPA list. The Qdrant collections are re-indexed nightly via a Python ingestion job. Documents that change get re-embedded. Documents that expire are removed.
When a QA analyst asks about a packaging line deviation, the deployed model responds: "Per SOP-OPS-2026-04-12 §4.3 [retrieved], the deviation should be classified Major because the batch exceeded the specification limit by more than the process tolerance defined in §3.1 [retrieved]. Recommend opening CAPA-2026-0041 [retrieved, open]. Section 5 deviation-report template follows…"
The section heading, the classification logic, the refusal to speculate beyond the document, and the report structure all come from the fine-tune. The SOP number, the version date, the section reference, and the CAPA linkage all come from RAG. An inspector reviewing this output can pull SOP-OPS-2026-04-12 from the DMS and verify every factual claim. That is ALCOA+ Attributable satisfied mechanically, not aspirationally.
The NBFC example
An NBFC compliance team uses the same architecture for a different job. The fine-tune is trained on the company's internal credit-memo format, their RBI-MD vocabulary glossary (formally defined terms for "NPA classification", "income recognition", "provisioning norms"), and — critically — a refusal pattern that prevents the model from including customer PII in any summary output. This last item is not optional: under DPDPA 2023, a "summary" that contains an identifiable loan account number transmitted to any downstream system that the customer's DPO has not explicitly cleared is a potential data-processing violation.
The RAG layer runs over the customer's loan file (per-query retrieval, access-controlled by the LiteLLM gateway), current RBI circulars scraped daily from the RBI website into a Qdrant collection, and the company's internal credit policy documents. The RBI circular collection is the reason RAG is mandatory here: circulars arrive without warning, take immediate effect on provisioning norms, and the model must reflect them the day they land — not at the next training run.
The result: the compliance team's credit analysts get responses that follow house format, use correct regulatory vocabulary, never surface PII in summaries, and cite the exact RBI circular paragraph that governs the provisioning decision being made. The LiteLLM gateway logs every request and retrieval event to Langfuse. When RBI's IT audit team arrives, the institution can produce an end-to-end trail: query, retrieved documents, model response, user action taken.
When NOT to fine-tune
Fine-tuning has a floor condition: you need enough high-quality training examples to move the model meaningfully, and those examples are expensive to produce. Under roughly 500 well-curated examples, fine-tuning on a 32B-class model typically yields marginal improvement over a well-prompted base model, at non-trivial cloud GPU cost. If your dataset is thin, pure RAG with a carefully written system prompt is the correct choice. Test first.
Similarly: if the base model already handles your domain fluently — general-purpose code generation is the canonical example — fine-tuning adds noise, not signal. Qwen 3 32B writes production-quality Python and SQL without any adapter. Adding a fine-tune for "our coding style" on 200 examples is almost certainly a waste of a $36 training run. Spend the budget on better retrieval.
When NOT to RAG
Three situations favour pure fine-tune with no retrieval layer.
First: a genuinely closed-world domain where all relevant knowledge is fixed and bounded. A model that enforces a specific game's rules, or answers questions about a fixed exam syllabus that does not change, does not need retrieval. Everything it needs to know can be in the weights. Adding RAG introduces latency, embedding-store operational overhead, and a failure mode (retrieval miss) for no benefit.
Second: a hard latency requirement below ~200 ms end-to-end. Qdrant vector search on the Mac Mini stack adds roughly 200 ms at the p50 for a 1 M vector collection. If your application has a 150 ms SLA — say, a real-time suggestion in a clinical data-entry form — RAG cannot meet it on current on-premise hardware. Pure fine-tune inference is faster.
Third: no source documents exist. If the knowledge you need is implicit in how your organisation thinks and writes, not in any document, then there is nothing to retrieve. Fine-tuning is the only path. This is rare in regulated environments — regulated environments generate documents — but it occurs in early-stage teams that have not yet formalised their SOPs.
The honest answer on why most vendors pitch RAG-only
This deserves to be said plainly. Fine-tuning requires cloud GPU time — typically $8–$36 per run for a 14B–32B model on an A100 or H100, depending on dataset size and iterations. It also requires someone who knows how to prepare a training dataset, run the fine-tuning job, evaluate the adapter, and iterate. That is a skill and a process. RAG requires none of that. You index documents, you call a retrieval endpoint. It is a simpler delivery with a faster demo.
For many workloads, RAG-only is genuinely the right answer. But for workloads where stylistic consistency, refusal behaviour, and domain vocabulary are non-trivial requirements — which describes most regulated pharma and NBFC workloads — RAG-only is underdelivering. The model will retrieve your SOPs and respond in generic language that fails the QA head's review. The model will retrieve the customer's loan file and summarise it in a format that the compliance team cannot use. The fine-tune is what makes the output fit for purpose. Skipping it because the GPU bill requires a separate line item on the SOW is not a technical decision. It is a commercial convenience dressed up as one.
Bottom line.
Fine-tuning freezes behaviour. RAG keeps facts current. Regulated workloads require both because regulated workloads have requirements in both dimensions: consistent, auditable format and refusal patterns that do not change week to week, and citations grounded in the current version of documents that do. The five axes — data churn rate, audit obligation, latency budget, token budget, and data sensitivity — let you calibrate the weight you give each technique. In the pharma and NBFC contexts Reyatech serves, the calibration almost always lands in the same place: moderate-to-heavy fine-tune for domain behaviour, mandatory RAG for document grounding, and Langfuse-backed observability so that every production response is explainable to an inspector who has never heard of either technique.
If you are evaluating on-premise AI for a regulated workload and your vendor is proposing only one of the two, ask them to explain the gaps in writing. If they cannot, that explanation belongs in your SOW.