LLM FundamentalsMidQuestion 7 of 13

When would you choose fine-tuning over RAG, and vice versa?

30-second answer

Use RAG when the model needs fresh, private, or citable knowledge, or when data changes often. Use fine-tuning to change behaviour: consistent format or tone, domain-specific style, a narrow skill, or to shorten long prompts at high volume. Fine-tuning is unreliable for injecting facts. Often the best answer is both: fine-tune for behaviour, retrieve for knowledge, and start with prompting because it's cheapest to iterate.

Try answering out loud before reading on. The walkthrough below is paced for a 40-minute interview round.

0–5 min Clarify and frame

This is a decision question, so ask about the use case before answering:

What's wrong today: missing knowledge, wrong behaviour/format, or cost/latency?
How often does the knowledge change? Is it private? Must answers cite sources?
Volume, latency budget, and whether there's labelled data.

Framing: "RAG changes what the model knows at query time; fine-tuning changes how it behaves. I'll compare them on several axes and give a decision process."

5–15 min Core explanation

RAG: retrieve relevant documents at query time and put them in the prompt.

Knowledge stays fresh (update the index, not the model).
Supports citations and per-user permissions.
Adds retrieval latency and prompt tokens; quality depends on retrieval.

Fine-tuning: continue training the model on examples.

Changes style, format, tone, domain jargon handling, task skill.
Can shorten prompts (instructions baked in), cutting cost at high volume.
Poor at reliably adding facts; can increase confident hallucination of half-learned facts; slow to update; needs data and evals.

Prompting comes first: it's cheapest to iterate and often enough.

15–25 min Deep dive: comparison

Axis	Prompting	RAG	Fine-tuning
Adds new/private facts	✗	✓ (best)	Weak
Freshness	—	Minutes (re-index)	Retrain cycle
Citations / auditability	✗	✓	✗
Consistent format/style	Medium	Medium	✓ (best)
Per-request cost	Prompt length	+ retrieved tokens	Lower (shorter prompts)
Upfront effort	Low	Medium (pipeline)	Medium–high (data, training, eval)
Access control	—	✓ at retrieval	✗ (baked into weights)

Decision process:

Prompt engineering plus an eval set. Measure.
Missing knowledge → RAG.
Wrong behaviour despite good prompts, or prompts too long/costly at scale → fine-tune (often LoRA).
Both problems → fine-tune for behaviour, RAG for knowledge. You can even fine-tune the model to use retrieved context better.

25–32 min Trade-offs and pitfalls

Fine-tuning on proprietary docs to "teach facts" tends to give fluent but unreliable recall and leaks content across users.
RAG failures are usually retrieval failures; fix chunking, hybrid search and reranking before blaming the model.
Fine-tuned models need re-validation when the base model updates.
Distillation: fine-tune a small model on a large model's outputs for a narrow task to cut cost and latency.

32–37 min Production example

A legal-tech startup wanted answers grounded in each customer's contracts in a strict house format. They used RAG over per-tenant indexes for facts and citations, and LoRA fine-tuned a mid-size model on ~2,000 reviewed examples to produce the exact clause-analysis format. The fine-tune also let them drop 1,500 tokens of format instructions from every prompt.

37–40 min Likely follow-ups

How much data for fine-tuning? Hundreds to a few thousand high-quality examples for behaviour; quality beats quantity.
How do you evaluate the choice? Same golden set across options; measure accuracy, groundedness, format adherence, latency and cost.
What about long context instead of RAG? Viable for small static corpora; RAG wins on cost, freshness, permissions and citations at scale.

What interviewers listen for

Asking clarifying questions instead of naming one answer.
Knowledge vs. behaviour distinction.
Starting with prompting and evals.

Red flags

"Fine-tune it on our documents" as the default for knowledge.
No mention of evaluation.