Fine-tuning, RAG or prompting: which one your problem actually needs

Teams that want an AI model to "know our business" usually hear three answers: write better prompts, add retrieval (RAG), or fine-tune the model. They are not competing brands of the same thing. Each changes a different part of the system, and each has a different cost and failure mode. Choosing the wrong one is the most common way an early pilot burns budget.

What each approach changes

Prompting means giving the model clear instructions, context and a few examples at the moment you ask. Nothing is trained and nothing is stored. It is the fastest to try and the easiest to undo.

Retrieval-augmented generation (RAG) adds a search step. When a question arrives, the system finds the relevant passages in your documents and places them in the prompt, so the model answers from your material instead of from memory. The documents stay outside the model and can be updated at any time.

Fine-tuning continues training a model on examples you supply, so the model's behavior shifts: a consistent format, a house style, a narrow classification task. The knowledge is baked into the model's weights, which means changing it later means training again.

The cost row is our general rule of thumb, not a measured figure.

What the evidence says about combining them

A 2024 study tested Llama 2 13B, GPT-3.5 and GPT-4 on an agricultural question-answering task. The authors report an accuracy increase of over 6 percentage points from fine-tuning, with RAG adding about 5 percentage points beyond that, and describe the gains as cumulative ^[1]. That is one domain and one dataset, so treat it as an illustration that the techniques complement each other, not as a universal ratio.

Retrieval quality is where RAG projects succeed or fail

RAG is only as good as the passages it finds. Anthropic's published work on "contextual retrieval" shows how much that step matters. In their tests, retrieving the top 20 chunks failed to surface the right information 5.7% of the time. Adding context to each chunk before indexing cut that failure rate by 35%, combining it with keyword search cut it by 49%, and adding a reranking step cut it by 67% ^[2].

The lesson for buyers: when a RAG system gives wrong answers, the usual cause is retrieval, not the model. Ask how retrieval is measured before asking which model is used.

A shortcut for small knowledge bases

Anthropic notes that if your knowledge base is smaller than about 200,000 tokens (roughly 500 pages), you can place all of it in the prompt and skip RAG entirely, and that prompt caching makes this faster and cheaper ^[2]. The company reports caching can cut cost by up to 90% and latency by up to 85% on long prompts ^[3]. For a handful of policy manuals or a product catalog, this can be the simplest architecture that works.

A practical order of operations

1. Write a test set first. Twenty to fifty real questions with known good answers, drawn from actual work. Without it you cannot tell whether any change helped.

2. Prompt. Get the instructions, tone and output format right. Many internal tools stop here.

3. Add retrieval when answers must come from documents the model has never seen or that change often. Measure retrieval separately from answer quality.

4. Fine-tune last, and only if the model still gets the behavior wrong after steps 2 and 3, for example a strict output format at high volume, or a narrow classification task with plenty of labeled examples.

Fine-tuning is rarely the right first move for "the AI doesn't know our products." That is a retrieval problem.

For most businesses, the practical implication is to treat these as a ladder, not a menu: prove value with prompting, add retrieval when documents are involved, and fine-tune only when a measured gap remains.