# Fine-tuning, RAG or prompting: which one your problem actually needs

- **Published:** 2025-12-02
- **Updated:** 2026-10-12
- **Category:** Harnesses & Tooling
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/fine-tuning-vs-rag-vs-prompting

> Three ways to make a language model work with your business knowledge, what each one changes, and a simple order in which to try them.

Teams that want an AI model to "know our business" usually hear three answers: write better prompts, add retrieval (RAG), or fine-tune the model. They are not competing brands of the same thing. Each changes a different part of the system, and each has a different cost and failure mode. Choosing the wrong one is the most common way an early pilot burns budget.

**Key takeaways**

- Prompting changes the instructions, RAG changes the information the model sees, fine-tuning changes the model itself.
- Start with prompting, add retrieval when the answer depends on your documents, and fine-tune only when behavior or format still falls short.
- One published study found fine-tuning added over 6 points of accuracy and RAG about 5 more on top, so the methods can stack [1].
- Whichever you pick, build a small test set first so you can tell whether it helped.

## What each approach changes

**Prompting** means giving the model clear instructions, context and a few examples at the moment you ask. Nothing is trained and nothing is stored. It is the fastest to try and the easiest to undo.

**Retrieval-augmented generation (RAG)** adds a search step. When a question arrives, the system finds the relevant passages in your documents and places them in the prompt, so the model answers from your material instead of from memory. The documents stay outside the model and can be updated at any time.

**Fine-tuning** continues training a model on examples you supply, so the model's behavior shifts: a consistent format, a house style, a narrow classification task. The knowledge is baked into the model's weights, which means changing it later means training again.

|  | Prompting | RAG | Fine-tuning |
| --- | --- | --- | --- |
| What it changes | Instructions | Information supplied per question | The model itself |
| Best for | Tone, format, simple tasks | Answers that depend on current or private documents | Consistent behavior or style on a repeated task |
| Updating knowledge | Edit the prompt | Update the document store | Retrain |
| Typical cost to start | Hours | Days to weeks | Weeks, plus training data |
| Main risk | Long, brittle prompts | Poor retrieval returns the wrong passage | Stale knowledge, needs labeled examples |

The cost row is our general rule of thumb, not a measured figure.

## What the evidence says about combining them

A 2024 study tested Llama 2 13B, GPT-3.5 and GPT-4 on an agricultural question-answering task. The authors report an accuracy increase of over 6 percentage points from fine-tuning, with RAG adding about 5 percentage points beyond that, and describe the gains as cumulative [1]. That is one domain and one dataset, so treat it as an illustration that the techniques complement each other, not as a universal ratio.

## Retrieval quality is where RAG projects succeed or fail

RAG is only as good as the passages it finds. Anthropic's published work on "contextual retrieval" shows how much that step matters. In their tests, retrieving the top 20 chunks failed to surface the right information 5.7% of the time. Adding context to each chunk before indexing cut that failure rate by 35%, combining it with keyword search cut it by 49%, and adding a reranking step cut it by 67% [2].

**Retrieval failure rate, top-20 chunks**

Anthropic contextual retrieval tests

Unit: %

| Category | Failure rate |
| --- | --- |
| Baseline | 5.7% |
| Contextual embeddings | 3.7% |
| Plus contextual BM25 | 2.9% |
| Plus reranking | 1.9% |

*Lower is better. Figures are from the vendor's own experiments on its test datasets; your documents will behave differently. Source: [2], Anthropic*

The lesson for buyers: when a RAG system gives wrong answers, the usual cause is retrieval, not the model. Ask how retrieval is measured before asking which model is used.

## A shortcut for small knowledge bases

Anthropic notes that if your knowledge base is smaller than about 200,000 tokens (roughly 500 pages), you can place all of it in the prompt and skip RAG entirely, and that prompt caching makes this faster and cheaper [2]. The company reports caching can cut cost by up to 90% and latency by up to 85% on long prompts [3]. For a handful of policy manuals or a product catalog, this can be the simplest architecture that works.

## A practical order of operations

1. **Write a test set first.** Twenty to fifty real questions with known good answers, drawn from actual work. Without it you cannot tell whether any change helped.
2. **Prompt.** Get the instructions, tone and output format right. Many internal tools stop here.
3. **Add retrieval** when answers must come from documents the model has never seen or that change often. Measure retrieval separately from answer quality.
4. **Fine-tune last,** and only if the model still gets the behavior wrong after steps 2 and 3, for example a strict output format at high volume, or a narrow classification task with plenty of labeled examples.

Fine-tuning is rarely the right first move for "the AI doesn't know our products." That is a retrieval problem.

For most businesses, the practical implication is to treat these as a ladder, not a menu: prove value with prompting, add retrieval when documents are involved, and fine-tune only when a measured gap remains.

## Sources

1. [RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture](https://arxiv.org/abs/2401.08406), arXiv (Balaguer et al.), 2024-01-16
2. [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval), Anthropic, 2024-09-19
3. [Prompt caching with Claude](https://claude.com/blog/prompt-caching), Anthropic, 2025-08-14

## How Sinthetix helps

**Generative AI & LLM**: Integrate powerful AI models to turn unstructured data into insights and power chatbots.

- RAG systems
- LLM-powered chatbots
- Document analysis

[Explore Generative AI & LLM](https://sinthetix.com/services/generative-ai-llm) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)