Measuring retrieval quality in RAG: recall, precision and MRR in plain English

When a retrieval-augmented generation (RAG) system gives a wrong answer, two things could have gone wrong: the search step never found the right passage, or the model had it and still answered badly. Those are different faults with different fixes, and a single end-to-end "was the answer good?" score cannot tell them apart. The original RAG paper describes the design as two parts: a pre-trained generator paired with a neural retriever over an external index ^[6]. Test the parts separately.

Why retrieval gets its own test

If the right passage is not in the context the model sees, no prompt change will rescue the answer. Retrieval metrics tell you whether the evidence arrived. Generation metrics tell you what the model did with it. Measuring retrieval first is cheaper too: it needs no model call to grade an answer, only a comparison between what came back and what should have come back.

The four metrics

Each metric is computed per question and then averaged. "@k" means only the top k results are considered ^[5].

Recall and precision follow the definitions in Weaviate's guide, which calls them not rank-aware because they only count relevant items in the top k; MRR and nDCG are rank-aware because they also consider position ^[5]. Ragas, an open-source evaluation library, offers retrieval metrics under the same names and ideas: context recall asks how much of the needed information was retrieved and requires a reference to compare against ^[4], while context precision asks whether relevant chunks are ranked above irrelevant ones ^[3].

A tiny worked example

This is an illustrative calculation with made-up rankings, not measured data. Take two golden questions and a top-5 result list for each.

• Question A has 2 relevant passages. They appear at ranks 2 and 4.

• Question B has 1 relevant passage. It does not appear in the top 5.

For question A: recall@5 is 2 of 2, so 1.0. Precision@5 is 2 of 5, so 0.4. The first relevant passage is at rank 2, so its reciprocal rank is 0.5. For nDCG with simple relevant-or-not scoring, the gain at rank r is discounted by log base 2 of (r + 1): the actual ranking scores 0.631 + 0.431 = 1.062, the ideal ranking (ranks 1 and 2) scores 1 + 0.631 = 1.631, and nDCG@5 is about 0.65.

For question B every metric is 0.

Averaged over the two questions: recall@5 is 0.5, precision@5 is 0.2, MRR is 0.25, and nDCG@5 is about 0.33. The average hides the real story, which is that one question was found imperfectly and the other was missed entirely.

Build a golden set

Metrics need answers to compare against. A golden set is a list of real questions, each paired with the passage or passages that contain the answer. Ragas's recall metric depends on exactly this kind of reference, and its documentation notes that annotating reference contexts can be very time-consuming ^[4].

• Draw questions from real user queries or from the people who will use the system, not from questions invented after reading the documents.

• Record the exact source passage for each, so you can score by ID rather than by judging text.

• Include questions whose answer spans several passages and some that the documents cannot answer.

• Keep the set fixed so that changes in the score reflect changes in the system.

The knobs, and how to read the results

Two settings move retrieval scores most often. Chunking: Anthropic notes that chunk size, boundaries and overlap can affect retrieval performance ^[1]. Search type: Anthropic reports that combining embeddings with BM25 keyword search, merged by rank fusion, beat embeddings alone ^[1], and the BEIR benchmark across 18 datasets found BM25 a robust baseline ^[2]. Re-ranking models scored best on average there, at higher computational cost ^[2].

Anthropic measured its own gains as retrieval failure rate, 1 minus recall@20, on its test set, so the figures show direction rather than a promise for your documents ^[1].

Reading your own results is simple. Low recall means the answer never reached the model: fix chunking, add keyword search, or re-rank before touching the prompt. Good recall with low precision or MRR means the evidence is present but buried or surrounded by noise. Good retrieval with bad answers moves the investigation to generation.

In practice, a few dozen well-chosen golden questions and a recall@k number tracked on every change will tell you more about a RAG system than reading sample answers by eye.