# Measuring retrieval quality in RAG: recall, precision and MRR in plain English

- **Published:** 2026-06-24
- **Updated:** 2026-10-12
- **Category:** Harnesses & Tooling
- **Author:** Sinthetix
- **URL:** https://sinthetix.com/blog/measuring-retrieval-quality-recall-precision-mrr

> Test the retrieval step of a RAG system on its own: recall@k, precision@k, MRR and nDCG, a small worked example, and how to build a golden question set.

When a retrieval-augmented generation (RAG) system gives a wrong answer, two things could have gone wrong: the search step never found the right passage, or the model had it and still answered badly. Those are different faults with different fixes, and a single end-to-end "was the answer good?" score cannot tell them apart. The original RAG paper describes the design as two parts: a pre-trained generator paired with a neural retriever over an external index [6]. Test the parts separately.

## Why retrieval gets its own test

If the right passage is not in the context the model sees, no prompt change will rescue the answer. Retrieval metrics tell you whether the evidence arrived. Generation metrics tell you what the model did with it. Measuring retrieval first is cheaper too: it needs no model call to grade an answer, only a comparison between what came back and what should have come back.

## The four metrics

Each metric is computed per question and then averaged. "@k" means only the top k results are considered [5].

| Metric | What it asks | Formula in words | Use it when |
| --- | --- | --- | --- |
| Recall@k | Did the needed passages show up at all? | Relevant passages in the top k, divided by all relevant passages | You care most that the evidence reaches the model |
| Precision@k | How much of what came back is useful? | Relevant passages in the top k, divided by k | Context space or noise is a concern |
| MRR | How high is the first correct passage? | Average of 1 divided by the rank of the first relevant passage | One good passage is enough to answer |
| nDCG@k | Are the most relevant passages ranked highest? | Discounted gain of your ranking, divided by the gain of the ideal ranking | Passages vary in usefulness and order matters |

Recall and precision follow the definitions in Weaviate's guide, which calls them not rank-aware because they only count relevant items in the top k; MRR and nDCG are rank-aware because they also consider position [5]. Ragas, an open-source evaluation library, offers retrieval metrics under the same names and ideas: context recall asks how much of the needed information was retrieved and requires a reference to compare against [4], while context precision asks whether relevant chunks are ranked above irrelevant ones [3].

## A tiny worked example

This is an illustrative calculation with made-up rankings, not measured data. Take two golden questions and a top-5 result list for each.

- Question A has 2 relevant passages. They appear at ranks 2 and 4.
- Question B has 1 relevant passage. It does not appear in the top 5.

For question A: recall@5 is 2 of 2, so 1.0. Precision@5 is 2 of 5, so 0.4. The first relevant passage is at rank 2, so its reciprocal rank is 0.5. For nDCG with simple relevant-or-not scoring, the gain at rank r is discounted by log base 2 of (r + 1): the actual ranking scores 0.631 + 0.431 = 1.062, the ideal ranking (ranks 1 and 2) scores 1 + 0.631 = 1.631, and nDCG@5 is about 0.65.

For question B every metric is 0.

Averaged over the two questions: recall@5 is 0.5, precision@5 is 0.2, MRR is 0.25, and nDCG@5 is about 0.33. The average hides the real story, which is that one question was found imperfectly and the other was missed entirely.

## Build a golden set

Metrics need answers to compare against. A golden set is a list of real questions, each paired with the passage or passages that contain the answer. Ragas's recall metric depends on exactly this kind of reference, and its documentation notes that annotating reference contexts can be very time-consuming [4].

- Draw questions from real user queries or from the people who will use the system, not from questions invented after reading the documents.
- Record the exact source passage for each, so you can score by ID rather than by judging text.
- Include questions whose answer spans several passages and some that the documents cannot answer.
- Keep the set fixed so that changes in the score reflect changes in the system.

## The knobs, and how to read the results

Two settings move retrieval scores most often. Chunking: Anthropic notes that chunk size, boundaries and overlap can affect retrieval performance [1]. Search type: Anthropic reports that combining embeddings with BM25 keyword search, merged by rank fusion, beat embeddings alone [1], and the BEIR benchmark across 18 datasets found BM25 a robust baseline [2]. Re-ranking models scored best on average there, at higher computational cost [2].

Anthropic measured its own gains as retrieval failure rate, 1 minus recall@20, on its test set, so the figures show direction rather than a promise for your documents [1].

**Top-20 retrieval failure rate**

1 minus recall@20, lower is better

Unit: %

| Category | Failure rate |
| --- | --- |
| Embeddings only | 5.7% |
| Contextual embeddings | 3.7% |
| Contextual embeddings + BM25 | 2.9% |
| Plus reranking | 1.9% |

*Anthropic's reported results on its own evaluation, September 2024. Source: [1], Anthropic*

Reading your own results is simple. Low recall means the answer never reached the model: fix chunking, add keyword search, or re-rank before touching the prompt. Good recall with low precision or MRR means the evidence is present but buried or surrounded by noise. Good retrieval with bad answers moves the investigation to generation.

**Key takeaways**

- Evaluate retrieval separately from generation: it shows whether the evidence reached the model.
- Recall@k measures coverage, precision@k measures noise, MRR and nDCG add ranking position.
- A golden set of real questions with known source passages is the foundation for every metric.
- Low recall means the answer never reached the model, so fix chunking and search before the prompt.

In practice, a few dozen well-chosen golden questions and a recall@k number tracked on every change will tell you more about a RAG system than reading sample answers by eye.

## Sources

1. [Introducing Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval), Anthropic, 2024-09-19
2. [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://arxiv.org/abs/2104.08663), arXiv (Thakur et al.; NeurIPS 2021 Datasets and Benchmarks), 2021-04-17
3. [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/), Ragas documentation
4. [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/), Ragas documentation
5. [Evaluation Metrics for Search and Recommendation Systems](https://weaviate.io/blog/retrieval-evaluation-metrics), Weaviate, 2024-05-28
6. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401), arXiv (Lewis et al.; NeurIPS 2020), 2020-05-22

## How Sinthetix helps

**Generative AI & LLM**: Integrate powerful AI models to turn unstructured data into insights and power chatbots.

- RAG systems
- LLM-powered chatbots
- Document analysis

[Explore Generative AI & LLM](https://sinthetix.com/services/generative-ai-llm) | [Talk to us](https://sinthetix.com/contact)

Sinthetix is an AI engineering studio. We ship agents, automation, and internal tools for operators who need results, not reports. [About Sinthetix](https://sinthetix.com/about)