RAG explained: how AI answers questions from your own documents

A language model knows what it absorbed in training, which does not include your contracts, your product manuals or last week's policy change. Retrieval-augmented generation, universally shortened to RAG, is the standard way to close that gap. It is behind most "chat with your documents" products, and understanding its moving parts explains both why it works and why it sometimes fails.

The idea in plain terms

Think of an open-book exam. A model answering from memory alone is taking the test closed-book. RAG hands it the relevant pages first.

The 2020 paper that gave the technique its name described two kinds of memory. A pre-trained model stores factual knowledge in its parameters, which the authors call parametric memory. Their approach added "non-parametric memory," in their case a vector index of Wikipedia accessed by a neural retriever, and combined the two for generation ^[1]. They reported state-of-the-art results on three open-domain question-answering tasks and more specific, diverse and factual language than a model using parametric memory alone ^[1].

Anthropic's description of the production version is shorter: RAG pulls relevant information from a knowledge base and adds it to the user's prompt, which improves the model's response ^[2].

The steps

1. Prepare. Documents are split into passages ("chunks"), and each chunk is converted into a numerical representation (an embedding) that captures its meaning.

2. Store. The embeddings go into an index, usually a vector database, alongside the original text.

3. Retrieve. When someone asks a question, the system finds the chunks most similar in meaning to the question.

4. Generate. The top chunks are placed in the prompt, and the model is instructed to answer using them.

5. Cite. A well-built system shows which passages it used, so a person can check.

A concrete example: an employee asks, "How many days of bereavement leave do contractors get?" The system retrieves the paragraph of the HR policy that covers contractors and gives it to the model, which answers from that text rather than guessing from general knowledge. This is a generic illustration of the pattern.

When you don't need it

Anthropic's own guidance in its retrieval write-up is notable: if your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can include the entire knowledge base in the prompt, without RAG or similar methods ^[2]. Context windows have grown, so for a modest set of documents the simplest design may be to supply them all. RAG earns its keep when the corpus is larger than what fits in a prompt, or when cost and speed matter per query.

Where it fails

Most RAG failures are retrieval failures: the right passage exists but is not found, so the model answers from the wrong one or from nothing. Anthropic notes that traditional RAG can lose context when it encodes chunks, so the system sometimes fails to retrieve the right information. As our own illustration, a chunk that reads "revenue grew 3% over the previous quarter" does not say which company or which period unless that context travels with it.

Anthropic measured how much better indexing can help. Using the share of cases where the correct passage was missing from the top 20 retrieved results, a 5.7% baseline fell with each improvement ^[2]:

Even a 1.9% failure rate across thousands of daily questions is a lot of wrong answers, and retrieval is only one failure source. In a 2024 Stanford evaluation of commercial retrieval-augmented tools (which happened to be research tools for a professional domain), the tools still returned incorrect responses more than 17% of the time in two cases and more than 34% in a third. The authors credited retrieval with reducing errors against general-purpose models but titled a section "RAG Is Not a Panacea," pointing to hard retrieval problems, sources from the wrong context or time period, and sycophancy toward false premises in the question ^[3].

Questions to ask before you build or buy

• What is the corpus, and how current is it? An index is a snapshot until someone updates it.

• Who is allowed to see which documents? Retrieval must respect existing permissions, or the system becomes a way around them.

• Does the answer cite its sources? If not, review is slow and trust erodes.

• How will you measure it? Assemble a set of real questions with known answers and track the retrieval and answer quality separately.

• What happens when nothing relevant is found? The right behavior is to say so, not to improvise.

Practical implication

RAG is a sound way to ground an AI system in your own documents, provided you treat retrieval quality as the thing to test and keep citations visible. For a small document set, check whether simply supplying everything in the prompt does the job first.