Why AI models hallucinate, and how to contain it
A plain-English explanation of why language models state false things confidently, what research says about the cause, and practical ways to limit the damage.
A hallucination is an output that sounds plausible and is wrong: an invented citation, a made-up policy clause, a figure that appears in no source document. For a business, the problem is not that models are sometimes wrong. People are too. The problem is that the model sounds equally confident when it is right and when it is wrong.
Why it happens
A language model predicts likely text. It does not look facts up unless it is connected to a source. A September 2025 paper by Kalai, Nachum, Vempala and Zhang argues that hallucinations arise from training and evaluation procedures that reward guessing over acknowledging uncertainty, so models produce "plausible yet incorrect statements instead of admitting uncertainty" [1]. The authors frame the problem as a classification issue: if incorrect statements cannot be distinguished from facts, statistical pressures in pretraining will produce them [1].
Evaluation makes it worse. The paper notes that most evaluations are graded so that guessing improves a score, so "language models are optimized to be good test-takers" [1]. Think of an exam with no penalty for wrong answers: the best strategy is never to leave a question blank. The authors call for changing how widely used benchmarks are scored, rather than only adding new hallucination tests [1].
The takeaway for buyers is that confident wrong answers are a predictable property of current systems, not a defect that the next version will quietly remove.
A concrete example
Imagine a distributor's support team asks an assistant, "What is our return window for opened electrical components?" If the assistant answers from general training, it may produce a reasonable-sounding number that matches no company policy. If it retrieves the actual returns policy and answers from it, the answer is traceable. The difference is where the model gets its facts. (This is an illustration, not a reported case.)
How to contain it
Technique | How it helps | Limit |
|---|---|---|
Ground answers in your documents (retrieval-augmented generation) | The model answers from retrieved passages, not memory | Retrieval can fetch the wrong passage |
Require citations to a specific source | Reviewers can check the claim quickly | The model can still cite poorly |
Allow "I don't know" | Counters the guessing incentive | Needs testing so it does not refuse everything |
Test against a fixed question set | Catches regressions when models or prompts change | Only as good as the question set |
Human review on high-stakes outputs | Catches what automation misses | Costs time |
Retrieval-augmented generation (RAG) comes from a 2020 paper by Lewis and colleagues, which paired a language model's internal memory with a searchable index of documents [2]. The authors reported more specific, diverse and factual language than a baseline that relied on the model's memory alone [2]. They also described provenance for a model's decisions and updating its knowledge as open research problems at the time [2], which is a fair description of where production systems still need care.
A policy that works in practice
Classify use cases by consequence. A draft marketing email and a contract summary do not deserve the same controls.
Make grounding the default for anything factual about your business.
Log inputs, sources and outputs so mistakes can be traced.
Set an accuracy bar before launch and measure against it with real questions from your own staff.
Assign an owner for monitoring and for changes to models or prompts.
NIST's AI Risk Management Framework is a voluntary framework for organizing this kind of work, and NIST has also published a Generative AI Profile (NIST-AI-600-1, July 2024) that identifies risks specific to generative AI [3].
Practical implication
Treat hallucination as a design constraint, not a vendor promise: ground answers in your own documents, require sources, and keep a person responsible for decisions that carry real cost.
Sources
- Why Language Models Hallucinate arXiv (Kalai, Nachum, Vempala, Zhang), 2025-09-04. arxiv.org/abs/2509.04664
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks arXiv (Lewis et al.), NeurIPS 2020, 2020-05-22. arxiv.org/abs/2005.11401
- NIST AI Risk Management Framework NIST, 2023-01-26. www.nist.gov/itl/ai-risk-management-framework
.jpg){kind=link}