An AI hallucination is output from a generative model that sounds fluent and confident but is false, invented, or not supported by the sources the model was given. Examples include a refund policy that doesn't exist, a wrong delivery date, or a citation to a page that was never written.
The term is borrowed from psychology and is imperfect: the model is not perceiving anything. NIST prefers confabulation, which it defines as confidently stated but erroneous or false content, and notes that "hallucination" and "fabrication" are the colloquial names.4
For a company, the important distinction is between a model that is wrong and one that is wrong and sure of itself. Customers can work around an assistant that says "I don't know." They act on one that invents an answer.
Why language models hallucinate
A large language model is trained to predict likely text. It has no built-in step that checks a sentence against the world, so a likely-sounding falsehood and a true statement come out the same way.
Guessing is rewarded
Kalai and colleagues at OpenAI argue that training and benchmark scoring reward a guess over an admission of uncertainty, much as a multiple-choice exam rewards guessing. The model learns to answer anyway.3
Gaps in knowledge
Models struggle with rare, long-tail facts and know nothing after their training data ends. Your new returns policy is, by definition, not in the training data.2
Decoding and overconfidence
Text is sampled one token at a time, and an early wrong token can cascade through the rest of the answer, like a snowball.2
Missing or ignored context
When retrieval fails or the right passage is buried in a long prompt, the model fills the gap from memory instead of saying the answer isn't there.
Types of hallucination
Researchers use two overlapping taxonomies. The older one, from work on summarization, asks how the output relates to the source.1 The newer one, written for chat models, asks whether the output conflicts with the world or with the user's input.2
| Type | What goes wrong | Support example |
|---|---|---|
| Intrinsic | The output contradicts the source it was given1 | The policy says 30 days; the agent says 14 |
| Extrinsic | The output adds content the source neither supports nor contradicts1 | The agent adds "free return shipping," which no document mentions |
| Factuality | The output conflicts with verifiable real-world facts2 | Wrong opening hours, a product that doesn't exist |
| Faithfulness | The output departs from the user's instructions or context, or contradicts itself2 | Asked about order 48213, answers about another order |
Support examples are illustrative. In customer service, extrinsic errors are the most common worry: plausible extras nobody approved.
The same question, with and without controls
A traveler asks an airline's assistant about a refund. Both lanes use the same model. The second is built to answer only from the booking and the fare rules.
No controls
Invents- RecallNo booking data or fare rules in context. Draws on general patterns about schedule changes.
- Reply"Yes, any change over 3 hours qualifies for a full refund plus €250 compensation. I've noted your request."
Two inventions. A compensation amount from nowhere, and a claim that a request was logged when no action was taken.
Grounded, with checks
Sourced- Toolbookings.get(pnr)→ fare class · schedule change: +4h
- Retrievepolicy.search("schedule change refund", fare_class)→ schedule-change policy §2
- CheckDraft claims compared with the booking and §2. Compensation is not covered by these sources, so it is removed.
- ReplyExplains the refund option §2 allows, links it, and offers a person for the compensation question.
Every claim traceable. What the sources don't cover goes to a person instead of being guessed.
Illustrative example. Thresholds, amounts, and policy sections are placeholders, not real airline rules.
How support and commerce agents reduce hallucination
No single fix removes hallucination. Production agents stack several controls, each catching what the last one missed.
Answer from sources
ControlUse retrieval and tool calls so the model sees the policy and the order record. Retrieval substantially reduced knowledge hallucination in dialogue research.5
Permit "I don't know"
ControlInstruct the agent to say when sources don't cover a question, and reward that in testing. This targets the guessing incentive directly.3
Check before sending
ControlCompare each claim with its sources and block or rewrite unsupported ones. Run stricter checks on prices, dates, amounts, and legal terms.
Separate saying from doing
ControlLet the agent claim an action only after a tool call has returned success. "I've issued your refund" must follow a confirmed refund.
Hand over with context
ControlRoute questions outside the sources to a person, with the conversation and retrieved documents attached.
How to measure hallucination
Measure it on your own conversations. Public leaderboards test general knowledge, while your risk sits in your policies and products.
- Atomic facts in an answer
- 12
- Supported by the knowledge source
- 11
- Factual precision
- 92%
Illustrative numbers. FActScore splits text into short atomic facts and checks each against a trusted source. In the original study, ChatGPT scored 58% on people's biographies.6
- Atomic fact checking: split answers into single claims and verify each against the knowledge base.6
- Consistency sampling: ask the same question several times. Facts the model knows tend to repeat; invented ones vary.7
- Faithfulness scores for RAG: check whether the answer stays within the retrieved passages, separately from whether retrieval found the right ones.8
- Human review: a weekly sample of live conversations, scored by people who know the policies.
Where hallucination costs the most
Risk follows consequence. An invented fact in small talk is harmless; an invented fee in a contract answer is not.
Terms, fees, and claims
A wrong statement about an installment fee or a claim's coverage can create a complaint or a regulatory issue. Answers here need strict grounding and citations.
contract → answer → cite clauseProducts that exist
A shopping agent must recommend only items in the catalog and in stock. Specs come from product data, not from what similar products usually have.
catalog → filter stock → recommendPlans and billing
Prices and contract terms change often. The agent quotes the current tariff and the customer's own plan.
customer plan → tariff → quoteChanges and refunds
Fare rules differ per booking. The agent reads them for each case and hands over when they don't answer the question.
booking → fare rules → answer or hand overCommon questions
Why does ChatGPT make things up?
Language models generate likely text and have no built-in fact check. Research from OpenAI argues that training and evaluation reward guessing over admitting uncertainty, so models answer even when they shouldn't.
Can AI hallucinations be eliminated?
Not fully with current models. They can be reduced sharply by grounding answers in sources, checking claims before sending, letting the agent say "I don't know," and handing unclear cases to a person.
Does RAG prevent hallucination?
It reduces it, because the model answers from retrieved text instead of memory. The model can still misread or go beyond a passage, so RAG systems also need faithfulness checks.
How do you test an AI agent for hallucinations?
Run it on real customer questions with known correct answers, split its replies into individual claims, and check each against your sources. Keep scoring a sample of live conversations after launch.
References
- Ji, Z. et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 2022.
- Huang, L. et al. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. arXiv, 2023.
- Kalai, A. T. et al. Why Language Models Hallucinate. arXiv, 2025.
- NIST. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). July 2024.
- Shuster, K. et al. Retrieval Augmentation Reduces Hallucination in Conversation. arXiv, 2021.
- Min, S. et al. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. EMNLP 2023.
- Manakul, P. et al. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. EMNLP 2023.
- Es, S. et al. Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv, 2023.