AI Glossary Models and how they work

AI hallucination

  • Tidio Agents team
  • Updated November 19, 2026
  • 6 min read
  • 8 sources
Definition

An AI hallucination is output from a generative model that sounds fluent and confident but is false, invented, or not supported by the sources the model was given. Examples include a refund policy that doesn't exist, a wrong delivery date, or a citation to a page that was never written.

The term is borrowed from psychology and is imperfect: the model is not perceiving anything. NIST prefers confabulation, which it defines as confidently stated but erroneous or false content, and notes that "hallucination" and "fabrication" are the colloquial names.4

For a company, the important distinction is between a model that is wrong and one that is wrong and sure of itself. Customers can work around an assistant that says "I don't know." They act on one that invents an answer.

Causes

Why language models hallucinate

A large language model is trained to predict likely text. It has no built-in step that checks a sentence against the world, so a likely-sounding falsehood and a true statement come out the same way.

Guessing is rewarded

Kalai and colleagues at OpenAI argue that training and benchmark scoring reward a guess over an admission of uncertainty, much as a multiple-choice exam rewards guessing. The model learns to answer anyway.3

Gaps in knowledge

Models struggle with rare, long-tail facts and know nothing after their training data ends. Your new returns policy is, by definition, not in the training data.2

Decoding and overconfidence

Text is sampled one token at a time, and an early wrong token can cascade through the rest of the answer, like a snowball.2

Missing or ignored context

When retrieval fails or the right passage is buried in a long prompt, the model fills the gap from memory instead of saying the answer isn't there.

Types

Types of hallucination

Researchers use two overlapping taxonomies. The older one, from work on summarization, asks how the output relates to the source.1 The newer one, written for chat models, asks whether the output conflicts with the world or with the user's input.2

TypeWhat goes wrongSupport example
IntrinsicThe output contradicts the source it was given1The policy says 30 days; the agent says 14
ExtrinsicThe output adds content the source neither supports nor contradicts1The agent adds "free return shipping," which no document mentions
FactualityThe output conflicts with verifiable real-world facts2Wrong opening hours, a product that doesn't exist
FaithfulnessThe output departs from the user's instructions or context, or contradicts itself2Asked about order 48213, answers about another order

Support examples are illustrative. In customer service, extrinsic errors are the most common worry: plausible extras nobody approved.

Example

The same question, with and without controls

A traveler asks an airline's assistant about a refund. Both lanes use the same model. The second is built to answer only from the booking and the fare rules.

No controls

Invents
CustomerMy flight on Friday was moved by 4 hours. Am I entitled to a full refund?
  1. RecallNo booking data or fare rules in context. Draws on general patterns about schedule changes.
  2. Reply"Yes, any change over 3 hours qualifies for a full refund plus €250 compensation. I've noted your request."

Two inventions. A compensation amount from nowhere, and a claim that a request was logged when no action was taken.

Grounded, with checks

Sourced
CustomerMy flight on Friday was moved by 4 hours. Am I entitled to a full refund?
  1. Toolbookings.get(pnr)→ fare class · schedule change: +4h
  2. Retrievepolicy.search("schedule change refund", fare_class)→ schedule-change policy §2
  3. CheckDraft claims compared with the booking and §2. Compensation is not covered by these sources, so it is removed.
  4. ReplyExplains the refund option §2 allows, links it, and offers a person for the compensation question.

Every claim traceable. What the sources don't cover goes to a person instead of being guessed.

Illustrative example. Thresholds, amounts, and policy sections are placeholders, not real airline rules.

Controls

How support and commerce agents reduce hallucination

No single fix removes hallucination. Production agents stack several controls, each catching what the last one missed.

Answer from sources

ControlUse retrieval and tool calls so the model sees the policy and the order record. Retrieval substantially reduced knowledge hallucination in dialogue research.5

Permit "I don't know"

ControlInstruct the agent to say when sources don't cover a question, and reward that in testing. This targets the guessing incentive directly.3

Check before sending

ControlCompare each claim with its sources and block or rewrite unsupported ones. Run stricter checks on prices, dates, amounts, and legal terms.

Separate saying from doing

ControlLet the agent claim an action only after a tool call has returned success. "I've issued your refund" must follow a confirmed refund.

Hand over with context

ControlRoute questions outside the sources to a person, with the conversation and retrieved documents attached.

Measure

How to measure hallucination

Measure it on your own conversations. Public leaderboards test general knowledge, while your risk sits in your policies and products.

Factual precision (FActScore method)
supported atomic facts ÷ all atomic facts in the answer
Atomic facts in an answer
12
Supported by the knowledge source
11
Factual precision
92%

Illustrative numbers. FActScore splits text into short atomic facts and checks each against a trusted source. In the original study, ChatGPT scored 58% on people's biographies.6

  • Atomic fact checking: split answers into single claims and verify each against the knowledge base.6
  • Consistency sampling: ask the same question several times. Facts the model knows tend to repeat; invented ones vary.7
  • Faithfulness scores for RAG: check whether the answer stays within the retrieved passages, separately from whether retrieval found the right ones.8
  • Human review: a weekly sample of live conversations, scored by people who know the policies.
In practice

Where hallucination costs the most

Risk follows consequence. An invented fact in small talk is harmless; an invented fee in a contract answer is not.

Fintech and insurance

Terms, fees, and claims

A wrong statement about an installment fee or a claim's coverage can create a complaint or a regulatory issue. Answers here need strict grounding and citations.

contract → answer → cite clause
Shopping

Products that exist

A shopping agent must recommend only items in the catalog and in stock. Specs come from product data, not from what similar products usually have.

catalog → filter stock → recommend
Telecoms

Plans and billing

Prices and contract terms change often. The agent quotes the current tariff and the customer's own plan.

customer plan → tariff → quote
Travel

Changes and refunds

Fare rules differ per booking. The agent reads them for each case and hands over when they don't answer the question.

booking → fare rules → answer or hand over
FAQ

Common questions

Why does ChatGPT make things up?

Language models generate likely text and have no built-in fact check. Research from OpenAI argues that training and evaluation reward guessing over admitting uncertainty, so models answer even when they shouldn't.

Can AI hallucinations be eliminated?

Not fully with current models. They can be reduced sharply by grounding answers in sources, checking claims before sending, letting the agent say "I don't know," and handing unclear cases to a person.

Does RAG prevent hallucination?

It reduces it, because the model answers from retrieved text instead of memory. The model can still misread or go beyond a passage, so RAG systems also need faithfulness checks.

How do you test an AI agent for hallucinations?

Run it on real customer questions with known correct answers, split its replies into individual claims, and check each against your sources. Keep scoring a sample of live conversations after launch.

Sources

Tidio Agents

Build an agent that resolves, not only answers

We design and build custom AI agents for customer service, shopping and back-office work, on your infrastructure, with the code handed over to you. Projects start with a 21-day private AI audit.

Request a call