Retrieval-augmented generation (RAG) is a method in which a language model first retrieves relevant passages from an external source, such as a knowledge base or a product catalog, and then writes its answer using those passages. The model answers from current, company-specific text it was given at query time, instead of relying only on what it memorized in training.
The term comes from a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research, which combined a trained text generator ("parametric memory") with a searchable index of documents ("non-parametric memory").1 The authors pointed to two problems with models that rely on training alone: they are hard to update, and they cannot show where an answer came from. Retrieval addresses both.
For a support or shopping agent, RAG is the difference between a model that knows what return policies usually say and one that quotes your return policy, as published this morning, with a link to the page.
The four steps of a RAG pipeline
A RAG system runs the same short pipeline for every question. Before any question arrives, the company's documents are split into passages (see chunking), converted into embeddings, and stored in an index, often a vector database.
Query
The customer's message, often rewritten into a clean search query that includes context from earlier turns ("it" becomes "the blue sofa, order 48213").
Retrieve
The system finds the passages most similar in meaning to the query. Dense retrievers that match embeddings beat keyword search (BM25) by 9 to 19 points on top-20 passage accuracy in open-domain question answering.2 Many production systems combine both.
Augment
The best passages are inserted into the prompt with an instruction such as "answer only from the sources below, and say so if they don't contain the answer."
Generate
The model writes the reply from those passages and can cite them. Lewis et al. found that RAG models produced more specific and factual language than a model without retrieval.1
One question, with and without retrieval
A customer asks an online retailer's assistant about returns. The model is the same in both lanes. Only the second can see the retailer's documents.
Model alone
Memory- RecallDraws on general patterns from training data: many retailers allow 14 or 30 days.
- Reply"Most furniture can be returned within 30 days, so you should be fine. Just pack it in the original box."
Plausible and wrong. This retailer extends returns for sale items but excludes made-to-order sofas. The model had no way to know.
RAG
Retrieved- QueryRewrites to: return policy, furniture, sale purchase, 20 days after delivery.
- Retrievekb.search(query, top_k=4)→ returns-policy §3 · holiday-sales-terms · made-to-order-items
- AugmentAdds the three passages to the prompt with the rule: answer only from these sources.
- ReplyExplains the extended return window for sale items, asks whether the sofa was made to order (those can't be returned), and links both policy pages.
Correct, current, and checkable. The answer cites the pages it used, so a person can verify it.
Illustrative example. Policy names and terms are placeholders, not any real retailer's rules.
RAG, fine-tuning, and long prompts
Teams usually weigh RAG against fine-tuning and against pasting everything into a large context window. They solve different problems and are often combined.
| Approach | What changes | Best for | Updating facts | Can cite sources |
|---|---|---|---|---|
| RAG | What the model sees at query time | Facts that change: policies, prices, stock, order data | Edit the source document; the next answer uses it | Yes, the retrieved passages |
| Fine-tuning | The model's weights | Tone, format, domain language, a narrow task done the same way every time | Retrain the model | No, knowledge is blended into the weights |
| Long prompt | The prompt, with every document pasted in | Small, stable document sets | Edit the prompt | Partly, but accuracy drops for text buried mid-prompt4 |
| Search only | Nothing in the model | Users who want documents, not answers | Re-index | Returns the documents themselves |
A common production setup fine-tunes for tone and task, and uses RAG for facts.
Where RAG does the work
RAG fits wherever the right answer lives in a document or record the company controls and updates.
Policy answers that match the policy
A telecom's agent answers roaming, billing, and contract questions from the current tariff documents, not from what tariffs looked like when the model was trained.
question → retrieve tariff → answer with linkSearch that answers
A marketplace's help search returns a direct answer built from several help articles, with the articles listed underneath.
query → retrieve top passages → summarize → citeAdvice from the real catalog
An electronics retailer's shopping agent retrieves product specs and stock before recommending, so it never suggests an item that doesn't exist.
needs → retrieve products → compare → recommendOne place to ask
An insurer's claims staff ask about procedures and get answers drawn from internal manuals, with the clause cited.
ask → retrieve manual → cite clauseWhere RAG goes wrong
RAG reduces hallucination but does not remove it. Research on dialogue systems found that retrieval substantially reduced knowledge hallucination,5 yet a model can still misread a passage or answer from memory when retrieval finds nothing useful.
Wrong passage retrieved
ControlMeasure retrieval on its own, with a test set of real questions and the passages that should answer them. Fix chunking and ranking before touching the prompt.
Outdated or conflicting sources
ControlGive every document an owner and a review date. Remove old versions from the index instead of leaving both.
Answer goes beyond the sources
ControlInstruct the model to answer only from the passages and to say when they don't cover the question. Score faithfulness on live conversations.6
Data leakage
ControlApply the user's access rights at retrieval time, so an agent never retrieves another customer's records or internal-only files.7
Poisoned content
ControlAccept documents only from trusted sources and audit the index for hidden instructions. Treat retrieved text as untrusted input that cannot change what the agent is allowed to do.7
Common questions
Is RAG better than fine-tuning?
They do different jobs. RAG gives a model facts it can cite and that you can update by editing a document. Fine-tuning changes how the model behaves, such as tone or output format. Most customer-facing agents use RAG for facts and, where needed, fine-tuning for style.
Does RAG stop hallucinations?
It reduces them, because the model answers from retrieved text instead of memory. It does not stop them. The model can still misread a passage or fill a gap when retrieval misses, so teams measure faithfulness and set the agent to say "I don't know" or hand over.
Do I need a vector database for RAG?
Not always. Small document sets can use keyword search or a simple embedding index. A vector database helps when you have many documents, need fast similarity search, or want to filter results by metadata such as market, language, or customer.
What is the difference between RAG and grounding?
Grounding is the goal: tying an answer to verifiable sources. RAG is one common way to achieve it, by retrieving those sources and giving them to the model before it answers.
References
- Lewis, P. et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020.
- Karpukhin, V. et al. Dense Passage Retrieval for Open-Domain Question Answering. EMNLP 2020.
- Gao, Y. et al. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv, 2023.
- Liu, N. F. et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the ACL (accepted 2023).
- Shuster, K. et al. Retrieval Augmentation Reduces Hallucination in Conversation. arXiv, 2021.
- Es, S. et al. Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv, 2023.
- OWASP GenAI Security Project. LLM08:2025 Vector and Embedding Weaknesses. OWASP Top 10 for LLM Applications, 2025.