Retrieval-augmented generation, explained for business
03 November 2026
A language model on its own answers from what it learned in training. It knows nothing about your price list, your manuals or last month’s policy change, and when it does not know, it can still produce a fluent, wrong answer.
Retrieval-augmented generation, or RAG, fixes that by changing the order of operations. Before the model answers, the system looks up the relevant passages in your own documents and hands them to the model with the question. The model writes the answer from those passages.
What is RAG in AI?
RAG has two stages that run on every question.
- Retrieve. Your documents were split into passages (chunks) beforehand and indexed so they can be searched by meaning, not just by keyword. When a question arrives, the system pulls the few passages most likely to contain the answer. A second model, a reranker, often re-orders them by how well they fit the question.
- Generate. The language model receives the question plus those passages and is told to answer only from them. A good system also shows which passages it used, so a person can check.
That is why a RAG chatbot can answer from the installation manual and name the section it used, instead of guessing. The answer comes from a document you control. When the document changes, you update the index, and the answers change with it. No retraining.
What is the difference between RAG and fine-tuning?
Fine-tuning changes the model itself by training it further on examples. RAG leaves the model alone and gives it the right material at question time.
For business knowledge, RAG is usually the better fit:
- Freshness. Update a document and the next answer uses it. A fine-tuned model has to be retrained.
- Traceability. RAG can point to the source passage. A fine-tuned model cannot tell you where an answer came from.
- Deletion. Removing a document removes its content from the system. Removing something a model has already learned is hard.
- Cost. Indexing documents is cheap next to training runs.
Fine-tuning still has a place, mainly for style, format or a narrow task the model keeps getting wrong. Many production systems use RAG for the facts and, occasionally, a lightly tuned model for the tone.
How much data do I need for RAG?
Less than people expect, and the quality matters more than the quantity. A modest library can be enough. The assistant we built for a German home-security manufacturer runs on 112 documents and videos, which became more than 1,700 indexed passages once they were transcribed and split. What matters more than count:
- Coverage. Do the documents contain the answers to the questions people actually ask? Read your last 100 customer emails and check.
- Currency. Outdated PDFs create confident wrong answers. Someone has to own the content.
- Structure. Scanned images, tables and videos need processing before they can be searched. Budget time for it.
- No contradictions. If two documents disagree, the system will pick one.
The audit usually finds that the biggest gap is an answer that lives in one person’s head.
What is the best way to evaluate a RAG chatbot?
Do not test it by chatting with it for ten minutes. Build a fixed set of real questions with the correct answer for each, and run the whole set every time something changes: new documents, a new model, a changed prompt.
Measure three things separately:
- Retrieval. Did the right passage come back at all? If not, the model never had a chance.
- Answer quality. Is the answer correct and complete against the source?
- Behaviour on questions it should not answer. Does it say it does not know, or hand over to a person, instead of inventing something?
On our own RAG assistants, an evaluation suite runs on every deploy, and a release does not go out if it regresses. The last release of the home-security assistant scored 21 of 22 on that suite.
Most teams skip the third measure. We replay real past escalations through the assistant’s guards before each release. Handover works through a calibrated threshold: when the best retrieved passage scores below it, the assistant does not answer. It passes the conversation to a person with the full context attached. Recalibrating that threshold did not cause regressions on the evaluation suite.
Is RAG GDPR compliant?
RAG is an architecture, not a legal status. Compliance depends on what is in the documents and where each component runs.
Check these points:
- Your documents. If they contain personal data, the index holds it too. Decide whether it needs to be there.
- The index. Where is it stored, and who can query it? Access to the index should follow the same permissions as the source documents.
- The model provider. The question and the retrieved passages are sent to the model. Under GDPR article 28, a provider that processes personal data on your behalf needs a data processing agreement. Check the region and the provider’s retention terms.
- Conversation logs. Chat history is personal data when users identify themselves. Set a retention period.
For sensitive material, a private deployment keeps the index and the application on infrastructure you control. Our GDPR checklist goes through the points in order. This is not legal advice, and your data protection officer decides on your own set-up.
Where RAG breaks
Retrieval finds the wrong passage because the question uses different words from the document. Chunks cut a table in half. The documents contradict each other. The model ignores its instructions and fills a gap from memory. Each of these is testable, which is why the evaluation set matters more than the model you pick.
To see what a RAG assistant looks like on your own files, read about our RAG chatbot, or start with the free automation audit. We check your documents against your real customer questions and tell you what the assistant would answer today.