Retrieval-augmented generation (RAG) lets a language model answer from your own documents. Hallucinations arise when the model gets “creative” beyond the passage it found. The fix lies in architecture: better retrieval quality when finding the passage, and a strict, checkable tie between every statement and that passage.
At a glance
- Hybrid search combining keyword and vector search finds more relevant passages than either method alone.
- What follows decides: every single sentence of an answer is checked against the retrieved source sentences with defined thresholds.
- If there is not enough evidence, the system has to say so and must not guess. Declining is a valid result.
Where RAG hallucinates
A model that answers from retrieved documents takes two separate steps. It searches for suitable passages (retrieval) and writes an answer from them (generation). Hallucination mostly arises in the second step. The model writes more than the passage supports, or merges several sources into a statement that no single source supports. A solid architecture treats the two steps separately.
Hybrid search as the base
Pure vector search finds meaning and loses exact terms such as article numbers, names and file references. Classic keyword search (BM25) is strong on exact terms and weak on paraphrase. Combining the two, for example through reciprocal rank fusion, raises retrieval quality measurably compared with either method alone. From the larger set of candidates, a cross-encoder reranker then picks the most relevant passages. Only these go to the model, with less noise and a more precise basis for the answer.
The check, sentence by sentence
Every single sentence is checked against the closest source sentences by meaning. A checking model assesses for each answer sentence whether the retrieved source sentences support it. If the support falls short of the defined thresholds, the sentence is dropped, and the system reports that it finds no sufficient evidence.
When the evidence is missing
If the system finds no sufficient evidence for a statement, it leaves the statement out and says so. The user then sees where the sources do not hold.
Hallucinations cannot be ruled out completely. The system checks every sentence against the retrieved source sentences, and anything that fails the check does not appear in the answer.
Traceability down to the source sentence
Beyond checking the answer, faithfulness to the evidence can be anchored in the knowledge base itself, for example in a knowledge graph. Every stored relation carries two separate time axes there: since when a fact holds in the real world, and since when the system has known about it. Attached to each edge are the source sentence, the character position and the verbatim evidence text. Corrections never delete an earlier version. They close it and open a new one, so the old evidence stays traceable. Every statement in the system can be clicked and followed back to the exact sentence in the source document.
What this looks like in an archive system is shown by our product RETRAYA. Why this architecture also makes deletion requests easier is covered in the article AI and the GDPR in business.