Skip to content

Archive and knowledge systems · 4 min read

RAG with sources: how an answer can be checked against its sources

An AI that answers from documents can go wrong in two places. Both errors can be checked technically, and the system can decline to answer when in doubt.

Retrieval-augmented generation (RAG) lets a language model answer from your own documents. Hallucinations arise when the model gets “creative” beyond the passage it found. The fix lies in architecture: better retrieval quality when finding the passage, and a strict, checkable tie between every statement and that passage.

At a glance

  • Hybrid search combining keyword and vector search finds more relevant passages than either method alone.
  • What follows decides: every single sentence of an answer is checked against the retrieved source sentences with defined thresholds.
  • If there is not enough evidence, the system has to say so and must not guess. Declining is a valid result.

Where RAG hallucinates

A model that answers from retrieved documents takes two separate steps. It searches for suitable passages (retrieval) and writes an answer from them (generation). Hallucination mostly arises in the second step. The model writes more than the passage supports, or merges several sources into a statement that no single source supports. A solid architecture treats the two steps separately.

Hybrid search as the base

Pure vector search finds meaning and loses exact terms such as article numbers, names and file references. Classic keyword search (BM25) is strong on exact terms and weak on paraphrase. Combining the two, for example through reciprocal rank fusion, raises retrieval quality measurably compared with either method alone. From the larger set of candidates, a cross-encoder reranker then picks the most relevant passages. Only these go to the model, with less noise and a more precise basis for the answer.

The check, sentence by sentence

Every single sentence is checked against the closest source sentences by meaning. A checking model assesses for each answer sentence whether the retrieved source sentences support it. If the support falls short of the defined thresholds, the sentence is dropped, and the system reports that it finds no sufficient evidence.

When the evidence is missing

If the system finds no sufficient evidence for a statement, it leaves the statement out and says so. The user then sees where the sources do not hold.

Hallucinations cannot be ruled out completely. The system checks every sentence against the retrieved source sentences, and anything that fails the check does not appear in the answer.

Traceability down to the source sentence

Beyond checking the answer, faithfulness to the evidence can be anchored in the knowledge base itself, for example in a knowledge graph. Every stored relation carries two separate time axes there: since when a fact holds in the real world, and since when the system has known about it. Attached to each edge are the source sentence, the character position and the verbatim evidence text. Corrections never delete an earlier version. They close it and open a new one, so the old evidence stays traceable. Every statement in the system can be clicked and followed back to the exact sentence in the source document.

What this looks like in an archive system is shown by our product RETRAYA. Why this architecture also makes deletion requests easier is covered in the article AI and the GDPR in business.

Frequently asked questions

Can RAG rule out hallucinations completely?

RAG can reduce them strongly, and a small remainder stays. That works with hybrid search, reranking and a sentence-level check with a defined threshold, where “no sufficient evidence found” is a valid answer.

Why is pure vector search not enough?

It finds meaning well and finds exact terms such as names or file references poorly. Combining it with classic keyword search raises retrieval quality measurably compared with either method alone.

What happens when the system finds no evidence for a statement?

It omits the statement and names the missing evidence.

How does a sentence-level check differ from a general source citation?

A general citation shows roughly where an answer comes from. A sentence-level check with a defined threshold tests whether every single sentence is supported by a concrete passage.

Questions on evidence-based answers from your documents?

We show you how the sentence-by-sentence check works on your own records.