RAG that doesn't hallucinate: grounding, evals, and guardrails
Retrieval-augmented generation is only as trustworthy as its retrieval. A practical guide to building RAG that cites its sources and knows when to say 'I don't know.'
Retrieval-augmented generation (RAG) is the default architecture for putting a language model on top of your own data. It’s also where a lot of “AI answers” go wrong — confidently citing things that were never in the source material.
The fix isn’t a bigger model. It’s treating retrieval as the product and generation as the last step.
Garbage retrieval, garbage answer
A RAG system has two halves, and the first one does most of the work. If retrieval surfaces the wrong chunks, no amount of prompt engineering saves the answer — the model is faithfully summarizing the wrong context.
Most hallucination in production RAG traces back to retrieval, not generation:
- Chunks that split a fact across two pieces, so neither is retrievable
- Embeddings that match on vocabulary instead of meaning
- Stale content the pipeline never refreshed
- No re-ranking, so the best chunk sits at position 8 and never makes the cut
Building RAG you can trust
Ground every claim, and cite it
Answers should point back to the source passages they came from. Citations aren’t decoration — they’re how a user (and your eval harness) verifies the answer is real. If a claim has no supporting chunk, the system should say so rather than invent one.
Re-rank before you generate
Vector search gets you candidates; a re-ranking step gets you the right candidates in the right order. This one addition often does more for answer quality than swapping models.
Teach it to abstain
The most underrated feature in production RAG is a confident “I don’t know.” A system that declines to answer when the context doesn’t support one is far more valuable than one that always produces something.
Users forgive “I don’t have that information.” They don’t forgive a confident, wrong answer that cost them a decision.
Measure it like software
Build an eval set of real questions with known-good answers and citations. Score retrieval quality (did we fetch the right chunk?) separately from answer quality (did we use it correctly?). Now every change is measurable instead of vibes-based.
The payoff
Done right, RAG stops being a demo that impresses and becomes a system your team relies on — grounded, cited, monitored, and honest about its limits. That’s the difference between a chatbot and infrastructure.
If you’re building on your own knowledge base and want it to be trustworthy in production, we can help.