Your RAG Isn't Hallucinating — It's Retrieving Garbage

Most 'hallucination' bugs are retrieval bugs. The five failures we fix most often — bad chunking, no reranking, stale indexes, query mismatch, missing eval loop — and how to diagnose which one you have.

A model can only be as right as the text you hand it. When a RAG system makes something up, the reflex is to blame the LLM — swap the model, drop the temperature, add "do not hallucinate" to the prompt. Most of the time that changes nothing, because the model never saw the correct answer. It answered from whatever your retriever pulled, and your retriever pulled the wrong chunks.

We debug these systems often, and the pattern repeats. Print the retrieved context before it reaches the prompt and read it yourself. In the majority of "hallucination" tickets, the answer simply isn't in there. Once you accept that, the problem stops being mysterious and becomes five concrete, fixable failures.

Read the context before you blame the model

Before touching anything, log the exact chunks passed to the LLM for the question that failed. There are two outcomes. If the correct passage is present and the model still got it wrong, you have a generation problem: prompt, context window, or model. If it's absent, stop — every prompt tweak from here is wasted time. That single check routes you to the right half of the system and saves days.

• The answer exists in your source docs but never appears in the retrieved chunks.
• Retrieved chunks are on-topic but cut off mid-sentence or missing the key number or table.
• The same query returns different chunks after a re-index.

Chunking is where retrieval quality is won or lost

Naive fixed-size splitting (say 512 tokens, no overlap) cuts tables in half, separates a heading from its paragraph, and strands the number you need one chunk away from the sentence that names it. Embeddings of half-thoughts retrieve badly. Fix the chunk boundaries before you touch the model.

• Split on structure (headings, sections, markdown), not a raw token count.
• Keep 10-20% overlap so a fact near a boundary survives in both neighbors.
• Keep tables, code blocks, and lists intact as a single chunk.
• Prepend the section or document title to each chunk so it carries context.
• Typical useful size: 200-500 tokens for prose, larger for structured data.

Vector search gets you close; a reranker gets you right

Bare vector similarity (top-k by cosine distance) is a coarse filter. It reliably lands the correct passage in the top 20, and just as reliably ranks it at position 8 instead of 1. If you feed the top 3 to the model, you dropped it. A cross-encoder reranker rescores query-document pairs and reorders them, usually lifting the correct passage into the first few slots.

• Retrieve wide (top 20-50 from the vector store), rerank, then keep the top 3-5.
• Use a cross-encoder reranker (Cohere Rerank, bge-reranker, or similar).
• Expect a real jump in answer quality; the cost is one extra call of 50-200ms.
• If reranking changes nothing, your problem is recall — look at chunking and embeddings, not ranking.

A stale index answers yesterday's questions

The retriever only knows what you embedded. Docs change, prices change, policies change; if your pipeline doesn't re-embed on update, the system confidently serves last quarter's answer. To the user this looks exactly like a hallucination — the fact is wrong — but the model is faithfully repeating stale text.

• Re-embed when a document changes, not on a monthly cron you forgot to run.
• Store a content hash and updated-at per chunk; skip what's unchanged, refresh the rest.
• Delete embeddings for removed documents — orphaned vectors keep getting retrieved.
• When you switch embedding models, re-embed everything; vectors from two models don't compare.

The user's words and the document's words rarely match

A user asks "why was my card declined?" The policy doc says "authorization failure due to insufficient funds." Different vocabulary, low similarity, wrong retrieval. The embedding space doesn't magically bridge that gap on short or jargon-heavy queries. You have to help the query meet the document.

• Query rewriting: have a cheap LLM expand or rephrase the question before embedding it.
• Hybrid search: combine dense vectors with BM25/keyword so exact terms (IDs, error codes, SKUs) still hit.
• HyDE: generate a hypothetical answer and embed that — it often sits closer to the real doc than the question does.
• Multi-query: fire 2-3 phrasings and merge the results before reranking.

Without evals you're tuning blind

Every change above trades off against the others. Bigger chunks help recall and hurt precision. Aggressive rewriting helps some queries and drifts others. Without measurement you're guessing, and you'll "fix" one query while quietly breaking ten. Build the eval set before you optimize.

• Assemble 50-200 real question/answer pairs from actual usage, not invented ones.
• Measure retrieval directly: recall@k and MRR — was the right chunk retrieved at all?
• Score answers with an LLM-judge on faithfulness and relevance, on every change.
• Wire it into CI so a chunking or prompt change can't silently regress retrieval.

Diagnose in order: context, then chunks, then ranking

The next time someone says the model is hallucinating, don't reach for a bigger model. Log the retrieved context and read it. An absent answer means retrieval; walk the five in order — chunking, then reranking, then index freshness, then query matching — and gate every change behind an eval set. A wrong answer with the correct context already in hand is the only case that's genuinely a generation problem. Most of the time you'll never get there.