The RAG assistant started confidently making things up

The system

• An internal documentation assistant: user question → embed → top-k from a vector index → stuff into a prompt → answer with citations.
• The corpus is ~40,000 internal documents, re-indexed weekly by a batch job.
• Chunking is fixed-size 512 tokens with 50-token overlap. k = 5.
• Quality is tracked by a thumbs-up/down widget and a weekly 50-question eval set scored by a human.

What you observe

• Weekly eval score dropped from 82% to 51% in one week.
• Thumbs-down rate roughly doubled.
• Answers are fluent and cite documents that exist — but the cited passage often does not support the claim.
• Retrieval latency and index size look normal.

Diagnostic order

1. 1

   Separate retrieval failure from generation failure.

   For each failing eval question, check whether the correct passage is in the retrieved top-k at all. If it is absent, this is a retrieval problem and no amount of prompt engineering fixes it. If it is present and the answer still contradicts it, the problem is in generation or prompt assembly.

2. 2

   Measure recall@k on a labelled question→passage set.

   You cannot debug retrieval by reading answers. Build (or reuse) a set of questions with the known correct passage and compute recall@5 before and after the re-index.

3. 3

   Diff the index build, not the model.

   Compare document count, chunk count, embedding model version, distance metric, and normalisation between the last good index and the current one. Re-index jobs are the most common source of silent regressions in RAG systems.

4. 4

   Inspect what the retriever returns for a failing query, by hand.

   Five minutes of looking at the actual retrieved chunks usually beats an hour of hypothesising. Look for truncated chunks, boilerplate headers, or chunks that are pure navigation text.

Root cause

The re-index picked up a new document exporter that wrapped every page in a large HTML navigation header. With fixed 512-token chunking, the first chunk of every document became almost entirely boilerplate, and because that boilerplate was near-identical across 40,000 documents, those chunks clustered tightly in embedding space and dominated retrieval for short, generic queries. Recall@5 for the eval set fell from 0.88 to 0.41. The generator was behaving correctly — asked to answer from five irrelevant chunks, it produced a plausible answer and cited them.

The fix

• Strip boilerplate at ingestion and re-index; recall@5 returned to 0.86.
• Move from fixed-size to structure-aware chunking that respects headings.
• Add a relevance floor: if the best retrieved score is below a threshold, answer 'I don't have that documented' rather than proceeding.
• Instruct the generator to answer only from the provided context and to say so when the context is insufficient — and test that it actually does.

What should have caught it

• Gate every index build on an automated recall@k check against a labelled set; block promotion on regression.
• Version indexes and keep the previous one live so rollback is a pointer change.
• Monitor the distribution of top-1 similarity scores — a sudden shift is a re-index problem, not a user problem.
• Track retrieval and generation quality as separate metrics; a single end-to-end score hides which half broke.

Follow-ups you should expect

• How do you build the labelled question→passage set cheaply?
• What would a hybrid keyword + vector retriever have done here?
• Is a reranker the right fix, or does it just mask bad candidates?
• How do you set the relevance floor without refusing to answer legitimate questions?