Why can a RAG answer be wrong even when the right document was retrieved?

The model can still answer from its training data, for example an outdated limit seen many times on the web, because nothing forces it to use only the retrieved text.

How do you verify citations in a RAG answer?

Ask for a source ID and an exact quote for each claim, then check in code that each quote appears in the cited source; reject or regenerate answers that fail.

Why is 'I can't find that' a good outcome to measure rather than avoid?

It is the correct answer when retrieval misses, it prevents confident wrong answers, and its rate shows where retrieval or content needs improving.

Grounded Answers and Verified Citations — RAG in Production

Scenario

Example company: an insurance company's assistant answers policy questions from the current policy documents and shows "Sources" under each answer.

What you see in production

A customer is told their policy covers water damage from burst pipes up to 10,000. The current policy says 5,000; 10,000 was the limit two versions ago and appears in older public material. The answer listed a source, the right policy document, but the number came from the model's memory, not from the retrieved text.

The naive code

prompt = f"Answer the question using these documents.\n\n{documents}\n\nQuestion: {question}"
answer = llm(prompt)
sources = [d.title for d in retrieved]               # listed whether or not they were used

Reproduce it

Ask for answers with quoted evidence, then check each quote really appears in the retrieved text:

Verify that every cited quote exists in its sourceTry it yourself

sources = {
    "policy-2026 s4.2": "Water damage from burst pipes is covered up to 5,000 per claim, after the deductible.",
    "policy-2026 s4.3": "Gradual leaks and damp are not covered.",
}

def verify(answer):
    problems = []
    for claim in answer["claims"]:
        text = sources.get(claim["source"])
        if text is None:
            problems.append(f"unknown source {claim['source']}")
        elif claim["quote"] not in text:
            problems.append(f"quote not found in {claim['source']}: {claim['quote']!r}")
    return "verified" if not problems else "REJECT: " + "; ".join(problems)

from_memory = {"text": "Burst pipes are covered up to 10,000.",
               "claims": [{"source": "policy-2026 s4.2", "quote": "covered up to 10,000"}]}
grounded = {"text": "Burst pipes are covered up to 5,000 per claim after the deductible; gradual leaks are not covered.",
            "claims": [{"source": "policy-2026 s4.2", "quote": "covered up to 5,000 per claim"},
                       {"source": "policy-2026 s4.3", "quote": "Gradual leaks and damp are not covered."}]}
print("answer from memory:", verify(from_memory))
print("grounded answer:   ", verify(grounded))

sources = {
    "policy-2026 s4.2": "Water damage from burst pipes is covered up to 5,000 per claim, after the deductible.",
    "policy-2026 s4.3": "Gradual leaks and damp are not covered.",
}

def verify(answer):
    problems = []
    for claim in answer["claims"]:
        text = sources.get(claim["source"])
        if text is None:
            problems.append(f"unknown source {claim['source']}")
        elif claim["quote"] not in text:
            problems.append(f"quote not found in {claim['source']}: {claim['quote']!r}")
    return "verified" if not problems else "REJECT: " + "; ".join(problems)

from_memory = {"text": "Burst pipes are covered up to 10,000.",
               "claims": [{"source": "policy-2026 s4.2", "quote": "covered up to 10,000"}]}
grounded = {"text": "Burst pipes are covered up to 5,000 per claim after the deductible; gradual leaks are not covered.",
            "claims": [{"source": "policy-2026 s4.2", "quote": "covered up to 5,000 per claim"},
                       {"source": "policy-2026 s4.3", "quote": "Gradual leaks and damp are not covered."}]}
print("answer from memory:", verify(from_memory))
print("grounded answer:   ", verify(grounded))

Output

answer from memory: REJECT: quote not found in policy-2026 s4.2: 'covered up to 10,000'
grounded answer:    verified

Root cause

Retrieved documents are just more text in the prompt; nothing forces the model to prefer them over what it learned in training, especially for facts that appear often on the web. Listing retrieved documents as "sources" does not show which of them, if any, supports each statement.

The fix

Instruct clearly: answer only from the provided documents; if they do not contain the answer, say so; never use outside knowledge for facts such as limits, prices or dates.
Require evidence: ask for each claim's source ID and a short exact quote, in structured output, or use a citations feature that returns cited spans.
Verify in code: check that every quote appears in the cited source; reject or regenerate otherwise, and show users only verified citations.
Prefer "I don't know" to a guess, and make that outcome visible in metrics.

response = client.messages.create(
    model=MODEL, max_tokens=800,
    system="Answer only from the documents. Quote the exact sentence that supports each claim. "
           "If the documents don't answer the question, say you can't find it.",
    messages=[{"role": "user", "content": [*document_blocks, {"type": "text", "text": question}]}],
)

Trade-offs

Strict grounding increases "I can't find that" answers; that is correct behaviour when retrieval misses, and a signal to improve retrieval.
Exact-quote checks fail on harmless paraphrases; ask for verbatim quotes, or compare normalised text.
Verification catches fabricated quotes, not quotes taken out of context; sample answers for human review too.

Detect it in production

Share of answers with all citations verified, unverified citations, and "not found" answers.
Human review of a weekly sample, focusing on numbers, dates and limits.
User feedback ("this is wrong") linked to the answer and its sources.

Key points

Retrieved text does not automatically override what the model already "knows".
Require a source and an exact quote for each claim, and verify quotes in code.
Make "I can't find that" an acceptable, measured outcome.

Further reading: Citations · Reduce hallucinations