What do recall@k and MRR measure in RAG evaluation?

Recall@k is the share of questions whose gold passage appears in the top k results; MRR averages 1/rank of the first gold passage, rewarding systems that rank it higher.

Why should retrieval and generation be evaluated separately?

They fail for different reasons; separate metrics show whether a bad answer came from missing passages or from the model misusing good passages, which determines what to fix.

What is faithfulness in RAG evaluation?

Whether every claim in the answer is supported by the retrieved passages, regardless of whether it happens to be true elsewhere.

Measuring Retrieval and Answer Quality — RAG in Production

Scenario

Example company: a team keeps tweaking its RAG pipeline: new chunk sizes, a different embedding model, a reranker. Each change is judged by trying five favourite questions in the chat window.

What you see in production

A change that "looked better" on the five questions made dozens of other questions worse, which the team learns from support tickets two weeks later. Nobody can say whether retrieval or generation is at fault for a given bad answer.

The naive code

for q in ["reset password", "change plan", "export data", "delete account", "add user"]:
    print(q, "->", answer(q))                       # eyeball it, ship it

Reproduce it

Compute retrieval metrics for two pipeline configurations on a small labelled set:

recall@3 and MRR for two retrieval configurationsTry it yourself

# question -> id of the passage that answers it (labelled once, reused for every change)
gold = {"q1": "p7", "q2": "p2", "q3": "p9", "q4": "p4", "q5": "p1", "q6": "p6"}

runs = {
    "baseline": {"q1": ["p7", "p3", "p5"], "q2": ["p8", "p2", "p1"], "q3": ["p3", "p5", "p6"],
                 "q4": ["p4", "p1", "p2"], "q5": ["p5", "p6", "p1"], "q6": ["p2", "p3", "p4"]},
    "new-chunking": {"q1": ["p7", "p2", "p3"], "q2": ["p2", "p8", "p1"], "q3": ["p9", "p3", "p5"],
                     "q4": ["p1", "p2", "p3"], "q5": ["p1", "p5", "p6"], "q6": ["p3", "p4", "p2"]},
}

def recall_at_k(results, k=3):
    return sum(gold[q] in r[:k] for q, r in results.items()) / len(results)

def mrr(results):
    return sum(1 / (r.index(gold[q]) + 1) if gold[q] in r else 0 for q, r in results.items()) / len(results)

for name, results in runs.items():
    lost = [q for q in results if gold[q] in runs["baseline"][q] and gold[q] not in results[q]]
    print(f"{name:13} recall@3={recall_at_k(results):.2f}  MRR={mrr(results):.2f}  regressions={lost}")

# question -> id of the passage that answers it (labelled once, reused for every change)
gold = {"q1": "p7", "q2": "p2", "q3": "p9", "q4": "p4", "q5": "p1", "q6": "p6"}

runs = {
    "baseline": {"q1": ["p7", "p3", "p5"], "q2": ["p8", "p2", "p1"], "q3": ["p3", "p5", "p6"],
                 "q4": ["p4", "p1", "p2"], "q5": ["p5", "p6", "p1"], "q6": ["p2", "p3", "p4"]},
    "new-chunking": {"q1": ["p7", "p2", "p3"], "q2": ["p2", "p8", "p1"], "q3": ["p9", "p3", "p5"],
                     "q4": ["p1", "p2", "p3"], "q5": ["p1", "p5", "p6"], "q6": ["p3", "p4", "p2"]},
}

def recall_at_k(results, k=3):
    return sum(gold[q] in r[:k] for q, r in results.items()) / len(results)

def mrr(results):
    return sum(1 / (r.index(gold[q]) + 1) if gold[q] in r else 0 for q, r in results.items()) / len(results)

for name, results in runs.items():
    lost = [q for q in results if gold[q] in runs["baseline"][q] and gold[q] not in results[q]]
    print(f"{name:13} recall@3={recall_at_k(results):.2f}  MRR={mrr(results):.2f}  regressions={lost}")

Output

baseline      recall@3=0.67  MRR=0.47  regressions=[]
new-chunking  recall@3=0.67  MRR=0.67  regressions=['q4']

Root cause

RAG quality has two parts that fail independently: retrieval (did the right passage reach the model?) and generation (did the model answer correctly and faithfully from it?). Trying a handful of questions measures neither, and averages hide regressions on specific questions.

The fix

Build an evaluation set and run it on every change:

Collect 100 to 300 real questions from logs and support tickets, covering common and difficult types.
Label the gold passages (which chunks answer each question) and a reference answer or key facts.
Retrieval metrics: recall@k (is a gold passage in the top k?) and MRR (how high is it?).
Answer metrics: correctness against the reference, faithfulness (every claim supported by the retrieved text) and citation validity, graded by a rubric with a model grader and checked against human labels.
Compare per question, not just averages; list regressions.
Keep a production feedback loop: thumbs-down answers become new evaluation cases.

Trade-offs

Labelling takes hours up front; it pays for itself the first time it catches a regression.
Model graders are fast but biased; calibrate them against a human-labelled sample and re-check periodically.
An evaluation set goes stale as content changes; review it when documents are reorganised.

Detect it in production

Evaluation results stored per pipeline version, with a dashboard of trends.
CI gate: fail the change if recall or faithfulness drops beyond a threshold.
Production signals: "not found" rate, negative feedback rate, escalations to humans.

Key points

Evaluate retrieval and generation separately.
Use a labelled set of real questions with recall@k, MRR and faithfulness.
Look at per-question regressions, not only averages, and gate changes on them.

Further reading: Evaluation measures in information retrieval · Mean reciprocal rank · Ragas metrics · Create strong empirical evaluations