RAG interview question · Most asked
How do you evaluate a RAG system?
Summary
Separately for retrieval and generation, on a labelled set of real questions: retrieval with recall@k and MRR against gold passages; generation with correctness against key facts, faithfulness (claims supported by retrieved text) and citation validity, using code graders and calibrated model graders. Compare per question to find regressions, and gate changes in CI.
Longer Answer
Production signals (negative feedback, 'not found' rate, escalations) feed new cases into the set.
Key Points
- Retrieval: recall@k, MRR.
- Generation: correctness, faithfulness, citations.
- Per-question comparison.
- CI gates and production feedback.
Deep dive: Measuring Retrieval and Answer Quality
More most-asked RAG questions
- RAG leaf: Explain query rewriting and multi-query retrieval.
- Why use hybrid search in RAG, and how do you combine the results?
- How do you enforce permissions and tenant isolation in RAG?
- How do you make RAG answers grounded and verifiable?
- RAG leaf: How do you measure retrieval quality independently of generation?
- RAG leaf: How do you prevent prompt injection via retrieved documents?
- How do you choose a chunking strategy for RAG?
- RAG leaf: How do you evaluate faithfulness of answers?