Interview Q&A · 39 questions · 8 answered publicly
Evaluation Interview Questions
Ordered by how often they're asked. Sign in free to see every answer, rate your confidence and run timed mock interviews.
-
Most asked Evaluation leaf: What is a golden dataset?
Summary
Curated inputs with expected outputs/docs, versioned, covering edge cases.
Longer Answer
Curated inputs with expected outputs/docs, versioned, covering edge cases. Used for offline regression in CI.
Key Points
- Curated inputs with expected outputs/docs, versioned, covering edge cases
- Used for offline regression in CI
- Foundation
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: What is a golden dataset?'
Comparison / Tradeoffs
Aspect Notes Core idea Curated inputs with expected outputs/docs, versioned, covering edge cases. Practical tradeoff Foundation. -
Most asked Evaluation leaf: Pros/cons of LLM-as-judge?
Summary
Scales grading with rubrics but can be biased/noisy.
Longer Answer
Scales grading with rubrics but can be biased/noisy. Calibrate with humans and use pairwise comparisons carefully.
Key Points
- Scales grading with rubrics but can be biased/noisy
- Calibrate with humans and use pairwise comparisons carefully
- Judging
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: Pros/cons of LLM-as-judge?'
Comparison / Tradeoffs
Aspect Notes Core idea Scales grading with rubrics but can be biased/noisy. Practical tradeoff Judging. -
Most asked Evaluation leaf: What is faithfulness/groundedness?
Summary
Claims in the answer are supported by retrieved context.
Longer Answer
Claims in the answer are supported by retrieved context. Unfaithful fluent answers are still failures.
Key Points
- Claims in the answer are supported by retrieved context
- Unfaithful fluent answers are still failures
- Quality dimension
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: What is faithfulness/groundedness?'
Comparison / Tradeoffs
Aspect Notes Core idea Claims in the answer are supported by retrieved context. Practical tradeoff Quality dimension. -
Most asked How do you build an evaluation set for an LLM feature?
Summary
Start from real traffic (logs, tickets), remove personal data, classify cases by type (intent, language, difficulty, risk), sample a minimum per type (stratified), write explicit success criteria per case (must include, must not include, expected tool call or refusal), add edge and adversarial cases, version it, and add every production failure.
Longer Answer
Hand-written prompts reflect imagination, not usage; random samples miss rare but important types. 50 to 100 well-labelled cases beat thousands of unlabelled ones.
Key Points
- Real traffic, anonymised.
- Stratified by type.
- Explicit success criteria.
- Grows with production failures.
Deep dive: From Vibes to an Eval Set
-
Most asked When do you use code graders, model graders or human graders?
Summary
Code graders for anything checkable (valid JSON, labels, tool calls, numbers, required or forbidden phrases), model graders with a specific rubric for qualities code can't check (correctness against a reference, tone, completeness), and humans to calibrate model graders and judge the hardest cases.
Longer Answer
Measure a model grader's agreement with human labels (accuracy, Cohen's kappa) before trusting it, and re-check periodically. One criterion per grader prompt with a fixed output format.
Key Points
- Code graders first.
- Rubric-based model graders.
- Humans calibrate.
- Measure grader agreement.
Deep dive: Code Graders, Model Graders and Humans
-
Most asked How do you decide whether a prompt change is an improvement?
Summary
Run the eval set on both versions and compare per case and per type, not only the average; report confidence intervals (and repeat noisy cases); review every newly failing case; apply stricter rules for critical types; then confirm with a canary or shadow test on live traffic.
Longer Answer
A change can break one category completely while the average barely moves. With a few hundred cases, a 2-point difference may be noise.
Key Points
- Per-case and per-type comparison.
- Confidence intervals.
- Review new failures.
- Confirm with canary or shadow traffic.
Deep dive: Evals in CI
-
Most asked What guardrails would you put around an LLM feature?
Summary
Input: keep secrets out of prompts, screen for injection and abuse, treat retrieved and tool content as untrusted. Capabilities: least-privilege tools with authorisation in code and approvals for consequential actions. Output: schema and content validation, policy checks, personal-data checks, escaping for the destination. Plus monitoring and rate limits.
Longer Answer
Each layer can fail, so they are combined. Measure guardrail errors in both directions: harmful content allowed and legitimate requests refused.
Key Points
- Input checks.
- Least-privilege capabilities.
- Output validation.
- Measure both error types.
Deep dive: LLM Evals and Guardrails
-
Most asked Evaluation leaf: Offline vs online evaluation tradeoffs?
Summary
Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both.
Longer Answer
Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both. Eval strategy.
Key Points
- Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both
- Eval strategy
- Relevant areas: Ai, Evaluation
- Use the answer to connect the concept back to a realistic production or interview scenario for 'Evaluation leaf: Offline vs online evaluation tradeoffs?'
Comparison / Tradeoffs
Aspect Notes Core idea Offline is fast/safe but may miss distribution shift; online is real but slower/riskier-use both. Practical tradeoff Eval strategy. - Evaluation leaf: Why separate retrieval metrics from generation metrics?
- Evaluation leaf: Online metrics that matter?
- Evaluation leaf: Red teaming for LLM apps?
- Evaluation leaf: How do you detect regressions after a prompt change?
- Evaluation leaf: How do you evaluate agents?
- Evaluation leaf: How do you sample production traffic for review?
- Evaluation leaf: What is answer relevance?
- Evaluation leaf: How do you evaluate citation quality?
- Evaluation leaf: How do toxicity/safety classifiers fit?
- Evaluation leaf: What is calibration of confidence?
- Evaluation leaf: CI gate example for RAG PRs?
- What biases do LLM judges have, and how do you mitigate them?
- How do you monitor LLM quality in production?
- How do you roll out a prompt or model change safely?
- Why is model output untrusted input, and what does that mean in practice?
- How do you handle personal data in LLM features?
- How would you red-team an LLM feature?
- Evaluation leaf: Inter-annotator agreement-why care?
- Evaluation leaf: Statistical significance in A/B for LLM features?
- Evaluation leaf: Synthetic eval data-when OK?
- Evaluation leaf: Tracking eval debt?
- Evaluation leaf: Multimodal eval challenges?
- How do you balance safety filtering against over-refusal?
- How do you handle non-determinism in LLM evaluations?
- What is faithfulness, and how is it different from correctness?
- How do you write a good rubric for a model grader?
- A safety eval passes at 99%. Is the feature safe to launch?
- What metrics would you track for an LLM support assistant?
- How do you evaluate an LLM feature before you have real traffic?
- How would you evaluate a production RAG or agent system beyond 'it looks good'?
- What guardrails would you put around an LLM-powered customer-facing feature?