Evaluation interview question · Most asked
How do you decide whether a prompt change is an improvement?
Summary
Run the eval set on both versions and compare per case and per type, not only the average; report confidence intervals (and repeat noisy cases); review every newly failing case; apply stricter rules for critical types; then confirm with a canary or shadow test on live traffic.
Longer Answer
A change can break one category completely while the average barely moves. With a few hundred cases, a 2-point difference may be noise.
Key Points
- Per-case and per-type comparison.
- Confidence intervals.
- Review new failures.
- Confirm with canary or shadow traffic.
Deep dive: Evals in CI
More most-asked Evaluation questions
- Evaluation leaf: What is a golden dataset?
- When do you use code graders, model graders or human graders?
- What guardrails would you put around an LLM feature?
- How do you build an evaluation set for an LLM feature?
- Evaluation leaf: What is faithfulness/groundedness?
- Evaluation leaf: Pros/cons of LLM-as-judge?
- Evaluation leaf: Offline vs online evaluation tradeoffs?