How do you build an evaluation set for an LLM feature?

Summary

Start from real traffic (logs, tickets), remove personal data, classify cases by type (intent, language, difficulty, risk), sample a minimum per type (stratified), write explicit success criteria per case (must include, must not include, expected tool call or refusal), add edge and adversarial cases, version it, and add every production failure.

Longer Answer

Hand-written prompts reflect imagination, not usage; random samples miss rare but important types. 50 to 100 well-labelled cases beat thousands of unlabelled ones.

Key Points

• Real traffic, anonymised.
• Stratified by type.
• Explicit success criteria.
• Grows with production failures.

Deep dive: From Vibes to an Eval Set