From Vibes to an Eval Set

Scenario

Example company: a travel company's assistant answers booking, refund and visa questions. The team's "test set" is 20 prompts the product manager wrote at launch.

What you see in production

The test set passes at 95 %, yet complaints rise. Analysis of a week of real traffic shows that 28 % of questions are about refunds and cancellations, a type that appears only once in the test set, and that many users write in Spanish, which the test set does not cover at all.

The naive code

TEST_PROMPTS = [
    "How do I book a flight?", "What is your baggage policy?",  # ... 18 more written at launch
]

Reproduce it

Compare category coverage of the hand-written set, a random sample and a stratified sample of real traffic:

import random
from collections import Counter
random.seed(11)

traffic = (["booking"] * 380 + ["refund"] * 280 + ["visa"] * 120 + ["baggage"] * 150 +
           ["spanish-booking"] * 60 + ["complaint"] * 10)
random.shuffle(traffic)

hand_written = ["booking"] * 12 + ["baggage"] * 6 + ["visa"] * 1 + ["refund"] * 1

def stratified(population, per_type=8):
    out = []
    for t in sorted(set(population)):
        items = [x for x in population if x == t]
        out += random.sample(items, min(per_type, len(items)))
    return out

for name, sample in [("hand-written", hand_written), ("random 40", random.sample(traffic, 40)),
                     ("stratified 48", stratified(traffic))]:
    counts = Counter(sample)
    missing = sorted(set(traffic) - set(counts))
    print(f"{name:13} {dict(sorted(counts.items()))}  missing: {missing}")

hand-written  {'baggage': 6, 'booking': 12, 'refund': 1, 'visa': 1}  missing: ['complaint', 'spanish-booking']
random 40     {'baggage': 3, 'booking': 14, 'refund': 13, 'spanish-booking': 4, 'visa': 6}  missing: ['complaint']
stratified 48 {'baggage': 8, 'booking': 8, 'complaint': 8, 'refund': 8, 'spanish-booking': 8, 'visa': 8}  missing: []

Root cause

An eval set is only as useful as its resemblance to real use. Hand-written prompts reflect what the team imagines, not what users do; a pure random sample under-represents rare but important types (complaints, other languages). And without a definition of a good answer for each case, "passing" is a matter of opinion.

The fix

• Start from real traffic (logs, support tickets, search queries), with personal data removed.
• Classify cases into types (intent, language, difficulty, risk) and stratify: a minimum number of cases per type, more for high-risk types.
• Write success criteria per case: the key facts that must appear, things that must not appear, the expected tool call or refusal.
• Include edge cases and adversarial cases: ambiguous requests, out-of-scope questions, policy violations.
• Version the set in the repository, and add every production failure as a new case.
• Start small (50 to 100 cases) and grow; a small, good set beats a large, unlabelled one.

{"id": "refund-017", "type": "refund", "lang": "es", "input": "Quiero cancelar mi vuelo de mañana...",
 "must_include": ["24 hours", "cancellation fee"], "must_not_include": ["full refund guaranteed"],
 "expected_tool": "get_booking"}

Trade-offs

• Labelling criteria takes expert time; start with the highest-risk types.
• Real traffic contains personal data; anonymise before it enters the eval set.
• An eval set reflects past traffic; review its distribution when the product changes.

Detect it in production

• Compare the type distribution of production traffic with the eval set monthly.
• Track how many production failures were already covered by an eval case (coverage of failures).

Key points

• Build eval sets from real traffic, not from imagination.
• Stratify by type so rare but important cases are covered.
• Give every case explicit success criteria, and add each production failure as a case.

Further reading: Define your success criteria · Create strong empirical evaluations