Sign in Get started

AI Engineering Tutorial · Quality and Production · Lesson 20 of 25

Evaluating Outputs

Measure quality with a small evaluation set and graders, instead of trying a few prompts by hand.

Create a free account to save this page, take notes and track your progress. Sign in

An evaluation set

Collect real examples of inputs with what a good answer must contain (or the exact expected output for classification). Start with 30 to 100; grow it with every bug you find.

Graders

  • Code graders for anything checkable: exact labels, valid JSON, required phrases, numbers.
  • Model graders with a clear rubric for qualities like tone or completeness, checked against human labels.

Run the set on every prompt or model change and compare per case, not just the average.

Score two prompt versions on the same casesTry it yourself
cases = [("I was charged twice", "billing"), ("App crashes on start", "technical"),
         ("Change my email", "account"), ("Refund please", "billing"), ("Can't log in", "account")]
prompt_v1 = {"I was charged twice": "billing", "App crashes on start": "technical", "Change my email": "billing",
             "Refund please": "billing", "Can't log in": "technical"}
prompt_v2 = {"I was charged twice": "billing", "App crashes on start": "technical", "Change my email": "account",
             "Refund please": "billing", "Can't log in": "account"}

for name, outputs in [("v1", prompt_v1), ("v2", prompt_v2)]:
    passed = [text for text, expected in cases if outputs[text] == expected]
    failed = [text for text, expected in cases if outputs[text] != expected]
    print(f"{name}: {len(passed)}/{len(cases)} correct; failing: {failed}")

Output

v1: 3/5 correct; failing: ['Change my email', "Can't log in"]
v2: 5/5 correct; failing: []

See Create strong empirical evaluations and the course LLM Evals and Guardrails.

Test yourself

Why compare results per case and not only the average score?

  1. Averages are always wrong
  2. A change can fix some cases and break others while the average barely moves
  3. Per-case results are cheaper
  4. Graders only work per case
Show answer

Answer: A change can fix some cases and break others while the average barely moves

Per-case comparison reveals regressions hidden in averages.

Was this page helpful?