Code Graders, Model Graders and Humans

Scenario

Example company: a team grades its support assistant by asking a model "Is this a good answer? Reply yes or no." for every eval case.

What you see in production

The grader says yes to 90 % of answers. Spot checks by support leads find many of those answers wrong on details (wrong fee amounts, a missing step). Meanwhile, a strict string comparison tried earlier failed almost every answer, even good ones, because wording varies.

The naive code

def grade(answer):
    return llm(f"Is this a good answer? Reply yes or no.\n\n{answer}") == "yes"

Reproduce it

Measure how well two graders agree with human labels, using Cohen's kappa:

human = [1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 1]
vague = [1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1]   # says "good" to almost everything
rubric = [1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1]

def kappa(a, b):
    n = len(a)
    observed = sum(x == y for x, y in zip(a, b)) / n
    pa, pb = sum(a) / n, sum(b) / n
    expected = pa * pb + (1 - pa) * (1 - pb)
    return (observed - expected) / (1 - expected)

for name, grades in [("vague yes/no", vague), ("rubric", rubric)]:
    agree = sum(g == h for g, h in zip(grades, human)) / len(human)
    print(f"{name:12} pass rate={sum(grades) / len(grades):.0%}  agreement={agree:.0%}  kappa={kappa(grades, human):.2f}")

vague yes/no pass rate=90%  agreement=65%  kappa=0.24
rubric       pass rate=55%  agreement=90%  kappa=0.80

Root cause

A vague question gets a lenient answer: models tend to rate fluent text as good. Exact string matching has the opposite problem. The real issue is that different criteria need different graders, and a model grader is itself a model whose accuracy must be measured.

The fix

Choose the grader per criterion, from cheapest and most reliable to most flexible:

• Code graders for anything checkable: JSON validity, required fields, the right tool called, numbers within tolerance, forbidden phrases, regular expressions for formats.
• Model graders with a rubric for qualities code cannot check: does the answer state the 24-hour rule? Is the tone appropriate? One criterion per question, with a short reasoning step and a fixed output (PASS / FAIL, or a 1 to 5 scale with definitions).
• Humans for calibration and for the hardest cases: label a sample, measure the model grader's agreement (accuracy, Cohen's kappa) and revise the rubric until agreement is high.

You are grading a support answer against one criterion.
Criterion: The answer states that cancellations within 24 hours of booking are free.
Answer: <answer>...</answer>
Think briefly, then output exactly PASS or FAIL on the last line.

Trade-offs

• Model graders cost money and add variance; use code graders wherever they suffice.
• Rubrics need maintenance when policies change.
• High agreement on one dataset does not transfer automatically; re-check calibration when the product or grader model changes.

Detect it in production

• Grader agreement with human labels, re-measured monthly on a fresh sample.
• Pass rates that are suspiciously high (above 95 %) or that never change: likely a lenient grader.

Key points

• Use code graders for checkable criteria, rubric-based model graders for the rest, humans for calibration.
• One criterion per model-grader question, with a fixed output format.
• Measure grader agreement with humans before trusting its numbers.

Further reading: Create strong empirical evaluations: grading methods · Cohen's kappa · Judging LLM-as-a-judge