Run an eval set with pytest

Approach

Keep cases in a JSONL file with the input and success criteria. A parametrised pytest test calls your feature for each case and applies code graders (expected label, required and forbidden phrases). Run it in CI on prompt and model changes.

Solution

import json
import pathlib

import pytest

from app.support import classify_ticket, draft_reply      # the feature under test

CASES = [json.loads(line) for line in pathlib.Path("evals/support_cases.jsonl").read_text().splitlines() if line]


@pytest.mark.parametrize("case", CASES, ids=[c["id"] for c in CASES])
def test_case(case):
    label = classify_ticket(case["input"])
    assert label == case["expected_label"], f"{case['id']}: got {label}"
    reply = draft_reply(case["input"]).lower()
    for phrase in case.get("must_include", []):
        assert phrase.lower() in reply, f"{case['id']}: missing {phrase!r}"
    for phrase in case.get("must_not_include", []):
        assert phrase.lower() not in reply, f"{case['id']}: contains {phrase!r}"

# evals/support_cases.jsonl, one case per line:
# {"id": "refund-017", "input": "I was charged twice...", "expected_label": "billing",
#  "must_include": ["refund"], "must_not_include": ["guaranteed"]}

Pitfalls

• Asserting exact full-text matches for generated text; check criteria instead.
• Running the whole suite on every commit when it's slow and costly; run a fast subset per commit, all before release.
• Ignoring flaky cases; run them several times and assert a pass rate.

Variations

• Add a model-graded rubric test for qualities code can't check, calibrated against human labels.
• Write results per case to a file and diff them between branches.

Further reading: Create strong empirical evaluations