Compare two prompt versions with an unbiased model judge

Approach

Ask a judge model to compare answers A and B for each case against a rubric, in both orders. Count a win only when both verdicts agree; disagreement is a tie. Hide which answer is new.

Solution

import anthropic

client = anthropic.Anthropic()
RUBRIC = "Correctness against the policy, completeness, and concision. Length alone is not a quality."


def verdict(question: str, first: str, second: str) -> str:
    response = client.messages.create(
        model="claude-sonnet-5-5", max_tokens=300,
        system=f"You compare two support answers. Criteria: {RUBRIC} "
               "Think briefly, then end with a final line containing exactly FIRST, SECOND or TIE.",
        messages=[{"role": "user", "content": f"<question>{question}</question>\n"
                                              f"<answer_1>{first}</answer_1>\n<answer_2>{second}</answer_2>"}],
    )
    return response.content[0].text.strip().splitlines()[-1].strip()


def compare(question: str, old: str, new: str) -> str:
    a = verdict(question, old, new)          # old shown first
    b = verdict(question, new, old)          # new shown first
    if a == "SECOND" and b == "FIRST":
        return "new"
    if a == "FIRST" and b == "SECOND":
        return "old"
    return "tie"


results = [compare(c["question"], c["old_answer"], c["new_answer"]) for c in load_cases()]
print({k: results.count(k) for k in ("new", "old", "tie")})

Pitfalls

• Always showing the current version first: position bias becomes the result.
• Letting longer answers win by default; say in the rubric that length isn't quality.
• Trusting the judge without checking its agreement with human preferences on a sample.

Variations

• Grade each answer separately against pass/fail criteria when possible; it's cheaper to interpret.
• Report win rates by question type, not only overall.

Further reading: Judging LLM-as-a-judge