LLM-as-Judge Pitfalls: Position and Length Bias

Scenario

Example company: a team compares a new prompt (B) with the current prompt (A) using a model judge: "Which answer is better, Answer 1 or Answer 2?", always with A first and B second.

What you see in production

The new prompt wins only 35 % of comparisons and is rejected. A month later, someone runs the same comparison with the order reversed and the new prompt wins the majority. A second experiment, with a verbose prompt variant, "wins" clearly, although users prefer shorter answers.

The naive code

def compare(question, a, b):
    verdict = judge(f"Question: {question}\nAnswer 1: {a}\nAnswer 2: {b}\nWhich is better? Reply 1 or 2.")
    return "A" if verdict == "1" else "B"

Reproduce it

A simulated judge with a position preference and a length preference, evaluated naively and with order swapping:

import random
random.seed(5)

pairs = [(random.random(), random.random()) for _ in range(400)]       # true quality of A and B

def judge(q1, q2, len1=1.0, len2=1.0):
    score1 = q1 + 0.15 + 0.1 * (len1 - 1)                               # bonus for being shown first and for length
    score2 = q2 + 0.1 * (len2 - 1)
    return 1 if score1 >= score2 else 2

truth = sum(b > a for a, b in pairs) / len(pairs)
naive = sum(judge(a, b) == 2 for a, b in pairs) / len(pairs)
swapped_votes = [(judge(a, b) == 2) + (judge(b, a) == 1) for a, b in pairs]   # count B's wins in both orders
swapped = sum(v == 2 for v in swapped_votes) / len(pairs)
ties = sum(v == 1 for v in swapped_votes) / len(pairs)
a_wins = sum(v == 0 for v in swapped_votes) / len(pairs)
print(f"B truly better: {truth:.0%}")
print(f"naive (A always first): B wins {naive:.0%}")
print(f"both orders: B wins {swapped:.0%}, A wins {a_wins:.0%}, inconsistent (tie) {ties:.0%}")

B truly better: 51%
naive (A always first): B wins 35%
both orders: B wins 35%, A wins 36%, inconsistent (tie) 29%

Root cause

Model judges have systematic biases. Two well documented ones are position bias (preferring the answer in a particular position) and verbosity bias (preferring longer answers). If the evaluation always presents candidates in the same order and does not control for length, the bias becomes the result.

The fix

• Swap and aggregate: judge every pair in both orders; count a win only when both orders agree, otherwise a tie.
• Prefer absolute grading against criteria where possible (pass/fail per rubric item) over "which is better".
• Control for length: instruct the judge that length is not a quality signal, compare answers of similar length, or report win rates by length difference.
• Hide identities: never reveal which answer is the new version.
• Calibrate the judge against human preferences on a sample, as with any grader.

Trade-offs

• Judging both orders doubles cost; for large evaluations, swap on a random half and check that the result holds.
• Ties reduce statistical power; you may need more cases to reach a confident decision.
• Absolute rubrics miss holistic qualities (overall helpfulness) that pairwise comparison captures; use both deliberately.

Detect it in production

• Win rate of "Answer 1" across all comparisons: far from 50 % on random pairs indicates position bias.
• Correlation between answer length and judged quality, compared with human ratings.

Key points

• Pairwise model judges are biased by position and length.
• Judge both orders and treat disagreement as a tie.
• Prefer criteria-based grading, control length, and calibrate against humans.

Further reading: Judging LLM-as-a-judge with MT-Bench and Chatbot Arena · Length-controlled AlpacaEval