LLM-as-Judge

LLM-as-Judge is a harness pattern that assigns the evaluator role to a model rather than a human: a judge LLM receives the output of another model or agent — alone, against a competing output, or against a reference answer — and returns a structured verdict such as a preference, a score, or a pass/fail with feedback. The pattern was named and systematically studied in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena as a scalable proxy for human preference on open-ended outputs that closed-ended benchmarks cannot grade. Two roles define the minimal form: a generator that produces candidate work, and a judge that never produces the deliverable itself — it only emits verdicts, which the harness consumes as a metric, a gate, generator feedback, or a training signal.

Problem and context

Most of what an agent team produces — dialogue, plans, research syntheses, review commentary, translations — has no mechanical correctness check. Closed-ended benchmarks (multiple-choice knowledge, short-answer retrieval) measure core capability but not whether an answer follows instructions or matches what a user actually prefers; human annotation measures exactly that but is too slow and costly to run at harness cadence (Zheng et al.).

The pattern rests on one empirical result: on MT-Bench, GPT-4's verdicts agree with human experts in 85% of non-tie cases — higher than the 81% agreement between the human annotators themselves (as of the paper's 2023-12-24 arXiv revision, Zheng et al.). The judge does not need to be infallible; it needs to be no noisier than the human evaluation it replaces. That bound is also the ceiling: the judge imports every bias listed below, and the harness design has to keep its verdicts inside the human-noise envelope.

Structure

The minimal data flow: a generator produces one or more candidates; the judge receives the task, the candidate(s), and optionally a reference; it emits a parsable verdict; a consumer turns verdicts into a metric, a merge gate, feedback, or a reward signal.

Zheng et al. formalize three grading protocols, usable independently or in combination (as of the 2023-12-24 revision):

Protocol	Judge input	Verdict	Failure surface
Pairwise comparison	task + two candidates	A / B / tie	position bias; comparisons grow quadratically with candidates
Single-answer grading	task + one candidate	score (e.g. 1–10)	absolute scale drifts across judge versions
Reference-guided	task + candidate + gold answer	score / verdict	requires a reference; on a 10-question math probe it cuts GPT-4's grading failure rate from 70% to 15%

Reference-guided grading works by generating the judge's own answer independently first and placing it into the judge's context-window as a reference before grading — the mode Zheng et al. recommend for math and reasoning content.

Production-grade judges add three structural components on top of the bare verdict prompt:

• Order symmetry. For pairwise verdicts, judge each pair twice with positions swapped and count a win only when both orders agree, otherwise tie (Zheng et al.); AlpacaEval randomizes candidate order for the same reason.
• Reason before verdict. OpenAI's Evals framework wraps the completion in a second evaluation prompt designed to be parsable and recommends cot_classify — chain-of-thought first, classification last — as the default for model-graded evals (eval templates).
• Grounding. Claude Code's Code Review runs specialized agents over a PR diff in parallel, then "a verification step checks candidates against actual code behavior to filter out false positives" before findings are deduplicated, severity-ranked, and posted (Code Review docs, as of 2026-07-10) — the judge's raw claims are checked against reality, not trusted directly.

Where the judge sits in the harness determines what it reads and what its verdicts feed:

Placement	Judge reads	Verdict feeds	Example
Offline evaluation	final outputs	leaderboard / regression metric	MT-Bench, AlpacaEval 2.0
Inference-time loop	draft output	feedback to the generator, looped until accepted	evaluator-optimizer workflow (Anthropic)
Trajectory evaluation	full step-by-step trajectory	per-requirement verdicts	Agent-as-a-Judge (Zhuge et al.)
Training time	pairs of sampled responses	preference/reward model for RL	RLAIF in Constitutional AI

The last two rows are easy to conflate: in rlaif the judge's preferences are distilled into a reward model before deployment, while an evaluator-optimizer judge runs live inside every request.

When to use / When not to

Use it when no mechanical check exists but the criteria are articulable. Anthropic's fit test for the evaluator-optimizer loop: clear evaluation criteria plus measurable value from iteration — concretely, when a human articulating feedback would demonstrably improve the response and an LLM can generate that same kind of feedback (Building effective agents). Their examples: literary translation nuance, and deciding whether another search round is warranted in multi-round research.

Use it when evaluation must scale. A full AlpacaEval 2.0 run — pairwise judgments against a fixed GPT-4-turbo baseline — costs under $10 and finishes in under 3 minutes (repo docs, as of 2026-07-10). Agent-as-a-Judge reports saving 97.72% of evaluation time and 97.64% of cost versus three human experts across DevAI's 55 development tasks (Zhuge et al., as of 2024-10).

Don't use it where ground truth is mechanical. swe-bench — the canonical coding-agent benchmark — grades by applying the generated patch to the real repository and running the repository's own tests in a container (evaluation guide); no judge opines on patch quality across its 2,294 real GitHub issues (Jimenez et al.). If a test suite, compiler, or type checker can answer the question, it is cheaper and immune to the biases below; reserve the judge for what tests cannot reach — clarity, design, requirement fit.

Don't optimize hard against an unguarded judge. The moment judge verdicts become an optimization target — RL reward, selection pressure among agents, leaderboard incentive — Goodhart dynamics apply (see failure modes).

Trade-offs and failure modes

The three measured biases. Magnitudes below are from Zheng et al. (as of the 2023-12-24 arXiv revision) unless noted otherwise.

Bias	Evidence	Standard mitigation
Position	zero-shot GPT-4 gives a consistent verdict across both presentation orders in only 65.0% of cases; few-shot examples raise consistency to 77.5% at ~4x the API cost	swap or randomize order; inconsistent verdicts count as ties
Verbosity	a "repetitive list" attack (same content, restated) fools Claude-v1 and GPT-3.5 as judges in 91.3% of cases each, GPT-4 in 8.7%; the AlpacaEval baseline's measured win rate swings between 22.9% and 64.3% purely on verbosity instructions (LC-AlpacaEval, as of 2024-04)	length-controlled win rate — regressing out length raised AlpacaEval's Spearman correlation with Chatbot Arena from 0.94 to 0.98 (same paper)
Self-enhancement	mixed evidence: GPT-4 favored its own outputs with a +10% higher win rate and Claude-v1 with +25%, but GPT-3.5 showed no self-preference, and the authors state their data cannot determine whether the bias is real	open question — treat same-vendor generator/judge pairs as an unverified risk, not a proven flaw

Judges are gameable end-to-end. A "null model" that returns one constant, task-irrelevant response — never attempting the task — scored an 86.5% length-controlled win rate on AlpacaEval 2.0, 83.0 on Arena-Hard-Auto, and 9.55 on MT-Bench, despite existing length and style controls (Cheating Automatic LLM Benchmarks, as of 2024-10). Any harness where the generator's incentives point at the judge inherits this attack surface.

Optimization targets rot. When a judge, or a reward model distilled from one, becomes an RL target, the policy learns the proxy's weaknesses instead of the task: a survey of reward hacking describes RLAIF's reward signal as inheriting "the supervising LLM's blind spots and linguistic biases, allowing the policy to reverse-engineer and pander to the AI judge" (reward-hacking survey, as of 2026-04) — the failure family covered under reward-hacking.

Aggregation collapses under conformity. In homogeneous multi-agent debate settled by majority vote, models "frequently shift from correct to incorrect answers in response to peer reasoning", and correct-to-incorrect flips outnumber the reverse (debate failure-mode analysis, as of 2025-09). A judge or vote aggregating a same-backbone fleet can amplify agreement rather than correctness.

One fix does not close the class. The CALM audit quantifies 12 distinct bias types in LLM judges and finds significant task-dependent biases persisting even in advanced judge models (Justice or Prejudice?, as of 2024-10). Order-swapping and length control each close one hole; none generalizes.

Judging costs real money and wall-clock. A production judge step is a second, often larger, inference pass over the same work: Claude Code's Code Review averages $15–25 and about 20 minutes per PR review, scaling with diff size and complexity (pricing docs, as of 2026-07-10). Budget the judge like a team member, not a free assertion.

Known implementations

Implementation	Placement	What it demonstrates
MT-Bench + Chatbot Arena	offline benchmark	origin of the term; GPT-4 judge validated against 3K expert votes and 3K crowdsourced Arena votes (as of 2023-12-24)
AlpacaEval 2.0	offline leaderboard	weighted_alpaca_eval_gpt4_turbo annotator: GPT-4-turbo judge with logprob-weighted preferences against a fixed baseline, randomized order (as of 2026-07-10)
OpenAI Evals model-graded	offline eval framework	the productized template: wrap the completion in an eval prompt, cot_classify by default (as of 2026-07-10)
Claude Code Code Review	production, PR scale	parallel specialized judge agents plus behavior-grounded verification; findings tagged Important / Nit / Pre-existing (as of 2026-07-10)
Constitutional AI	training time	a model judges pairs of sampled responses; the AI-preference dataset trains the preference model used as the RL reward (RLAIF)
Agent-as-a-Judge	trajectory evaluation	judges the full task-solving process against DevAI's 365 hierarchical requirements; reported as reliable as its human evaluation baseline (as of 2024-10)

Related patterns

• evaluator-optimizer — the judge wired directly into the generation loop, returning feedback rather than only verdicts (Anthropic).
• reflexion — evaluator signal converted into verbal self-reflections kept in an episodic memory buffer across trials, improving the agent without weight updates (Shinn et al.).
• best-of-n-sampling — the judge as selector: rank N candidates, keep one, discard the feedback.
• multi-agent-debate — agents argue before a judge or vote; inherits the conformity failures above.
• rlaif — the judge role moved to training time: verdicts become the reward signal instead of a runtime gate.