Agent-as-a-Judge

Agent-as-a-Judge is a harness pattern in which the evaluator of an agentic system is itself an agent: a judge equipped with tools to walk the evaluated agent's workspace, read its files, and trace its trajectory, returning a verdict per requirement instead of one score over the final output. Zhuge et al. introduced the pattern as "an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process" (arXiv:2410.10934, October 2024; published at ICML 2025). Where an llm-as-judge setup grades whatever fits into the judge's prompt, an agent judge gathers its own evidence — the pattern's core bet is that evidence-gathering, not verdict-writing, is what limits judge accuracy on agent work.

Problem and context

Evaluation of agent runs sits between two inadequate extremes, which the founding paper states directly: contemporary techniques "either focus exclusively on final outcomes — ignoring the step-by-step nature of agentic systems, or require excessive manual labour" (Zhuge et al.). Mechanical outcome checks — swe-bench's apply-the-patch-and-run-tests regime, terminal-bench-style end-state verification — are cheap and objective but give no credit for partially completed work and no diagnosis of where a run went wrong. Human review of trajectories measures exactly that, but at agent scale it is the bottleneck: three expert evaluators needed a self-reported 86.5 hours for one pass over 55 tasks, which the authors price at $1,297.50 assuming a $15/hour expert wage (Section 4.4, as of 2024-10).

As a testbed the authors release DevAI: 55 automated AI-development tasks annotated with 365 hierarchical user requirements, graded per requirement rather than by a binary solve flag (Zhuge et al.). The judged systems are three harness-driven coding agents — MetaGPT, GPT-Pilot, and OpenHands, all running gpt-4o-2024-05-13 as the backend model (experiment setup, as of 2024-10).

Structure

The founding implementation is a modular judge agent. Eight components were designed; ablations kept five (Appendix K, as of 2024-10):

Component	Role in the judge
graph	builds a graph of the project — files, modules, dependencies
locate	finds the folder or file a requirement refers to
read	parses multimodal artifacts (code, images, video, documents; 33 formats)
retrieve	extracts relevant segments from long trajectories
ask	decides, on the collected context, whether a requirement is satisfied

Three further components were built and then dropped as net-harmful: search (BM25 snippet retrieval introduced noise), planning (unstable action sequencing), and memory (errors in past judgments chained into later ones) (Appendix K). The judge runs in two regimes: black-box, which needs only the final workspace, and gray-box, which additionally reads manually collected trajectory data the authors call "nearly inaccessible in practical situations" (Table 3 note).

Alignment with the consensus of three human experts on DevAI, black-box setting (as of 2024-10, Table 3):

Judged agent	Agent-as-a-Judge	LLM-as-a-Judge
OpenHands	90.44%	60.38%
GPT-Pilot	83.88%	65.30%
MetaGPT	88.52%	84.15%

The MetaGPT gap is small for a measured reason: MetaGPT rarely met requirements, so a judge that says "no" often scores well on alignment — the authors flag the class imbalance and add precision–recall curves (Section 4.2). Gray-box scores are higher for both judge types (86.61–92.07% for Agent-as-a-Judge across the three agents). For calibration: individual human evaluators aligned with their own consensus at 76.23–92.63%, and a three-human majority vote reached 93.98–95.08% — on OpenHands the agent judge beat every individual human (90.44% vs at most 89.89%) but stayed under the 94.26% majority-vote ceiling (Table 3, as of 2024-10).

The judge "can evaluate tasks during or after execution" and produces "continuous, step-by-step feedback that can be used as reward signals for further agentic training and improvement" (README, as of 2026-07-15) — the reward-signal use is stated design intent, not a demonstrated training result.

When to use / When not to

Use it when the deliverable is a workspace, not an answer. Multi-file outputs judged against itemized requirements are where requirement-level verdicts pay off; a transcript-only judge cannot open the files it is grading.

Use it when human review of agent runs is the bottleneck. The full DevAI pass cost $30.58 in API calls and 118.43 minutes versus $1,297.50 and 86.5 hours for the human panel (Section 4.4, as of 2024-10) — a ~97–98% reduction in both. Cite the raw figures: the paper's introduction and Section 4.4 disagree about which percentage is time and which is cost.

Don't use it where an end-state check answers the question. A test suite, compiler, or scripted state assertion is cheaper and bias-free; the pattern's value starts where requirements are articulable but not mechanically checkable.

Don't use it to save money over a plain LLM judge. On the same run, LLM-as-a-Judge cost $29.63 and 10.99 minutes against the agent judge's $30.58 and 118.43 minutes (Section 4.4, as of 2024-10): dollars are near-parity and wall-clock is ~11x. The purchase is accuracy, so deploy it only where a transcript judge is measurably outside your accuracy budget.

Don't make it an unguarded optimization target. A judged agent trained or selected against judge verdicts inherits the gaming surface of llm-as-judge plus reward-hacking dynamics, and an agent aware it is being evaluated may behave unrepresentatively (evaluation-awareness).

Trade-offs and failure modes

Self-reported evidence base. Every headline number comes from the team that built both the judge and the benchmark, on 55 coding-domain tasks, with the human baseline priced by a $15/hour wage assumption (Zhuge et al.). No independent replication of the specific figures exists. The strongest third-party signal is directional: AJ-Bench (155 tasks, 516 annotated trajectories across search, data systems, and GUIs) reports "consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification" (arXiv:2604.18240, ACL 2026 Findings, as of 2026-04).

Below the human ceiling. "As reliable as our human evaluation baseline" (abstract) means within individual-expert variance — not matching the three-human majority vote, which stayed 3–10 points higher across judged agents (Table 3, as of 2024-10).

Agentic components can subtract accuracy. Planning, memory, and BM25 search — standard agent equipment — all reduced judge alignment and were cut (Appendix K); a judge harness needs its own ablation discipline, not a maximal toolbelt.

Latency scales with evidence-gathering. The ~11x wall-clock over a transcript judge (Section 4.4, as of 2024-10) lands in the inner loop if verdicts gate every iteration; trajectory length grows it further.

The benchmark is public and small. DevAI's 55 tasks ship openly (dataset and code at metauto-ai/agent-as-a-judge), so headline alignment numbers are exposed to benchmark-contamination as the tasks age into training corpora.

Known implementations

Implementation	What it demonstrates
metauto-ai/agent-as-a-judge	MIT-licensed reference implementation of the judge plus the DevAI benchmark harness (as of 2026-07-15)
DevAI	55 tasks / 365 hierarchical requirements — the proof-of-concept testbed for requirement-level trajectory grading
AJ-Bench	a benchmark of agent judges: measures information acquisition, state verification, and process verification across search, data systems, and osworld-style GUI environments (as of 2026-04)

Related patterns

• llm-as-judge — the parent pattern: same generator/judge role split, but the judge reads a prompt instead of investigating an environment.
• evaluator-optimizer — wiring judge feedback back into the generator's loop; step-level verdicts make the feedback more targeted.
• best-of-n-sampling — judge verdicts as a selection signal over multiple candidate runs rather than feedback on one.
• multi-agent-debate — an alternative way to spend evaluation compute: argue before a verdict instead of investigating before a verdict.
• rlaif — the training-time destination for the step-level reward signals the pattern is designed to produce.