Multi-Agent Debate

Multi-agent debate (MAD) is a harness pattern in which several LLM agents answer the same task independently, then read each other's answers and reasoning and revise their own over one or more rounds, converging on a common final answer. The pattern was introduced by Du, Li, Torralba, Tenenbaum & Mordatch (arXiv, May 2023; ICML 2024), who describe it as multiple language model instances that "propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer". The core bet: independently sampled agents err differently, so mutual critique recovers answers no single pass produces. The core trade-off: the bet pays only while agent errors stay decorrelated, and the price is a multiple of single-pass inference cost.

Problem and context

Inference-time quality levers that stay inside one model instance — self-reflection, scratchpads, self-consistency sampling — inherit that instance's blind spots. Du et al. position debate as complementary to these prompting techniques, and their own reflection baseline lost to debate on every task they measured (as of the ICML 2024 version). Debate is also more than majority amplification: the authors report cases where every agent starts wrong and the group still reaches the correct answer through mutual critique — the mechanism can create a correct answer, not merely select one. Whether it actually does depends on whether the agents' errors are independent, which is precisely where the pattern's known failure modes live (see below).

Structure

The minimal form has one role, replicated: N proposer agents, no judge. Round 0, every agent answers independently. Each later round, every agent receives its peers' answers and reasoning concatenated into its prompt with a consensus instruction, and produces a revised answer. The loop stops after a fixed round budget or when a termination governor fires; the final answer is the converged (or majority) position. The agent-harness owns the loop: fan-out, round transcripts, context assembly, stopping.

The origin configuration — 3 ChatGPT-based agents, two rounds (Du et al., as of the ICML 2024 version, accessed 2026-07-15):

Task	Single agent	Majority vote	Debate
Arithmetic (% correct)	67.0	69.0	81.8
GSM8K (% correct)	77.0	81.0	85.0
Chess move prediction (Stockfish pawn-score advantage)	91.4	102.2	122.9

The harness designer controls four knobs:

Knob	What it does	Evidence
Agent count N	arithmetic accuracy rises monotonically with more agents (rounds fixed at two); peer responses get summarized when concatenation overflows the context	Du et al., as of ICML 2024 version
Round count R	monotonic gains on arithmetic, plateauing above 4 rounds	Du et al., same version
Consensus-prompt stubbornness	prompts making agents more "stubborn" about their own solutions produce longer debates and better final answers; instruction-tuned agents are "agreeable" by default	Du et al., same version
Visibility protocol	in one controlled comparison, rank-adaptive cross-round (RA-CR) — an external judge scores agents each round, reorders them and silences the weakest — scored highest on consensus formation: 0.647 [0.555, 0.734] vs 0.359 [0.276, 0.443] for plain cross-round visibility	Zargari Marandi, single macroeconomic-forecasting case study, as of 2026-03

Termination can be governed instead of fixed: a Wald sequential probability ratio test (SPRT) monitoring per-round consensus stopped GSM8K debates after 1.01 rounds / 4.06 LLM calls on average at 97.0% accuracy, versus 15 calls at 99.0% for a fixed 5-round protocol — roughly 3.7x fewer calls for a 2pp accuracy cost (Morandi, single-author preprint, as of 2026-05).

When to use / When not to

Use it when agent errors are plausibly decorrelated and no mechanical verifier exists. The verified gains are on reasoning tasks — arithmetic, grade-school math, chess move prediction (Du et al.) — where independent sampling produces genuinely different solution paths that critique can arbitrate.

Use it with a compute governor, not a fixed round budget. Most debates converge early; paying for fixed rounds buys little (Morandi, as of 2026-05).

Don't use it where ground truth is mechanical. A test suite or verifier answers cheaper and without conformity dynamics — swe-bench-style patch grading needs no debate.

Don't use it where the crowd is confidently wrong. Under correlated errors, debate approximates majority voting in expectation (see failure modes) — it will polish the crowd's wrong answer rather than escape it.

Don't assume it beats cheaper compute-spending patterns. No verified head-to-head comparison of MAD against self-consistency or best-of-n-sampling at matched inference budget survived this page's research pass; treat "debate > N parallel samples + selection" as an open question, not a default.

Trade-offs and failure modes

Cost multiplies twice. Calls scale as N x (R+1), and each round every agent re-reads all peers' transcripts, so per-call input grows with both knobs — pressure on the context-window that the origin paper already handled by summarizing peer responses for larger N (Du et al.).

The Martingale Curse: correlated errors reduce debate to majority voting. Liu et al. prove that standard MAD with linear, symmetric belief aggregation is a martingale under correlated agent errors — expected belief never moves, so debate equals majority voting in expectation. Empirically, on challenging subsets where the initial majority is wrong (N=5, decentralized debate, averaged over TruthfulQA, ARC-C, BBH, LogiQA, MedQA, MMLU-Pro), majority voting scores 14.0% and standard MAD only 22.1% (as of 2026-03). Weigh the source: a single unreviewed preprint whose weak-baseline numbers motivate the authors' own fix (AceMAD, nonlinear aggregation via peer-prediction), with no independent replication yet.

Conformity flips correct answers. In debate transcripts, "models frequently shift from correct to incorrect answers in response to peer reasoning", and correct-to-incorrect flips outnumber the reverse; longer debates can degrade further, and adding a weaker model to a debate with a stronger one can drag the outcome down (Wynn et al., as of 2025-09). This cuts against the intuition that a heterogeneous pool is automatically safer: decorrelation helps only if the added agent is not simply worse.

The headline factuality claim is contested. The origin paper's abstract also claims improved factual validity and reduced hallucinations, with its own supporting numbers (Du et al.), but this unscoped claim did not survive adversarial verification during this page's research pass, and the correlated-error results above target exactly the factual-QA regime. Cite MAD for the benchmark-scoped reasoning gains; treat hallucination reduction as unsettled.

Protocol choice is load-bearing and under-measured. The RA-CR result (0.647 vs 0.359 consensus formation, Zargari Marandi, as of 2026-03) comes from one task domain and one judge model; it shows protocols matter, not which protocol wins in general.

Known implementations

Implementation	What it is	Status
Du et al. reference implementation	origin recipe: identical-model agents, concatenated peer responses, consensus prompt	research code, project page (as of 2026-07-15)
RA-CR protocol study	debate protocols with an external judge reordering and silencing agents between rounds	single-domain case study, unreviewed preprint (as of 2026-03)
AceMAD	nonlinear aggregation via peer-prediction, proposed to escape the martingale regime	unreviewed preprint; efficacy claims unverified (as of 2026-03)
Wald-SPRT governor	sequential-test early termination wrapped around any debate loop	single-author unreviewed preprint (as of 2026-05)

No production agent product is documented as shipping MAD as a built-in primitive among the sources reviewed here; as of 2026-07 the pattern is assembled at the harness level from these research recipes.

Related patterns

• llm-as-judge — one model evaluates instead of peers arguing; debate's RA-CR variant imports a judge as a moderator rather than a verdict-giver.
• agent-as-a-judge — trajectory-level evaluation by an agent; whether debate panels make better judges than a single judge has no verified evidence in this page's sources — open question.
• best-of-n-sampling — spends the same extra compute on parallel samples plus selection, with no inter-agent interaction; the matched-budget comparison against debate is unverified.
• evaluation-awareness — debate transcripts expose each agent's reasoning to peers, a distinct regime from an agent knowing it is being evaluated.