SWE-bench

SWE-bench is a software-engineering benchmark that measures whether an AI system can resolve real GitHub issues: given a snapshot of a repository and the text of an issue, the system must produce a patch, which is graded by running the repository's own tests (SWE-bench paper). Created by Jimenez, Yang et al. (Princeton / University of Chicago) and published at ICLR 2024, it grew into a family of splits — Lite, Verified, Multimodal, Multilingual, plus Scale AI's successor SWE-bench Pro — and became the headline coding number in frontier system cards. It is also a well-documented case study of why a benchmark score must be read as a measurement of model plus harness, and of what benchmark-contamination looks like at the frontier.

Methodology

The original dataset holds 2,294 task instances from 12 popular Python repositories, distilled from roughly 90,000 scraped pull requests in a three-stage pipeline: keep PRs that both resolve a GitHub issue and modify test files, then execution-filter to PRs where at least one test flips from failing to passing once the PR is applied; 40% of surviving instances have two or more such fail-to-pass tests (SWE-bench paper).

Grading is binary per instance: the candidate patch must make every FAIL_TO_PASS test pass (the issue is actually fixed) while every PASS_TO_PASS test keeps passing (nothing else regressed). The model never sees the tests — only the issue text and the repository state (Introducing SWE-bench Verified). Since June 2024 the official harness runs each instance in a Docker container; the maintainers recommend at least 120 GB of free storage, 16 GB RAM and 8 CPU cores for a full local run (SWE-bench repository).

At publication (October 2023) the benchmark was nowhere near saturated: the best configuration — Claude 2 fed BM25-retrieved files through a 13k-token context window — resolved 1.96% of instances (SWE-bench paper).

Splits

Split	Size	What it changes
Full (original)	2,294 instances, 12 Python repos	Baseline dataset (arXiv:2310.06770)
Lite	300 instances, 11 of the 12 repos	Cheaper iteration: drops multi-file patches, gold patches with >3 edit hunks, file creation/deletion, images/external links, <40-word problem statements (official page)
Verified	500 instances	Human-vetted subset; the split vendors usually quote (see below) (OpenAI)
Multimodal	617 instances, 17 JavaScript repos	Every instance carries at least one image (web UI, diagramming, data visualization, syntax highlighting, mapping); test-split answers private, scored via the cloud tool sb-cli (paper; repository)
Multilingual	300 instances, 42 repos	Nine languages: C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby, Rust (official page)
Pro	1,865 problems, 41 repos: 11 public, 12 held-out, 18 commercial	Scale AI's harder successor (published 2025-09-19): multi-file, hours-to-days tasks, designed for contamination resistance (Scale Labs)

Instance counts as of 2026-07-10. Pro is a separate benchmark reusing the format — see swe-bench-pro.

How Verified was built. OpenAI's Preparedness team had 93 professional Python developers triple-annotate 1,699 randomly sampled instances: 38.3% of samples were flagged for underspecified problem statements, 61.1% for unit tests that could reject valid solutions, and 68.3% overall — the survivors were sampled into the final 500 (Introducing SWE-bench Verified). Filtering removed impossible tasks rather than hard ones: at Verified's August 2024 release, GPT-4o with the open-source agentless scaffold resolved 33.2% of Verified versus 16% of the original test set — the unfiltered benchmark had been systematically underestimating models (same source).

Current results

Three snapshots below deliberately disagree; the disagreement itself is the point (see "How to read the scores" below).

Official community leaderboard (Verified). Entries are scaffold + model systems, not raw models (swebench.com leaderboard; data last updated 2026-02-27, accessed 2026-07-10):

System (scaffold + model)	Resolved	Submitted
live-SWE-agent + Claude 4.5 Opus (medium, 2025-11-01)	79.2%	2025-12-15
Sonar Foundation Agent + Claude 4.5 Opus	79.2%	2025-12-05
TRAE + Doubao-Seed-Code	78.8%	2025-09-28
live-SWE-agent + Gemini 3 Pro Preview	77.4%	2025-11-20
Atlassian Rovo Dev	76.8%	2025-09-02

Vendor self-reports. Anthropic's Claude Opus 4.8 system card reports 5-trial averages, with competitor figures drawn from those vendors' own published system cards and leaderboards (system card, accessed 2026-07-10; scores in %):

Split	Claude Opus 4.8	Claude Opus 4.7	GPT-5.5	Gemini 3.1 Pro
Verified	88.6	87.6	—	80.6
Pro	69.2	64.3	58.6	54.2
Multilingual	84.4	80.5	—	—
Multimodal (internal harness)	38.4	34.5	—	—

The vendor-claimed frontier has moved past this table: Claude Fable 5 (released 2026-06-09) self-reports 95% on Verified and 80% on Pro — 5-trial averages whose scores "reflect its production safeguards, including fallback to Opus 4.8"; its limited-access sibling Claude Mythos 5 posts 95.5% and 80.3% on the same splits (Fable 5 & Mythos 5 system card, §8.1–8.2, accessed 2026-07-11). Scale's third-party leaderboard lists no Fable 5 entry as of the same date (Scale leaderboard).

Third-party-run leaderboard (Pro public set, 731 instances). Scale AI executes submissions itself and reports pass@1 (see pass@k) with confidence intervals (Scale leaderboard, accessed 2026-07-10):

System	pass@1, %
Muse Spark 1.1	61.50 ± 3.10
gpt-5.4 (xHigh)	59.10 ± 3.56
Muse Spark	55.00 ± 3.60
claude-opus-4-6 (thinking)	51.90 ± 3.61
gemini-3.1-pro (thinking)	46.10 ± 3.60

Limitations and critiques

"Verified" does not mean flaw-free. Epoch AI, which maintains independent evaluation infrastructure, runs 484 of the 500 instances (16 excluded as unreliable in its environment — network requirements, dependency conflicts) and cites an estimated dataset error rate of 5–10% (Epoch AI). OpenAI's 2026 re-audit went further: of 138 Verified problems that o3 failed consistently across 64 independent runs, at least 59.4% had material flaws — 35.5% "narrow" tests enforcing unstated implementation details, 18.8% "wide" tests requiring functionality the problem never asked for, 5.1% other issues (Why SWE-bench Verified no longer measures frontier coding capabilities).

Training-data contamination. In the same audit, adversarial multi-turn probing led GPT-5.2-Chat, Claude Opus 4.5 and Gemini 3 Flash Preview to reproduce verbatim gold-patch content for tasks they had never been shown in full — Claude Opus 4.5 recalled one task's exact 4-line diff, its filename, method name and an inline code comment (OpenAI audit). Independent corroboration: frontier models identify the buggy file path from issue text alone with up to 76% accuracy on SWE-bench's own repositories but at most 53% on structurally similar tasks from outside repositories, and show up to 35% verbatim 5-gram overlap with reference solutions on SWE-bench Verified/Full versus at most 18% on comparable non-SWE-bench tasks (The SWE-Bench Illusion). OpenAI concluded that frontier gains on Verified increasingly reflect training exposure rather than capability, stopped reporting it, and at the time recommended SWE-bench Pro's public split (OpenAI audit). That endorsement did not survive the same scrutiny: on 2026-07-08 OpenAI audited SWE-bench Pro itself, found about 30% of its tasks broken (overly strict tests, underspecified prompts, low-coverage tests, misleading prompts), noted the 731-task public split's pass rate had climbed from 23.3% to 80.3% in eight months, and retracted its recommendation that the community use it as a leading coding eval (Separating signal from noise in coding evaluations).

The grader under-tests patches. Evaluation re-runs only the test files the original PR modified, not the repository's full suite. Running complete developer test suites fails 7.8% of officially "resolved" patches on average, and manual inspection suggests roughly 11% of plausible patches are not actually correct — mostly similar-but-divergent implementations (46.8% of inspected discrepancies) or extra unrequested changes (27.3%) (Wang, Pradel & Liu 2025).

Scope skew toward short tasks. 77.8% of the original dataset was estimated to take an experienced engineer under one hour; Lite and Verified push tasks estimated over one hour below 10% (Introducing SWE-bench Verified). The benchmark therefore measures short, self-contained bug fixing in mature codebases — not design-heavy, multi-day, or greenfield engineering, the gap Pro's hours-to-days tasks target (Scale Labs).

Saturation at the top. State of the art on Verified improved only from 74.9% to 80.9% over the six months preceding OpenAI's audit, which is part of why OpenAI questions whether remaining gains reflect model improvement at all (OpenAI audit).

How to read the scores

• A score prices a system, not a model. Every reported number conflates model, scaffold, prompts, inference settings and environment; a 2026 position paper argues single end-to-end scores make model-versus-tooling attribution impossible, and that grading against one reference solution penalizes valid alternative implementations (Gorinova et al. 2026). Concretely: OpenAI's GPT-5 system card evaluates a fixed 477-instance subset with an internal bash + apply_patch scaffold, pass@1 averaged over 4 tries — and flags that the 74.9% launch-blog figure was run at medium verbosity while system-card evals ran at maximum, which alone shifts results (GPT-5 System Card). Epoch runs 484 of 500 instances (Epoch AI). Three organizations, three different "SWE-bench Verified" measurements.
• Self-report ≠ third-party measurement. Anthropic self-reports 69.2% on Pro for Claude Opus 4.8 (own harness, 5-trial average, as of 2026-07-10 access) (system card), while Scale's own leaderboard — which runs submissions itself — shows its best Claude entry, claude-opus-4-6 (thinking), at 51.9 ± 3.61 and lists no Opus 4.8 at all as of 2026-07-10 (Scale leaderboard). Neither number is wrong; they are different measurements. Comparing a vendor bar chart against a leaderboard entry is a category error.
• Mind the noise floor. On 500 instances a single task is worth 0.2 points; Scale publishes ±3-point confidence intervals on 731 instances (Scale leaderboard); the dataset itself carries an estimated 5–10% error rate (Epoch AI). Differences of a few points between differently-harnessed runs carry no signal.
• Prefer contamination-resistant comparisons at the frontier — but audit the audit. Verified deltas between 2026-era frontier models are confounded by memorization (see above), and held-out or private-set designs (swe-bench-pro, terminal-bench) are structurally safer. They are not automatically sound, though: OpenAI's 2026-07-08 audit found about 30% of SWE-bench Pro tasks broken and withdrew its recommendation of the public split (Separating signal from noise). As of 2026-07-11 there is no single community-endorsed successor; check a benchmark's latest audit status before letting it drive a model decision.
• Transfer to your own stack is limited. Leaderboard leaders are scaffold + model systems (live-SWE-agent wrapping frontier models, proprietary agents like TRAE), so a reported score transfers to your setup only to the degree your harness — claude-code, a custom loop, a CI agent — resembles the submitted one. Treat SWE-bench as a smoke test of a specific model-plus-harness pairing, not a hiring bar for a model in isolation.

Related

• swe-bench-pro — Scale AI's longer-horizon successor with held-out and commercial sets
• terminal-bench — adjacent agentic benchmark for command-line tasks
• swe-agent — agent scaffold used for official SWE-bench baselines (e.g. the Multilingual release baseline)
• agentless — minimalist open-source scaffold used in the Verified release measurements
• pass-at-k — the metric family behind reported pass@1 numbers
• benchmark-contamination — the failure mode SWE-bench now exemplifies