Terminal-Bench

Terminal-Bench measures whether an AI agent can complete real, end-to-end tasks in a command-line environment: each task drops the agent into a Docker container with an instruction in English, and test scripts grade the final state of that container (GitHub README; Terminal-Bench 2.0 paper). Released on 2025-05-19 by Mike Merrill, Alex Shaw, Chris Rytting, Ludwig Schmidt and Andy Konwinski (announcement) as "a stanford x laude collaboration" (tbench.ai), it has become the reference number for terminal-agent claims the way swe-bench is for issue-resolution claims. As of 2026-07-15, versions 2.0 and 2.1 are live, both run through the Harbor harness, and the leaderboard ranks agent+model pairs — a detail that matters more than most headline citations admit (leaderboards index).

Methodology

Terminal-Bench is two artifacts: a dataset of tasks and an execution harness. Each task packages an instruction in English, a test script that verifies completion, and a reference ("oracle") solution (GitHub README). Concretely, a task directory holds a Dockerfile or docker-compose file, a task.yaml, the oracle solution.sh, and run-tests.sh plus test dependencies (announcement). The launch dataset, Terminal-Bench-Core-v0, contained 80 hand-crafted tasks spanning end-to-end work like compiling code, training models and setting up servers (announcement; GitHub README).

Terminal-Bench 2.0 (paper published 2026-01-17) rebuilt the dataset: 93 contributors created 229 candidate tasks, of which 89 survived a three-phase human review — on average roughly three hours of combined reviewer attention per task (Terminal-Bench 2.0 paper). Verification is deliberately outcome-driven: "the tests verify that all outcomes described in the instruction have been achieved by testing properties of the final container state; they do not test the agent's commands or console output" (same paper). At the paper's release, frontier models and agents scored below 65% on the 2.0 set (abstract).

The agent-environment interface is a real shell, not a simulated API: agents run inside the container and act by issuing shell commands. The maintainers' reference scaffold, Terminus — at its 2025-05-19 release the second-highest-scoring agent on the benchmark, behind only Claude Code (Terminus announcement) — exists to give models a minimal common harness; its successor Terminus 2 uses a single tool, a headless terminal, and completes tasks with Bash alone (Terminal-Bench 2.0 paper).

Versions

Since 2.0, tasks use the Harbor task format and run on the Harbor harness; leaderboard submissions for 2.0/2.1 must go through Harbor (Terminal-Bench 2.0 paper; leaderboards index). Version status as of 2026-07-15 (leaderboards index):

Version	Status	Notes
1.0 (Terminal-Bench-Core v0.1.1)	legacy	original harness, terminal-bench-core==0.1.1
2.0	live	89 tasks; submissions via Harbor
2.1	live	current headline version; submissions via Harbor
3.0	coming soon	announced as the next frontier version
Terminal-Bench-Science 1.0	coming soon	domain-specific: scientific computing in terminal environments

Registry and adapters

Beyond its own task sets, the project ships a registry that runs other agentic benchmarks through the same harness. At the registry's launch (2025-07-15), four benchmarks were adapted — SWE-bench Verified, AppWorld, DevEval and EvoEval — with eight more planned; adapters "restructure existing benchmarks to use the Terminal-Bench task framework but do not alter the task contents," validated by parity-testing resolved rates against the originals (registry announcement). For a practitioner this is the quiet win: one harness, one logging and parallelization stack, many benchmarks.

Current results

Terminal-Bench 2.1 snapshot, best agent+model run per model, as of 2026-07-12 (Terminal-Bench 2.1 leaderboard; task success rate, ± is the leaderboard's confidence interval):

Model	Agent	Success rate
GPT-5.5	Codex CLI	83.4% (CI 81.2–85.6)
Claude Fable 5	Claude Code	83.1% (CI 81.1–85.1)
Claude Opus 4.8	Claude Code	78.9% (CI 76.4–81.4)
Gemini 3 Pro	Terminus 2	74.4% (CI 71.8–77.0)
Gemini 3.1 Pro	Gemini CLI	70.7% (CI 67.8–73.6)
Claude Opus 4.7	Claude Code	69.7% (CI 67.0–72.4)
GLM-5.1	Claude Code	58.7% (CI 56.3–61.1)

Each row selects the model's best run across submitted agents; the same model with a different scaffold lands visibly lower — Claude Fable 5 scores 80.4 ± 2.3 with Terminus 2 versus 83.1 with Claude Code, and Gemini 3 Pro scores 66.3 ± 2.7 with Gemini CLI versus 74.4 with Terminus 2 (Terminal-Bench 2.1 leaderboard, as of 2026-07-12). Note that the leaderboard pages are client-side rendered — the table above comes from a dated snapshot of the rendered data, not a raw page fetch.

Limitations and critiques

Weak correlation with in-the-wild terminal work. TerminalWorld (paper dated 2026-05-21) reverse-engineered 1,530 evaluation tasks from 80,870 real terminal recordings — 18 task categories, 1,280 unique commands — and reports that Terminal-Bench scores show "only a weak correlation" with performance on its manually reviewed 200-task Verified subset: Pearson r = 0.20 (TerminalWorld). Read this with two caveats: the paper introduces a competing benchmark, and the Verified subset is small. But the direction of the critique is structural — Terminal-Bench tasks are expert-curated and hard by design, while everyday terminal work is dominated by different command distributions.

The benchmark is now a training target. NVIDIA's Nemotron-Terminal work (paper dated 2026-02) built a synthetic terminal-task generation pipeline and lifted Terminal-Bench 2.0 scores from 4.0% to 20.2% (14B model) and from 3.4% to 27.4% (32B) (Nemotron-Terminal). Once labs optimize directly for a benchmark's task distribution, its scores start measuring targeted training as much as general capability — the soft end of the benchmark-contamination spectrum, no test-set leakage required.

Outcome-only grading ignores the path. Tests inspect the final container state, not the commands the agent ran (Terminal-Bench 2.0 paper). That makes grading robust to solution diversity, but it also means the score says nothing about how the agent got there — destructive detours, wasted compute, or shortcut solutions that happen to satisfy the checked properties all grade identically to a clean run.

Small task count, wide intervals. With 89 tasks in 2.0, per-run confidence intervals on the leaderboard run about ±2–2.5 points (Terminal-Bench 2.1 leaderboard, as of 2026-07-12) — a single flaky task moves a score more than a percentage point.

How to read the scores

• A score names a pair, not a model. The leaderboard ranks agent+model combinations, and the scaffold effect is large (Gemini 3 Pro: 74.4 vs 66.3 depending on agent, as of 2026-07-12, leaderboard). A Terminal-Bench number quoted without the agent's name is not interpretable — the same discipline applies as with swe-bench scaffolds, and it is why the agent-harness is part of what you are benchmarking.
• Respect the intervals. The top two entries in the 2026-07-12 snapshot — 83.4 (CI 81.2–85.6) and 83.1 (CI 81.1–85.1) — overlap almost entirely; treating that as a ranking is noise-reading.
• Never compare across versions. 1.0, 2.0 and 2.1 are different task sets. "Frontier agents scored below 65%" (2.0, January 2026, abstract) and "83% at the top" (2.1, July 2026 snapshot) are measurements on different rulers, not a capability trend.
• Best-of-runs tables flatter. Snapshot tables (including the one above) usually keep each model's best agent pairing; the average submitted run sits lower.
• Expect limited transfer to your own workflows. The r = 0.20 result (TerminalWorld) is one data point, from a rival benchmark — but it matches the design: Terminal-Bench measures hard, curated, end-to-end tasks under a specific harness. Treat it as a controlled smoke test of a model+scaffold's terminal competence, not a forecast of performance on your team's actual shell work.

Related

• swe-bench — the adjacent agentic-coding benchmark; runnable through the Terminal-Bench registry adapter
• osworld — the GUI counterpart: real desktop environments instead of a terminal
• humanitys-last-exam — knowledge-frontier benchmark; contrast with execution-graded evaluation
• benchmark-contamination — the failure mode that training-target pressure invites
• agent-harness — why scores attach to harness+model pairs
• claude-code — the agent behind several of the leaderboard's top runs