OSWorld

OSWorld is a computer-use benchmark that measures whether a multimodal agent can complete real tasks on a live operating system, grading not the agent's output text but the resulting machine state: each task ships an initial-state setup configuration and a custom execution-based evaluation script (OSWorld paper). Introduced by Xie et al. in April 2024 and published at the NeurIPS 2024 Datasets & Benchmarks track, it packages 369 tasks on Ubuntu spanning real web and desktop applications, OS file I/O, and workflows across multiple applications (same paper). Its state-check evaluation design became the reference approach for scoring agents that act on real systems, and its ecosystem now includes a Verified maintenance pass and a separate long-horizon successor suite, OSWorld 2.0.

Methodology

The paper distinguishes the OSWorld environment — self-described as "the first-of-its-kind scalable, real computer environment for multimodal agents," architecturally supporting Ubuntu, Windows, and macOS — from the 369-task benchmark built on it; field usage treats the two as synonymous (paper). The standard leaderboard suite runs on Ubuntu; 43 additional Windows tasks exist for analysis only, so "supports three OSes" should not be read as "the leaderboard covers three OSes" (official project page).

Tasks are exercised through both GUI and CLI against real applications — the sources enumerate Chromium, GIMP, the LibreOffice suite, Thunderbird, VLC, VS Code, and the OS itself — rather than mocked interfaces (paper). Grading is execution-based: after the agent finishes, a per-task script checks whether the machine ended up in the correct state (file contents, app settings, browser state), drawing on 134 unique evaluation functions; tasks were manually annotated by 9 computer-science students at a cost of roughly 1,800 man-hours (paper). This is the key contrast with llm-as-judge and agent-as-a-judge grading: correctness is a deterministic predicate over system state, not a model's opinion of a transcript.

Variants

Variant	Tasks	What it changes
OSWorld v1 (April 2024)	369 on Ubuntu, +43 Windows analysis-only	Baseline suite (paper)
361-task subset	361	Officially permitted exclusion of 8 Google Drive tasks prone to setup failures from IP changes and network factors (official page)
OSWorld-Verified (2025-07-28)	Same suite, quality-fixed	Fixes community-reported broken tasks, adds AWS harness support (full run within ~1 hour), publishes verified trajectories on Hugging Face (official page)
OSWorld 2.0 (June 2026)	108 long-horizon workflows	Separate successor suite with its own score scale — not comparable to v1/Verified numbers (OSWorld 2.0 paper)

Current results

Launch-era baseline (2024, historical). In the original paper, the best configuration — GPT-4 driving the accessibility-tree input — completed 12.24% of tasks, against a human baseline above 72.36%; GPT-4V screenshot-based variants scored 5.26–5.80%, and failures concentrated in GUI grounding and operational knowledge (paper). These figures define the initial capability gap, not current capability.

OSWorld 2.0 (as of 2026-07-15; paper revised 2026-07-13). Under the suite's primary binary-completion metric at a 500-step budget (OSWorld 2.0 paper):

Model	Completion	Partial score
Claude Opus 4.8 (max thinking, batched tool calls)	20.6%	54.8%
GPT-5.5	plateaus near 13%	—

The same paper reports its tasks require an average of 318 tool calls (measured with Claude Opus 4.7) versus roughly 30 in the prior version (OSWorld 2.0 paper).

OSWorld-Verified. The official leaderboard renders its results client-side ("Loading verified benchmark data..." in the static HTML), and no specific Verified scores are reproduced here as of 2026-07-15 — check the official page or the announcing vendor's primary post before citing a number.

Limitations and critiques

• Denominator trap. Because excluding the 8 flaky Google Drive tasks is officially sanctioned, published scores may be computed over 369 or 361 tasks; the official page allows both (official page). Two scores a point apart may differ by denominator, not capability.
• Variant conflation. v1, Verified, and OSWorld 2.0 are three different measurements — 2.0 is a different task suite on a different score scale (OSWorld 2.0 paper). A "score on OSWorld" without a variant label is unusable; this page's own launch-era and 2.0 numbers differ by suite, not by regression.
• Binary success hides efficiency. OSWorld-Human annotated a human-determined trajectory for each of the 369 tasks and proposed the Weighted Efficiency Score (WES+/WES−), combining success with step efficiency; leading agents take 1.4–2.7× more steps than required, and the best-scoring system measured dropped from 42.5% accuracy to 17.4% on the strictest efficiency metric (OSWorld-Human). A leaderboard score says nothing about latency or token cost per task.
• Short-horizon skew. Most v1 tasks are minutes-scale. OS-Marathon (242 long-horizon repetitive tasks across 2 domains, e.g. processing expense reports and entering grades) targets the gap and attributes agent failures to three modes: logical incoherence in task ordering, hallucinated planning actions ungrounded in current state, and inability to maintain long-horizon consistency (OS-Marathon). OSWorld 2.0's 318-vs-30 tool-call jump is the same critique built into a successor suite (OSWorld 2.0 paper).
• Environment flakiness. Tasks depending on live network services can fail at initialization rather than through agent error (official page) — real-environment fidelity buys realism at the price of reproducibility, which is exactly what the Verified pass and its AWS harness try to claw back.

How to read the scores

• Check suite, variant, and denominator first. A 2.0 completion rate near 20% and a Verified score several times higher can both be true simultaneously; comparing across variants is a category error.
• State-check grading resists answer leakage. Since correctness is a predicate over the resulting machine state, a memorized answer string is worthless — the agent still has to operate the GUI. That makes OSWorld structurally more resistant to benchmark-contamination than static-answer evaluations like humanitys-last-exam, though environment familiarity from training on trajectories remains an open concern (editorial inference from the grading design, not a published measurement).
• Scores price model + input representation + scaffold. In the launch paper the same underlying model family spanned 5.26–12.24% depending on observation format alone (paper); OSWorld 2.0's leader is a specific configuration — max thinking, batched tool calls (OSWorld 2.0 paper). Transfer to your stack depends on your harness matching that configuration.
• The current failure axis is horizon, not grounding. Launch-era failures were GUI grounding (paper); the 2026 successor suites converge on state-maintenance over hundreds of steps as the binding constraint (OSWorld 2.0; OS-Marathon). For practitioners this points at context-compaction and checkpointing strategy, not better click accuracy.

Related

• terminal-bench — adjacent execution-graded benchmark for terminal-only agent tasks
• swe-bench — the analogous execution-graded standard for coding agents, and a case study in reading scores skeptically
• computer-use — the capability OSWorld measures
• benchmark-contamination — the failure mode OSWorld's grading design partially sidesteps
• humanitys-last-exam — contrast case: static-answer frontier benchmark