Humanity's Last Exam

Humanity's Last Exam (HLE) is a closed-ended benchmark of 2,500 expert-level academic questions spanning more than 100 subjects — mathematics, natural sciences, humanities — graded automatically against unambiguous reference answers (official site). It was created by the Center for AI Safety (CAIS) and Scale AI, with questions contributed by nearly 1,000 subject experts from over 500 institutions in 50 countries, as a deliberate response to benchmark saturation: frontier LLMs were scoring over 90% on MMLU-class evaluations, "limiting informed measurement of state-of-the-art LLM capabilities" (HLE paper). For practitioners it is the canonical live experiment in benchmark half-life: every frontier model scored under 10% at launch in January 2025, and the top leaderboard score passed 46% within 18 months.

Methodology

Dataset. The finalized set (2025-04-03) holds 2,500 questions; 14% require interpreting an image or diagram, and the remaining 86% are text-only — Scale maintains a separate text-only leaderboard for that subset (Scale text-only leaderboard, accessed 2026-07-14). Questions are exact-match short-answer or multiple-choice; each must have "a known solution that is unambiguous and easily verifiable, but cannot be quickly answered via internet retrieval" (HLE paper). HLE tests structured academic problems, not open-ended research or creative problem-solving (same source).

Question pipeline. Over 70,000 submitted questions were pre-filtered by testing them against frontier LLMs; roughly 13,000 that stumped the models advanced to human review — 1–3 first-round reviews by graduate-degree holders in the relevant field, then organizer approval of the top questions (HLE paper). Contributors competed for $5,000 prizes per question for the top 50 submissions ($500 for the next 500) plus paper co-authorship (Scale blog) — the arXiv paper lists Long Phan and 1,118 co-authors (arXiv:2501.14249).

Grading. Answers are graded by an LLM judge with structured decoding, which verifies whether the model's answer matches the reference within equivalent formats and a small numerical margin, and extracts the model's stated confidence. The judge model depends on the protocol: the paper's January 2025 launch evaluation used GPT-4o (gpt-4o-2024-08-06) (HLE paper), while the current official leaderboard uses o3-mini-2025-01-31 as the automatic extractor and judge (Scale leaderboard methodology, accessed 2026-07-15).

Contamination defenses. The public questions are paired with a private held-out set used to detect overfitting (HLE paper). At the April 2025 finalization, questions flagged by a bug-bounty program and questions answerable by web search were removed and replaced (official changelog). On 2025-10-08 the maintainers released HLE-Rolling, a dynamic fork of the benchmark (official changelog). See benchmark-contamination for the general failure mode these mechanisms target.

Publication. Introduced via arXiv in January 2025 (arXiv:2501.14249); published in Nature (Vol. 649, pp. 1139–1146) on 2026-01-28 as "A benchmark of expert-level academic questions to assess AI capabilities" (Nature).

Current results

Launch snapshot from the paper's Table 1, January 2025 (HLE paper):

Model (version)	Accuracy
GPT-4o (gpt-4o-2024-11-20)	3.3%
Grok 2 (grok-2-latest)	3.8%
Claude 3.5 Sonnet (claude-3-5-sonnet-20241022)	4.3%
Gemini 1.5 Pro (gemini-1.5-pro-002)	5.0%
Gemini 2.0 Flash Thinking (exp-1219)	6.2%
o1 (o1-2024-12-17)	9.1%
DeepSeek-R1 (2025-01-20 release)	9.4%

Scale's announcement summarized the launch state as models answering "fewer than 10 percent of the expert questions correctly" (Scale blog).

As of 2026-07-14, the #1 entry on Scale's official leaderboard is Gemini 3.1 Pro Preview (thinking high) at 46.44 ± 1.96% accuracy (Scale leaderboard, accessed 2026-07-14). The leaderboard is a live, client-side-rendered table with no versioned publication dates, so any number quoted from it decays quickly and must carry an access date. The trajectory — under 10% in January 2025 to above 46% in July 2026, roughly a 5x climb in 18 months — is the benchmark's most instructive output: HLE was built because MMLU saturated, and it is now visibly consuming its own headroom.

Limitations and critiques

• Static single-turn QA, not agentic capability. A model answers each question directly; there is no tool use, environment, or long-horizon execution. An HLE score measures expert knowledge and closed-form reasoning, not the ability to operate as an agent — for that, see osworld, terminal-bench, or swe-bench. No published evidence establishes how well HLE scores predict agent performance.
• Adversarial selection against launch-era models. Questions were pre-filtered to those that stumped frontier LLMs of late 2024 (HLE paper), so the difficulty distribution is defined relative to a specific model generation rather than an absolute scale.
• Severe overconfidence at launch — currency unknown. In the January 2025 evaluation, models paired sub-10% accuracy with RMS calibration error between 81.8% (DeepSeek-R1) and 93.9% (Gemini 2.0 Flash Thinking) (HLE paper). These figures are a launch snapshot; no comparable calibration numbers for 2026-era models have been published on the official pages as of 2026-07-15, so the "models are wildly overconfident on HLE" claim should not be repeated as a current fact.
• Judge-mediated grading. Correctness is decided by an LLM judge rather than string match — GPT-4o in the paper's launch evaluation (HLE paper), o3-mini-2025-01-31 on the current leaderboard (Scale leaderboard methodology, accessed 2026-07-15) — which inherits the usual llm-as-judge costs: format-boundary disagreements and judge errors are folded silently into reported accuracy.
• Both the question set and the judge changed under the leaderboard. Launch results were measured before the April 2025 finalization removed and replaced flawed and searchable questions (official changelog), and were graded by a GPT-4o judge, whereas the current leaderboard grades with o3-mini-2025-01-31 (Scale leaderboard methodology, accessed 2026-07-15) — so longitudinal comparisons across those boundaries, including the sub-10% → 46.44% trajectory, are approximate.

How to read the scores

• Date every number. HLE is the fastest-decaying headline number in evaluation: sub-10% to 46% in 18 months. A score without an access date is close to meaningless, and the official leaderboard renders client-side with no publication timestamps (Scale leaderboard, accessed 2026-07-14).
• Mind the confidence interval. The #1 entry carries a ±1.96-point interval on 2,500 questions (Scale leaderboard, accessed 2026-07-14); differences of a couple of points between entries carry little signal, a point Scale's own launch commentary conceded when attributing small score variations to randomness (Scale blog).
• Don't buy agentic conclusions. HLE ranks models on closed-ended academic QA. Choosing a model to run a coding or computer-use harness on the basis of an HLE delta conflates knowledge recall with tool-driven execution; check agentic benchmarks (swe-bench, osworld, terminal-bench) for that decision.
• Treat it as a headroom gauge, not a capability ceiling. The benchmark's own history — built to replace saturated MMLU, now half-consumed — is the cleanest public illustration that benchmark discriminating power is a depreciating asset. When HLE saturates, its successor will inherit the same curve.

Related

• benchmark-contamination — the failure mode HLE's held-out set, bug bounty, and HLE-Rolling fork are designed to resist
• llm-as-judge — HLE grading is judge-mediated in every protocol (GPT-4o at launch, o3-mini on the current leaderboard)
• swe-bench — agentic coding benchmark; the complementary axis to HLE's closed-ended QA
• osworld — computer-use benchmark for GUI-level agent capability
• terminal-bench — command-line agent benchmark
• evaluation-awareness — a confound for any benchmark whose questions circulate publicly