Needle in a Haystack

Needle in a Haystack (NIAH) is a long-context retrieval test: a short fact (the "needle") is inserted into a large body of unrelated filler text (the "haystack"), and the model must retrieve it — repeated across a grid of context lengths and insertion depths to produce the familiar green-and-red recall heatmap. Greg Kamradt created it as a practitioner diagnostic in November 2023 — a GitHub repository and posts, not a paper — with original runs against GPT-4-128K on 2023-11-08 and Claude 2.1 on 2023-11-21 (original repository). It hardened into the de-facto acceptance test behind vendor long-context claims, most visibly in Google's Gemini 1.5 report (arXiv:2403.05530). Peer-reviewed successors — RULER (COLM 2024) and NoLiMa (ICML 2025) — then showed that a perfect NIAH heatmap certifies only a superficial form of retrieval, largely solvable by literal string matching.

Methodology

The harness "runs a sweep of (context length × needle depth) cells against any configured model, scores each response, and writes one result row per cell to a JSONL file" (repository README). Each cell fixes two parameters: the total context length and depth_percent, the needle's position within the filler text. Per-cell rows record the score, token usage, cost and duration; plotted together they form the depth-versus-length heatmap that vendors reproduce in model reports.

Because both needle and haystack are arbitrary, the test is a protocol, not a fixed dataset: needles can be regenerated at will, which makes classic benchmark-contamination a non-issue for NIAH — its characteristic failure mode is saturation instead (see critiques below).

Details as of repository v2.0.0, released 2026-05-30 (README, accessed 2026-07-15):

• Task types: single-fact retrieval, multi-fact, UUID retrieval, and UUID-chain reasoning, scored by exact or fractional match depending on type.
• Multi-needle variant: an EvenSpreadInserter places the first needle at depth_percent and the rest at intervals of (100 − depth_percent) / N — e.g. 10 needles starting at depth 40% land at 40, 46, 52 … 94.
• Providers: OpenAI, Anthropic and Cohere out of the box, with a plugin model for others.

Reported results

No current leaderboard exists — NIAH is run per-vendor and per-paper, so every number below is a dated snapshot from its own source, not a comparable series.

Result	System / cohort	Source, as of
Near-perfect "needle" recall (>99.7%) up to 1M tokens across text, video and audio; recall maintained when extending to 10M tokens in text (~7M words)	Gemini 1.5 Pro (vendor self-report)	Gemini 1.5 report, Fig. 1, 2024
"Nearly perfect accuracy in the vanilla NIAH test" — while the same models degrade sharply on harder long-context tasks	17 long-context models, 2024 cohort	RULER, COLM 2024
99.3% short-context baseline falling to 69.7% at 32K once lexical overlap between question and needle is removed	GPT-4o, best-performing of 13 models tested	NoLiMa, ICML 2025

The Gemini figure is the canonical example of the genre: a vendor-run, single-needle result presented as evidence of long-context capability — exactly the kind of claim the successor benchmarks were built to probe. The report itself pairs these synthetic "needle-in-a-haystack" probes (credited to Kamradt) with separate real-world long-context evaluations (Gemini 1.5 report).

Limitations and critiques

Saturation masks degradation (RULER). NVIDIA's RULER paper states that "this simple retrieval-based test is indicative of only a superficial form of long-context understanding": despite nearly perfect vanilla-NIAH accuracy, almost all evaluated models "exhibit large performance drops as the context length increases" on harder tasks (RULER). RULER extends NIAH into 13 task configurations across four categories — multi-needle retrieval variants, multi-hop tracing, aggregation, and question answering (RULER repository). Against a fixed quality threshold (Llama-2-7B's 85.6% at 4K), only half of the 17 tested models — all claiming 32K context or more — maintained satisfactory performance at 32K, and almost all fell below the threshold before their claimed length. As of the COLM 2024 cohort: Llama3.1-70B claimed 128K but measured effective at 64K (96.5% at 4K → 66.6% at 128K); DBRX claimed 32K but measured effective at 8K (95.1% at 4K → 0.0% at 128K) (RULER repository).

Solvable by literal matching (NoLiMa). NoLiMa (Adobe Research + LMU Munich, ICML 2025) shows that in vanilla NIAH "models can exploit existing literal matches between the needle and haystack to simplify the task" — high scores are achievable through surface string matching without long-context comprehension. When questions and needles share minimal lexical overlap, forcing models to infer latent associations, 11 of 13 models claiming ≥128K context drop below 50% of their short-context baselines at 32K; the paper attributes the decline to the attention mechanism's difficulty retrieving relevant content in longer contexts once literal anchors are absent (NoLiMa).

Effective context is shorter than advertised (Needle Threading). An independent line of evidence at near-million-token scale: across 17 models evaluated on thread-following retrieval tasks, "the effective context limit is significantly shorter than the supported context length, with accuracy decreasing as the context window grows" — though many models proved "remarkably threadsafe", following multiple simultaneous threads without significant loss (Needle Threading, ICLR 2025).

Degradation is non-uniform even on trivial tasks (Context Rot). Chroma's report (2025-07-14) evaluated 18 models — GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 families — and found that "model performance varies significantly as input length changes, even on simple tasks", that the impact of distractors is non-uniform and amplifies with input length, and that "whether relevant information is present in a model's context is not all that matters; what matters more is how that information is presented" (Context Rot). Provenance note: Chroma sells retrieval infrastructure, so its "curate, don't dump" conclusion aligns with its product — though the findings echo RULER and NoLiMa, which have no such stake. See context-rot.

How to read the scores

• Failure is signal; perfection is not. NIAH is a necessary-condition smoke test for long-context plumbing. A red cell at some depth/length is a real defect worth investigating; a fully green heatmap demonstrates only that surface retrieval works — three independent research lines (RULER, NoLiMa, Needle Threading) converge on this asymmetry.
• Treat vendor heatmaps as a floor, not a measurement. Vanilla NIAH was already saturated by the 2024 model cohort (RULER). When a release quotes NIAH recall, ask for RULER- or NoLiMa-class numbers before drawing capability conclusions.
• Budget effective context, not advertised context. For agent-harness design — context-compaction thresholds, memory placement, how much retrieved material to inject into a context-window — assume the usable window is materially shorter than the spec sheet, and that placement within it matters (Context Rot).
• Cross-vendor comparison is a category error. NIAH is a protocol family: needle content, haystack corpus, prompt and scoring all vary per run, so two vendors' heatmaps are different experiments, not two rows of one leaderboard.
• Its weakness is not contamination. Unlike static-dataset benchmarks such as swe-bench or humanitys-last-exam, synthetic regenerable needles leave little to memorize; NIAH fails by being too easy, not by leaking into training data.

Related

• context-rot — the practitioner-facing extension of the NIAH critique
• context-engineering — the discipline the "presentation matters more than presence" finding motivates
• terminal-bench and osworld — end-to-end agentic benchmarks, versus NIAH's single-capability smoke test
• evaluation-awareness — synthetic, out-of-place needles are precisely the kind of artifact a model can recognize as a test
• benchmark-contamination — the failure mode NIAH largely avoids by construction