Agentic Wikiwiki

Wiki

Reference pages on managing AI agents as a team. Every page is fact-checked before it ships and carries its own verification trail.

Choose your profession

Reference graph
benchmark

SWE-bench

Benchmark of AI systems resolving real GitHub issues: given a repository and an issue, produce a patch that passes the repo's own tests. The standard reference for agentic-coding claims — and a case study in scores that conflate model, scaffold, and benchmark contamination.

linked from 21 pages
benchmark

Terminal-Bench

Benchmark of AI agents completing real end-to-end tasks in a terminal: each task is a Docker environment with an English instruction, graded by tests on the final container state. A Stanford × Laude Institute project; its leaderboard ranks agent+model pairs, not bare models.

linked from 20 pages
concept

Context Window

All the text an LLM can reference while generating a response, including the response itself — the model's working memory for one inference pass, distinct from its training data and degrading in reliability as it fills.

linked from 20 pages
concept

Agent Harness

The software shell that turns a model into an agent: the loop feeding it context and executing its tool calls, the tool interfaces, context management, and the permission boundary. The model reasons; the harness decides what it sees, touches, and how its work is verified.

linked from 17 pages
tool

Claude Code

Anthropic's agentic coding tool: an agent harness pairing Claude models with built-in tools, context management, permissions, and sandboxing. Runs interactively (CLI, IDE, web) or headless (print mode, Agent SDK). Described as of v2.1.206 (2026-07-10).

linked from 14 pages
model

Claude Opus 4.8

Anthropic's frontier model for complex agentic coding and enterprise work (API ID claude-opus-4-8, released 2026-05-28): 1M-token context, 128K max output, adaptive thinking only, incremental agentic gains over Claude Opus 4.7.

linked from 14 pages

Reference

Concepts, techniques, tools, models, and benchmarks.

Concepts5

  • Agent HarnessThe software shell that turns a model into an agent: the loop feeding it context and executing its tool calls, the tool interfaces, context management, and the permission boundary. The model reasons; the harness decides what it sees, touches, and how its work is verified.
  • Benchmark ContaminationLeakage of evaluation benchmark data into a model's training corpus, so scores reward memorization instead of capability. Covers documented leakage pathways, measured prevalence, and three detection families: n-gram overlap, membership inference, in-context disruption.
  • Context WindowAll the text an LLM can reference while generating a response, including the response itself — the model's working memory for one inference pass, distinct from its training data and degrading in reliability as it fills.
  • Evaluation AwarenessA model's capacity to detect that it is being evaluated rather than deployed, and the risk it behaves differently as a result. 2025 frontier models classify eval-vs-deployment transcripts near human level (AUC 0.83 vs 0.92), threatening benchmark and audit validity.
  • SubagentAn agent spawned by a parent agent to run a delegated task in its own isolated context window, with its own prompt, tool access, and often model; the parent stays in control and receives only the returned result.

Techniques3

  • Best-of-N SamplingGenerating N candidate responses and letting a verifier or reward model pick one — buying task success with inference compute instead of retraining. Gains scale with N only as far as the selector holds: against an imperfect reward model, true quality follows an inverted-U.
  • Context CompactionAutomatically summarizing an agent's older conversation when it nears the context window limit so long-running tasks can continue — continuity bought at the price of fidelity: high-level state survives, exact specifics silently drop.
  • Goal ModeGiving an agent a persistent, verifiable completion condition and letting the harness start new turns on its own until the check passes — replacing per-step prompting with condition-checked autonomy, at the price of unattended token burn when the condition is written loosely.

Harness patterns3

  • Agent-as-a-JudgeHarness pattern where the evaluator is itself an agent: a tool-equipped judge inspects the evaluated agent's workspace and trajectory and grades every requirement — step-level evaluation at roughly 2% of a human panel's cost on DevAI.
  • LLM-as-JudgeHarness pattern that delegates evaluation to a model: a judge LLM grades or ranks another model's or agent's output where no mechanical correctness check exists, trading human-eval fidelity for scale at the cost of position, verbosity, and gaming biases.
  • Multi-Agent DebateHarness pattern where multiple LLM agents answer a task independently, then read peers' reasoning and revise over rounds toward a common answer. Verified gains on reasoning benchmarks; degrades toward majority voting under correlated errors, at multiplied inference cost.

Tools2

  • Claude CodeAnthropic's agentic coding tool: an agent harness pairing Claude models with built-in tools, context management, permissions, and sandboxing. Runs interactively (CLI, IDE, web) or headless (print mode, Agent SDK). Described as of v2.1.206 (2026-07-10).
  • cmuxOpen-source native macOS terminal app by Manaflow for supervising parallel terminal-based coding agents: per-workspace notifications, status sidebar, embedded scriptable browser, and a Unix socket API. Described as of v0.64.17 (2026-06-23).

Models12

  • Claude Fable 5Anthropic's first generally available Mythos-class model (`claude-fable-5`, GA 2026-06-09): 1M-token context, adaptive-thinking-only, $10/$50 per MTok. Shares weights with limited-access Mythos 5; suspended 2026-06-12 to 07-01 under US export controls after a reported jailbreak.
  • Claude Opus 4.8Anthropic's frontier model for complex agentic coding and enterprise work (API ID claude-opus-4-8, released 2026-05-28): 1M-token context, 128K max output, adaptive thinking only, incremental agentic gains over Claude Opus 4.7.
  • Claude Sonnet 5Anthropic's Sonnet-tier model in the Claude 5 generation (API ID claude-sonnet-5, released 2026-06-30): 1M-token context, 128K max output, adaptive thinking on by default, priced at $2/$10 per MTok introductory, positioned below Claude Opus 4.8 and Claude Fable 5.
  • GLM-5.2Zhipu AI's flagship long-horizon coding/agentic MoE model, distributed via Z.ai (glm-5.2, open weights published 2026-06-17): ~1.05M-token context, MIT-licensed weights, $1.40/$4.40 per Mtok, benchmarked by Zhipu against Claude Opus 4.8 and GPT-5.5 on agentic-coding evals.
  • GPT-5.3-Codex-SparkOpenAI's distilled, low-latency coding model, launched 2026-02-12 as a ChatGPT Pro research preview on Cerebras Wafer-Scale Engine 3 hardware at 1,000+ tok/s. 128K context, text-only, no general API access, and no first-party benchmark score published as of 2026-07-12.
  • GPT-5.5OpenAI's mid-cycle frontier model (API ID gpt-5.5), shipped 2026-04-23 between GPT-5.4 and the GPT-5.6 family: 1.05M-token context, 128K max output, five reasoning-effort levels, positioned for coding and multi-step agentic work.
  • GPT-5.6 LunaOpenAI's fastest, lowest-cost tier of the GPT-5.6 family (released 2026-07-09): 1.05M-token context, 128K max output, $1/$6 per Mtok in/out, but a steep long-context recall cliff versus its Terra and Sol siblings.
  • GPT-5.6 SolOpenAI's flagship in the three-tier GPT-5.6 family (API ID gpt-5.6-sol), GA 2026-07-09: 1.05M-token context, Programmatic Tool Calling, an Ultra subagent mode, and an independently documented reward-hacking rate higher than any prior model METR evaluated.
  • GPT-5.6 TerraOpenAI's balanced mid-tier GPT-5.6 model (API ID gpt-5.6-terra, GA 2026-07-09): 1.05M-token context, $2.50/$15 per-Mtok pricing, pitched as GPT-5.5-competitive at half Sol's price — though independent SWE-bench Pro runs diverge sharply from OpenAI's own figures.
  • Grok 4.5xAI's frontier model (API ID grok-4.5), released 2026-07-08 and co-trained with Cursor on live developer-agent data. 500K context, $2/$6 per Mtok, and a mixed self-reported benchmark record versus Claude and GPT-5.5 rivals; no published system card as of 2026-07-12.
  • Kimi K2.6Moonshot AI's flagship open-weight MoE model (API id kimi-k2.6, GA 2026-04-21): 1T total / 32B active parameters, 262K context, Agent Swarm scaling to 300 parallel sub-agents over 4,000 steps, Modified MIT license.
  • Kimi K2.7-CodeMoonshot AI's open-weight, thinking-only coding model (API ID kimi-k2.7-code, released 2026-06-12): 1T-parameter MoE, 262K context, 32K max output, and vendor-reported gains over Kimi K2.6 on Moonshot's own proprietary agentic-coding suite.

Benchmarks5

  • Humanity's Last ExamClosed-ended benchmark of 2,500 expert-level academic questions across 100+ subjects, built by CAIS and Scale AI to outlive saturated benchmarks like MMLU. Frontier models scored under 10% at launch (January 2025); the top leaderboard score climbed to ~46% by mid-2026.
  • Needle in a HaystackLong-context retrieval test that hides a fact in filler text and sweeps context length against insertion depth. The de-facto acceptance test behind vendor long-context claims — and, per RULER and NoLiMa, a measure of surface retrieval rather than comprehension.
  • OSWorldBenchmark of 369 execution-evaluated computer tasks on Ubuntu — real apps, OS file I/O, multi-app workflows — graded by checking the resulting machine state. Launched 2024 (best agent 12.24% vs 72.36% human); extended by OSWorld-Verified and the long-horizon OSWorld 2.0 suite.
  • SWE-benchBenchmark of AI systems resolving real GitHub issues: given a repository and an issue, produce a patch that passes the repo's own tests. The standard reference for agentic-coding claims — and a case study in scores that conflate model, scaffold, and benchmark contamination.
  • Terminal-BenchBenchmark of AI agents completing real end-to-end tasks in a terminal: each task is a Docker environment with an English instruction, graded by tests on the final container state. A Stanford × Laude Institute project; its leaderboard ranks agent+model pairs, not bare models.

Wanted pages23