Figure 8 · Epoch-4

Context tokens per correct answer

This divides a run's context tokens (Figure 1) by the number of questions it answered correctly, so an arm that reads less but also answers less is not rewarded.

protocols
battery
model
columns

Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, grouped by arm, for how one arm moves across tiers. By model is the transpose: one group per model with every arm inside it, Corograph first, for the like-for-like read at each tier at a glance; the line under each group is Corograph against that ecosystem's bare harness. This choice applies to every figure on the site and is remembered.

Context tokens per correct answerlower is betterZeroShot + MultiTurn · both batteries
0200k400k600kCC Bare 454kCodex Bare 245k★448kHaiku 4.5★156kSonnet 5★126kOpus 4.6★134kOpus 4.8★140kOpus 5★63kFable 5★232kLuna★148kTerra131kSolCorographCorograph454k6 modelsCC Bare245k3 modelsCodex Bare712kHaiku 4.5439kSonnet 5366kOpus 5331kLuna167kTerra★113kSol★312kGrok 4.6★432kComp 2.5Cursor Barebare harness467k6 modelsGraphify465k6 modelsGitNexus359k6 modelsCodeGraph457k6 modelsCodebaseMemory502k6 modelsSerena

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Data behind this figure — ZeroShot + MultiTurn · both batteries
armrolecolumncontext tokens per correct answerrunscellsminmaxsdrank
(at this model)		runs in the repository
CorographCorographHaiku 4.5 ★448k39484k1.35M378k1st of 2Haiku 4.5
Sonnet 5 ★156k40451k399k109k1st of 2Sonnet 5
Opus 4.6 ★126k40427k434k115k1st of 1Opus 4.6
Opus 4.8 ★134k40429k425k122k1st of 1Opus 4.8
Opus 5 ★140k40446k311k78k1st of 2Opus 5
Fable 5 ★63k11226k106k34k1st of 1Fable 5
GPT 5.6 Luna ★232k40429k890k254k1st of 2GPT 5.6 Luna
GPT 5.6 Terra ★148k40422k509k154k1st of 2GPT 5.6 Terra
GPT 5.6 Sol131k40422k455k129k2nd of 2GPT 5.6 Sol
CC Barebare harness6 models454k20122112k2.17M382k3rd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Codex Barebare harness3 models245k1201272k901k165k1st of 7GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol
Cursor Barebare harnessHaiku 4.5712k40480k1.49M396k2nd of 2Haiku 4.5
Sonnet 5439k404172k988k243k2nd of 2Sonnet 5
Opus 5366k404219k865k158k2nd of 2Opus 5
GPT 5.6 Luna331k404157k823k161k2nd of 2GPT 5.6 Luna
GPT 5.6 Terra167k40470k438k94k2nd of 2GPT 5.6 Terra
GPT 5.6 Sol ★113k40466k369k53k1st of 2GPT 5.6 Sol
Grok 4.6 ★312k404143k594k140k1st of 1Grok 4.6
Composer 2.5 ★432k404179k843k205k1st of 1Composer 2.5
Graphifyindexing tool on Claude Code6 models467k20122119k2.04M380k6th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
GitNexusindexing tool on Claude Code6 models465k20122163k1.95M344k5th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodeGraphindexing tool on Claude Code6 models359k2032297k1.97M288k2nd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodebaseMemoryindexing tool on Claude Code6 models457k20122136k1.76M330k4th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Serenaindexing tool on Claude Code6 models502k20723138k2.09M417k7th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5

Figure 8. This divides a run's context tokens (Figure 1) by the number of questions it answered correctly, so an arm that reads less but also answers less is not rewarded. Each column is the mean over valid, scored runs at one model (or pooled over a single-vendor roster in the pooled-models view), lower is better, and Corograph and the bare harnesses are the dashed references. The ordering of Figure 1 survives the normalisation: the token saving is a saving per correct answer, not a saving bought with wrong answers.

How this figure is computed

Definition. For a scored run with score(r) > 0, ctx_per_correct(r) = context(r) / score(r). Runs with no correct answer have no defined value and are excluded from the mean; their count is in the table.

Aggregation. As in Figure 1: cell means averaged with equal weight; raw token counts are never pooled across vendors.

Reading it. Lower is better. Cost in currency is deliberately not shown here: the package publishes cost only where a vendor reported it, and Corograph's runs carry no vendor-reported figure. Tokens are the quantity a reader can recompute from the run records.

Provenance

Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.29, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.