Figure 15 · Epoch-4

Tool result tokens per run

Tool result tokens are the volume of text the arm's tools pushed back into the model's context over a run — file contents, search hits, shell output, or an index query's result — estimated per call from the result size.

protocols
battery
model
columns

Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, grouped by arm, for how one arm moves across tiers. By model is the transpose: one group per model with every arm inside it, Corograph first, for the like-for-like read at each tier at a glance; the line under each group is Corograph against that ecosystem's bare harness. This choice applies to every figure on the site and is remembered.

Tool result tokens per runlower is betterZeroShot + MultiTurn · both batteries
0500k1.0M1.5MCC Bare 125kCodex Bare 240k170k59kHaiku 4.580k21kSonnet 576k26kOpus 4.666k23kOpus 4.897k42kOpus 548k21kFable 5269k119kLuna186k77kTerra178k97kSolCorographCorograph125k3kCC Bare240k35kCodex Bare142k42kHaiku 4.574k19kSonnet 596k25kOpus 5301k44kLuna453k50kTerra718k52kSol210k66kGrok 4.6248k89kComp 2.5Cursor Barebare harness130k10kGraphify89k10kGitNexus128k7kCodeGraph82k10kCodebaseMemory109k9kSerena

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Data behind this figure — ZeroShot + MultiTurn · both batteries
armrolecolumntool result tokens per run
(estimated)		runscellsminmaxsdrank
(at this model)		runs in the repository
CorographCorographHaiku 4.5170k39459k292k83k2nd of 2Haiku 4.5
Sonnet 580k40421k151k43k2nd of 2Sonnet 5
Opus 4.6 ★76k40426k132k34k1st of 1Opus 4.6
Opus 4.8 ★66k40423k101k25k1st of 1Opus 4.8
Opus 597k40442k162k40k2nd of 2Opus 5
Fable 5 ★48k11221k77k22k1st of 1Fable 5
GPT 5.6 Luna ★269k404119k413k82k1st of 2GPT 5.6 Luna
GPT 5.6 Terra ★186k40477k269k57k1st of 2GPT 5.6 Terra
GPT 5.6 Sol ★178k40497k276k48k1st of 2GPT 5.6 Sol
CC Barebare harness6 models125k201223k536k147k4th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Codex Barebare harness3 models240k1201235k496k156k7th of 7GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol
Cursor Barebare harnessHaiku 4.5 ★142k40442k617k112k1st of 2Haiku 4.5
Sonnet 5 ★74k40419k156k45k1st of 2Sonnet 5
Opus 5 ★96k40425k370k74k1st of 2Opus 5
GPT 5.6 Luna301k40444k883k222k2nd of 2GPT 5.6 Luna
GPT 5.6 Terra453k40450k1.29M280k2nd of 2GPT 5.6 Terra
GPT 5.6 Sol718k40452k1.22M368k2nd of 2GPT 5.6 Sol
Grok 4.6 ★210k40466k585k132k1st of 1Grok 4.6
Composer 2.5 ★248k40489k600k146k1st of 1Composer 2.5
Graphifyindexing tool on Claude Code6 models130k2012210k517k144k6th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
GitNexusindexing tool on Claude Code6 models89k2012210k527k83k2nd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodeGraphindexing tool on Claude Code6 models128k203227k414k111k5th of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
CodebaseMemoryindexing tool on Claude Code6 models82k2012210k345k83k1st of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5
Serenaindexing tool on Claude Code6 models109k207239k445k111k3rd of 7Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5

Figure 15. Tool result tokens are the volume of text the arm's tools pushed back into the model's context over a run — file contents, search hits, shell output, or an index query's result — estimated per call from the result size. Every dot is one run, split per model for every arm; the solid tick is the mean and the thin line the range. This is the mechanism behind Figure 1 seen from the tool side: the index returns a small, precise answer where a search returns pages to read.

How this figure is computed

Definition. tool_result_tokens(r) = Σ over the run's tool calls of the result's token estimate, taken from the capture where the harness recorded one and otherwise result bytes ÷ 4. For Corograph index queries the estimate is the size of the result the model received.

Aggregation. None for the dots; the mean tick is the equal-weight mean of battery × model × protocol cells. Raw token counts are never pooled across vendors.

Reading it. Lower is better. Read it beside Figure 9 (tool calls): fewer calls returning less text is the signature of a good index.

Provenance

Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.29, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.