Accuracy on the hard battery
The hard battery is the twenty questions where matching text alone cannot reach the answer: fifteen that need several components understood together (multi-hop chains, absence proofs, dispatch, cross-stack tracing), and five with a false premise the arm must recognise and refuse.
Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, grouped by arm, for how one arm moves across tiers. By model is the transpose: one group per model with every arm inside it, Corograph first, for the like-for-like read at each tier at a glance; the line under each group is Corograph against that ecosystem's bare harness. This choice applies to every figure on the site and is remembered.
Data behind this figure — ZeroShot + MultiTurn · hard battery (20 q)
| arm | role | column | accuracy on the hard battery (% of questions correct) | runs | cells | min | max | sd | rank | runs in the repository |
|---|---|---|---|---|---|---|---|---|---|---|
| Corograph | Corograph | 8 models | 87.6% | 159 | 16 | 65.0% | 100.0% | 8.6% | 1st of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| CC Bare | bare harness | 6 models | 83.0% | 103 | 11 | 40.0% | 100.0% | 12.1% | 6th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Codex Bare | bare harness | 3 models | 73.5% | 60 | 6 | 55.0% | 90.0% | 7.2% | 9th of 9 | GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| Cursor Bare | bare harness | 8 models | 80.9% | 160 | 16 | 55.0% | 100.0% | 10.1% | 7th of 9 | Haiku 4.5 · Sonnet 5 · Opus 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol · Grok 4.6 · Composer 2.5 |
| Graphify | indexing tool on Claude Code | 6 models | 83.1% | 103 | 11 | 40.0% | 100.0% | 12.2% | 5th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| GitNexus | indexing tool on Claude Code | 6 models | 84.3% | 103 | 11 | 55.0% | 100.0% | 8.4% | 2nd of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodeGraph | indexing tool on Claude Code | 6 models | 80.4% | 104 | 11 | 40.0% | 95.0% | 13.3% | 8th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodebaseMemory | indexing tool on Claude Code | 6 models | 83.8% | 103 | 11 | 60.0% | 95.0% | 8.8% | 4th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Serena | indexing tool on Claude Code | 6 models | 84.3% | 108 | 12 | 50.0% | 100.0% | 11.3% | 3rd of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
Figure 3. The hard battery is the twenty questions where matching text alone cannot reach the answer: fifteen that need several components understood together (multi-hop chains, absence proofs, dispatch, cross-stack tracing), and five with a false premise the arm must recognise and refuse. Same statistic and same reading as Figure 2, restricted to that battery: every dot is one run, the tick its pooled mean, the thin line its range; the battery control above lets you put the base battery back. On the base battery every arm sits near the ceiling; here the tools separate, and Corograph's cloud sits highest at every shared model.
How this figure is computed
Definition and aggregation as in Figure 2, with the battery fixed to the hard battery (denominator 20). The base battery is excluded by default because its ceiling compresses every arm into the same band; the difference between an index and a search shows where the questions were designed to need one.
Reading it. Higher is better. The false-premise questions count a refusal as correct and an invented answer as wrong, so an arm that answers confidently from text matches is penalised exactly where a user would be misled.
Provenance
Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.29, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.