Run-to-run variation of accuracy
Consistency asks how far the same tool at the same model moves when the same questions are run again: the coefficient of variation of run accuracy, in percent, within a battery × model × protocol cell.
Pooled models shows one column per arm, its models averaged with equal weight — the readable overview. Per model shows one column per model for every arm, grouped by arm, for how one arm moves across tiers. By model is the transpose: one group per model with every arm inside it, Corograph first, for the like-for-like read at each tier at a glance; the line under each group is Corograph against that ecosystem's bare harness. This choice applies to every figure on the site and is remembered.
Data behind this figure — ZeroShot + MultiTurn · both batteries
| arm | role | column | run-to-run variation of accuracy (coefficient of variation, %) | runs | cells | min | max | sd | rank | runs in the repository |
|---|---|---|---|---|---|---|---|---|---|---|
| Corograph | Corograph | 9 models | 3.8% | 330 | 34 | 65.0% | 100.0% | 8.2% | 1st of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| CC Bare | bare harness | 6 models | 5.0% | 201 | 22 | 40.0% | 100.0% | 11.5% | 7th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Codex Bare | bare harness | 3 models | 5.5% | 120 | 12 | 55.0% | 100.0% | 13.6% | 9th of 9 | GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol |
| Cursor Bare | bare harness | 8 models | 4.4% | 320 | 32 | 55.0% | 100.0% | 11.4% | 4th of 9 | Haiku 4.5 · Sonnet 5 · Opus 5 · GPT 5.6 Luna · GPT 5.6 Terra · GPT 5.6 Sol · Grok 4.6 · Composer 2.5 |
| Graphify | indexing tool on Claude Code | 6 models | 5.0% | 201 | 22 | 40.0% | 100.0% | 11.6% | 8th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| GitNexus | indexing tool on Claude Code | 6 models | 4.3% | 201 | 22 | 55.0% | 100.0% | 9.2% | 3rd of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodeGraph | indexing tool on Claude Code | 6 models | 4.7% | 203 | 22 | 40.0% | 100.0% | 12.9% | 6th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| CodebaseMemory | indexing tool on Claude Code | 6 models | 4.1% | 201 | 22 | 60.0% | 100.0% | 9.7% | 2nd of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
| Serena | indexing tool on Claude Code | 6 models | 4.6% | 207 | 23 | 50.0% | 100.0% | 10.9% | 5th of 9 | Haiku 4.5 · Sonnet 5 · Opus 4.6 · Opus 4.8 · Opus 5 · Fable 5 |
Figure 4. Consistency asks how far the same tool at the same model moves when the same questions are run again: the coefficient of variation of run accuracy, in percent, within a battery × model × protocol cell. Each column is that variation averaged with equal weight over the arm's cells (or over its roster in the pooled-models view); lower is steadier, and Corograph and the bare harnesses are the dashed references. Corograph's columns are the lowest: the index does not only raise the mean of Figure 2, it narrows the spread a user experiences run to run.
How this figure is computed
Definition. Within a cell (arm × battery × model × exam) with scored runs of accuracies a₁…aₙ, cv = 100 · sd(a) / mean(a), with the n−1 standard deviation; a cell with one run has no spread and contributes nothing. The statistic is dimensionless, so cells at different means compare directly.
Aggregation. Cells averaged with equal weight over the selected batteries, models and protocols. A cell whose mean is zero has no defined variation and is excluded.
Reading it. Lower is better. Two arms with the same mean accuracy and different variation are not the same instrument: the variation is the part of the result that depends on the run rather than on the tool.
Provenance
Computed at build time from epochs/E004/summary/runs.json of the proof package (export tool tabench export public 0.1.29, scorer era tabench-1.1). Each row of the table above links to the run directories in the repository, where run.json, the answers, the verdicts and the transcript of every run can be read.