What this benchmark does not show, and what it shows with caveats

Everything below is also stated where it applies; this page collects it so nothing has to be found.

Scope

One repository under test, at one commit, with two question batteries. The batteries were authored by a frontier model with no access to Corograph and verified afterwards (how the questions were made); the benchmark's owner approved and froze them, and selection of the final set is a judgment the owner made. The questions are about code structure and behaviour (endpoints, schema, types, call paths, impact, cross-stack tracing, absence proofs, false premises); they are not a coding task benchmark and say nothing about editing or generating code.

One Corograph instrument freeze per epoch. The comparison group is measured as shipped, prompt assets and all; Claude Code cannot be forced to use an installed tool, so what is measured for the indexing tools is organic adoption. That is the measurand, not a limitation of the measurement.

Execution asymmetries

The Corograph arm executed on the host through its gateway; the comparison group executed in containers on the same machine. Accuracy, tokens, turns and tool counts are location-independent; wall-clock time is not, and Figure 10 is published for completeness rather than as a claim. Runs executed in parallel lanes shared the host; per-lane contention is not modelled.

Cursor Agent's model traffic transits Cursor's backend, and Codex runs inside its own harness loop; both are measured as their vendors ship them. Their token figures are the harness's own per-call usage with the derivation named in each run record.

Tokens and cost

Raw token counts are never pooled across vendors, because tokenizers differ; figures split by model whenever a selection spans vendors. Cost is the vendor-reported figure where the harness reported one (Claude Code's own result envelope) and otherwise an estimate from the published rate tables (epochs/…/summary/rate-tables.json) over the run's four token columns — list prices, so an arm on a subscription plan may pay less in practice; every run row names its cost_basis, and a model with no rate row has no cost and never enters a cost mean. Codex reports no cache-write split, so its estimate charges written tokens at the input rate, a disclosed floor.

Scoring

Tier T1 is a mechanical token-subset match that can only pass an answer. Tier T2 is an LLM judge under a fixed prompt with per-question exemplars, one candidate per call, whose reason is published beside every verdict; a verdict whose reason contradicts it is retried once and, if still contradictory, marked adjudicate and counted as a fail. The judge is an LLM and can be wrong; the per-question answers and verdicts are published so that any reader can re-grade. Every verdict in the epoch sits under one scorer era.

Did-not-finish runs

A run that did not complete every question is published with validity.ok = false and no score. It never enters a mean. The counts per arm in this epoch:

A partial exam is most often a single question that never completed in a per-question protocol; the run's validity.detail names it. By model and protocol:

Model-specific caveats

Fable 5 refused many questions. At Fable 5 the model declined to answer a large share of the per-question prompts — plain questions about a public codebase's structure, nothing touching security — so those runs did not finish and carry no score: 79 of 119 ZeroShot runs at Fable 5 are did-not-finish across the arms that ran it. Fable 5 is therefore a thin roster in this epoch (23 Corograph runs in total) and its columns should be read with their n; it is excluded from nothing, but it decides nothing either.

Haiku 4.5 and the whole-battery protocol. HumanExam presents the entire battery in one prompt, and at Haiku 4.5's 200K-token context window the accumulated tool results of a full battery can exceed the window. 5 of 25 HumanExam runs at Haiku 4.5 did not finish for that reason (all 5 on the Corograph arm, whose index results are large). The per-question protocols do not have this exposure and are the ones to read at Haiku 4.5.

Opus 4.8 in the ZeroShot protocol. The comparison group's ZeroShot runs at Opus 4.8 stop after ten valid repetitions per arm with twenty did-not-finish runs each; the run records carry the failing question in validity.detail. Those runs are published unscored like every other did-not-finish run and never enter a mean.

Capture and provenance

Capture gaps are disclosed per run in run.json and are never silently filled. Runs whose units were resumed across hosts or image builds list every vintage. The capture host's name appears in provenance; home directories are normalised; nothing else about the operators' machines is published.

What is withheld

The text of the queries the Corograph agent sends to its codebase index and the raw result envelopes, replaced in transcripts by the product's own display sentence with latency and result size; and the internal code names of Corograph components. Comparison-group transcripts are verbatim. The full policy is redaction/POLICY.md in the repository.

Per-figure disclosures

Figure 1 — Context tokens per run

Exam protocols have different token denominators (a MultiTurn question replays its history; ZeroShot pays the fixed prefix per question). Switch protocols with the control above; the site never pools across them.

A run whose token facts were not captured is excluded from its column, never counted as zero; the table's n is the number of runs behind each column.

Figure 7 — Outcome composition per run

Did-not-finish runs have no score and do not appear here; their count per arm is in the run records and in the Limitations page.

Figure 10 — Wall time per run

The Corograph arm executed on the host through its gateway; the comparison group executed in containers on the same machine. Accuracy, tokens and tool counts are location-independent; wall time is not, and the figure is published for completeness rather than as a claim.

Runs executed in parallel lanes shared the host; per-lane contention is not modelled.

Figure 13 — Cost per correct answer

Vendor-reported and estimated costs sit in the same column; the table beneath the figure carries the basis per row and the rate tables are published beside the run rows.

Figure 16 — Organic index adoption by model

Main thread only: MCP tools are not offered to sub-agent threads, so a sub-agent's reads and searches do not enter this figure. HumanExam is whole-battery and has no per-question tool attribution; it is not shown.

MultiTurn is reported per question; one conversation carries a whole battery, so a conversation that consulted the index once early is counted at the questions where it did so. The table reports the per-conversation figure beside it (conversations with at least one index call).

Graphify ships as a skill: its graph was built before the session and the skill invocation is the index call; reads of the graph files by other means are not distinguishable from ordinary file reads and are not counted.

CodebaseMemory indexes on demand: its index was not built before the session, so its index_repository and index_status calls are setup calls, excluded here and listed in the summary file. Serena has no prebuilt index by design; its project activation and instruction calls are setup calls likewise.

The share says how often the index was called, not whether its answer was used: a run that queried the index and then read the source is counted the same as one that answered from the query. The tool sequences themselves are in each run's transcript.