Benchmark procedure · Epoch 004 — line-4 · public proof package

Does Corograph's codebase index make an AI coding agent more accurate, more consistent and cheaper than lexical search?

This site reports one benchmark epoch in which the Corograph Ask agent, the model plus Corograph's index of the codebase, built by its own proprietary indexing system and designed for LLM consumption, answered 50 questions about usememos/memos under three exam protocols, beside bare Claude Code, bare Codex, bare Cursor Agent, and five codebase-indexing tools installed on Claude Code, at the same models and from the same clean checkout. Every figure is computed from the run records in the public repository, and every number links to the runs behind it.

What is measured

The benchmark measures codebase understanding: whether an agent, asked a question about a codebase it has never seen, arrives at the correct answer. Understanding precedes development. Correct code follows from a correct reading of the existing code and from loading the right context; an agent that misreads the codebase writes against a wrong model of it. No protocol in this epoch asks the agent to write code; every question tests context-building alone. Protocols in development extend the same method to planning code changes (refactors and new features) and to executing plans.

The questions are published with their grading key (gold answers, accepted variants, scoring policies and the judge's exemplars). Every arm'sruns are published in full: the answer to each question, thejudge's verdict with its stated reason, the per-run tokens, cost, tool calls and time, and the session transcript.

What every arm could do

No arm was restricted. Each comparison arm ran in its own container with full permission to call every tool it ships with and any shell command, so grep, directory listing and file reading were available to all of them. The codebase-indexing tools were installed inside their containers as shipped, and whatever each install adds to the environment (an MCP server, skills, instruction files) was kept; the container definitions are published. The Corograph arm had the same file tools and one addition: a tool that queries Corograph's index of the codebase. The comparison is therefore between tool sets, at the same models, on the same questions.

Caveats

Fable 5 refused many questions. The questions concern the structure of a public codebase and touch nothing sensitive, yet the model declined a large share of them under the per-question protocols. Repeated attempts did not yield a complete set, so Fable 5 is a thin roster in this epoch and its columns should be read with their run counts. The exact figures are on theLimitations page.
Haiku 4.5 and the whole-battery protocol. Haiku 4.5 has a 200K-token context window where the other models have 1M. Under HumanExam, which presents the entire battery in one prompt, some Haiku 4.5 runs exceeded it and did not finish. They are published unscored and never enter a mean.
Planned additions. Anthropic's Fable 5.1 and OpenAI's ChatGPT-6 (Astra) will be added in a later epoch.

The proof package on GitHub — Corograph/benchmarks2,692 run directories: every question, every answer, every verdict with the judge's reason, every transcript, and the export that produced them. Every number on this site links into it.

2,692

runs published
2,480 valid and scored · 212 did not finish

arms
1 Corograph · 8 comparison

models
Anthropic, OpenAI, xAI, Cursor

61,770

question verdicts
one scorer era: tabench-1.1

Key results — ZeroShot and MultiTurn protocols, both batteries (Epoch-4)

Context. At every model both arms ran, bare Claude Code read 1.9× to 2.9× the context tokens Corograph read to answer the same battery (Haiku 4.5: 8.75M vs 16.53M; Sonnet 5: 3.37M vs 9.69M; Opus 4.6: 2.63M vs 7.03M; Opus 4.8: 2.86M vs 5.51M; Opus 5: 3.24M vs 6.37M; Fable 5: 1.90M vs 4.66M). — Figure 1
Accuracy. Corograph's pooled accuracy is 93.3% against 89.4% for bare Claude Code and 90.6% for the best indexing tool (GitNexus). — Figure 2
Reliability. 77% of Corograph's runs scored at least 90%, against 67% for bare Claude Code. — Figure 5

Pooled figures average protocol × battery × model cells with equal weight; see each figure's How this figure is computed. Change protocols, battery and model on every figure page. HumanExam (the whole battery in one prompt) is off by default because it is the stress probe, not the usage shape; switch it on to see it.

Headline figure

Context tokens per runlower is betterZeroShot + MultiTurn · both batteries

Raw token counts are never pooled across vendors (tokenizers differ), so arms whose selected models span vendors are shown one column per model; ★ marks the best value at each model.

Figure 1. Context tokens are the tokens the model read to answer a whole run: the uncached input plus every cache read, summed over every API call in the run, all threads included. Each column is the mean over the valid, scored runs of one arm at one model, every arm split by model because token counts are model facts; the dashed lines mark Corograph's and each bare harness's pooled value wherever pooling is licensed. At every model the Corograph column sits well below the same model running bare, and the indexing tools cluster at or above bare Claude Code: the index replaces reading. Controls, data table and method →

The figures

Figure 1Context tokens per runAt every model the Corograph column sits well below the same model running bare, and the indexing tools cluster at or above bare Claude Code: the index replaces reading.Figure 2Accuracy per runCorograph's distribution sits highest and tightest; the lower context use in Figure 1 is not bought with accuracy.Figure 3Accuracy on the hard batteryOn the base battery every arm sits near the ceiling; here the tools separate, and Corograph's cloud sits highest at every shared model.Figure 4Run-to-run variation of accuracyCorograph's columns are the lowest: the index does not only raise the mean of Figure 2, it narrows the spread a user experiences run to run.Figure 5Reliability: runs scoring at least 90 percentThe gap between arms is far wider here than in mean accuracy: the index converts a good average into a dependable result.Figure 6Effect size against bare Claude CodeBy convention d of 0.2 is small, 0.5 medium and 0.8 large; Corograph is the only arm above the large threshold.Figure 7Outcome composition per runThe dark bands are the honesty layer: the failures are on the page, not in a footnote.Figure 8Context tokens per correct answerThe ordering of Figure 1 survives the normalisation: the token saving is a saving per correct answer, not a saving bought with wrong answers.Figure 9Tool calls per runFewer calls with higher accuracy is the signature of a good index: the model asks its question once instead of searching for the answer many times.Figure 10Wall time per runTiming is environment-coupled and reported as indicative magnitude only; see the disclosures below before comparing across the host boundary.Figure 11Accuracy by modelThe comparison to make is diagonal: Corograph at a fast or everyday model against a bare harness or an indexing tool at a frontier model.Figure 12Pass rate by question category and difficultyThe categories where an index should matter most, structural questions such as callers, impact and cross-stack tracing, are where the separation is widest.Figure 13Cost per correct answerThe context saving of Figure 1 becomes a price saving at every model, and the ordering survives the normalisation by correct answers.Figure 14Cost per runA tight, low cloud is an arm whose price is predictable; a tall cloud is one whose price depends on the run.Figure 15Tool result tokens per runThis is the mechanism behind Figure 1 seen from the tool side: the index returns a small, precise answer where a search returns pages to read.Figure 16Organic index adoption by modelRead it as a description of behaviour rather than a ranking: it says how much of each arm's result was produced with its index in the loop, and how that changed as the model got stronger.

How to read this site

An arm is one configuration of tooling; a model is the LLM behind it; a run is one repetition of one arm × model × exam protocol × battery. Runs are independent sessions. Means are taken over valid, scored runs only; a run that did not finish is published unscored and never enters a mean.

Every figure page has three parts: the figure with its controls, the formal definition of what was computed, and the data table with a link per row into the repository. The Method page is the repository's README rendered; the Limitations page collects every disclosure in one place.