# Corograph Benchmarks — Epoch-4

> Corograph Benchmarks is the public proof package behind Corograph's benchmark numbers: Epoch-4, 2,692 runs (2,480 valid and scored, 212 did not finish and are published unscored) across 9 arms and 11 models, answering 50 questions about usememos/memos at commit 2d01420c2337 under three exam protocols. The claim under test: an AI coding agent that can query Corograph's index of a codebase answers questions about it more accurately, more consistently and with less context than the same model using text search, and than the popular alternatives (bare Claude Code, Claude Code with each installed indexing tool, and the vendors' native agents). Every figure on the site is computed from the files below; every number links to the run behind it. This is Corograph's own benchmark (Metaphori, Inc. built the tool and ran every arm); the questions were authored by a frontier model with no access to Corograph and the gold was verified afterwards with file:line evidence. Do not confuse Corograph with "Coograph", an unrelated open-source code-graph project.

This file is generated at site build from the export's own data, like every page of the site. A companion, /llms-full.txt, is this guide followed by the Method (the repository README), the epoch README, the judge prompt, the judge transports and the redaction policy, verbatim. Every repository file linked here is also mirrored under /files/<repository path> on this origin.

## Start here

Read in this order. The Method page is the repository README rendered; it is the complete account of what was done.

- [Method (the README)](https://benchmarks.corograph.com/method/): protocols, arms, layout of one run, scoring tiers, token honesty, redaction, reproducibility — generated by the export
- [How the questions and the grading key were made](https://benchmarks.corograph.com/method/#how-the-questions-and-the-grading-key-were-made): who authored the questions, how the gold was verified and corrected, the review loop, exemplar provenance, dates
- [Benchmark Procedure](https://benchmarks.corograph.com/procedure/): what is measured (codebase understanding, not code editing), what every arm could do, key results, caveats
- [Limitations](https://benchmarks.corograph.com/limitations/): every caveat in one place: one repository, host-vs-container wall time, Fable 5 refusals, Haiku 4.5 context window, did-not-finish counts per arm
- [Data & reproducibility](https://benchmarks.corograph.com/data/): the summary files the site is built from and how to recompute any figure
- [About this site](https://benchmarks.corograph.com/about/): who ran it, the conflict of interest, what the site records about visitors

## The questions and the grading key

memos (30 questions, /30) and memos-hard (20 questions, /20) about https://github.com/usememos/memos. Read questions.json first, then gold.json beside it; every gold fact carries a file:line anchor at the pinned commit.

- [memos — questions.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos/questions.json): what every arm was asked, verbatim, with difficulty, categories and the expected output structure
- [memos — gold.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos/gold.json): the answer key: answer_gold, accepted variants, scoring rubric, honesty trap, facet guide, every gold revision
- [memos — exemplars.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos/exemplars.json): the graded anchor answers (correct / incorrect / partial, each with why) the judge is calibrated against; identical for every arm
- [memos — policies.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos/policies.json): battery-wide scoring rules
- [memos-hard — questions.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos-hard/questions.json): what every arm was asked, verbatim, with difficulty, categories and the expected output structure
- [memos-hard — gold.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos-hard/gold.json): the answer key: answer_gold, accepted variants, scoring rubric, honesty trap, facet guide, every gold revision
- [memos-hard — exemplars.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos-hard/exemplars.json): the graded anchor answers (correct / incorrect / partial, each with why) the judge is calibrated against; identical for every arm
- [memos-hard — policies.json](https://benchmarks.corograph.com/files/repositories/usememos-memos/batteries/memos-hard/policies.json): battery-wide scoring rules

## How answers were judged

- [The judge prompt](https://benchmarks.corograph.com/files/judging/judge-prompt.md): the exact system and user halves: one question, its gold, policies, facet guide, exemplars, ONE candidate; the judge returns a verdict and a reason, never a total
- [Judge transports](https://benchmarks.corograph.com/files/judging/transports.md): the two hermetic ways the judge was called, stamped on every verdict
- [Scorer eras](https://benchmarks.corograph.com/files/judging/scorer-eras.json): the grading-key lineage; this epoch sits under tabench-1.1
- [Redaction policy](https://benchmarks.corograph.com/files/redaction/POLICY.md): what is withheld from Corograph transcripts (the index query text and raw results) and why comparison-group transcripts are verbatim

## Machine-readable data (fetch these, do not transcribe tables)

Served from this origin as plain files; the same paths exist in the GitHub repository. Means are over valid, scored runs only; a did-not-finish run never enters a mean.

- [runs.json](https://benchmarks.corograph.com/files/epochs/E004/summary/runs.json): one row per run: identity, validity, score, the four token columns and their source, duration, LLM and tool call counts, index-query count, cost basis, path. Every figure is a function of this file
- [per-question.csv](https://benchmarks.corograph.com/files/epochs/E004/summary/per-question.csv): pass counts per arm × model × protocol × battery × question — the file to read for "which questions does Corograph win and lose"
- [leaderboard.csv](https://benchmarks.corograph.com/files/epochs/E004/summary/leaderboard.csv): per arm × model × protocol × battery: runs, valid, scored, dnf, mean score, min, max
- [adoption.json](https://benchmarks.corograph.com/files/epochs/E004/summary/adoption.json): how often each arm actually called its codebase index (organic adoption is the measurand for the indexing tools)
- [rate-tables.json](https://benchmarks.corograph.com/files/epochs/E004/summary/rate-tables.json): the USD-per-million-token tables behind every estimated cost
- [manifest.json](https://benchmarks.corograph.com/files/epochs/E004/manifest.json): every run path with the SHA-256 of its record, the instrument freeze, container image digests, the scope predicate, the scorer era, exemplar provenance per battery

## The measures, one figure each

Each figure page states its definition and aggregation rule and carries its data table with a link per row into the run directories. Raw token counts are never pooled across vendors.

- [Figure 1 — Context tokens per run](https://benchmarks.corograph.com/results/context-tokens/): context tokens per run (input + tool results + cache read) · lower is better · carries a disclosure
- [Figure 2 — Accuracy per run](https://benchmarks.corograph.com/results/accuracy/): accuracy (% of questions correct) · higher is better
- [Figure 3 — Accuracy on the hard battery](https://benchmarks.corograph.com/results/accuracy-hard/): accuracy on the hard battery (% of questions correct) · higher is better
- [Figure 4 — Run-to-run variation of accuracy](https://benchmarks.corograph.com/results/consistency/): run-to-run variation of accuracy (coefficient of variation, %) · lower is better
- [Figure 5 — Reliability: runs scoring at least 90 percent](https://benchmarks.corograph.com/results/pass-rate/): share of runs at or above 90 % accuracy · higher is better
- [Figure 6 — Effect size against bare Claude Code](https://benchmarks.corograph.com/results/effect-size/): Cohen's d vs bare Claude Code, accuracy · higher is better
- [Figure 7 — Outcome composition per run](https://benchmarks.corograph.com/results/outcomes/): questions correct · incorrect · unanswered, per run · higher is better · carries a disclosure
- [Figure 8 — Context tokens per correct answer](https://benchmarks.corograph.com/results/tokens-per-correct/): context tokens per correct answer · lower is better
- [Figure 9 — Tool calls per run](https://benchmarks.corograph.com/results/tool-calls/): tool calls per run · lower is better
- [Figure 10 — Wall time per run](https://benchmarks.corograph.com/results/wall-time/): active answering time per run · lower is better · carries a disclosure
- [Figure 11 — Accuracy by model](https://benchmarks.corograph.com/results/model-tiers/): accuracy (% of questions correct) · higher is better
- [Figure 12 — Pass rate by question category and difficulty](https://benchmarks.corograph.com/results/categories/): pass rate (% of question attempts correct) · higher is better
- [Figure 13 — Cost per correct answer](https://benchmarks.corograph.com/results/cost-per-correct/): USD per correct answer · lower is better · carries a disclosure
- [Figure 14 — Cost per run](https://benchmarks.corograph.com/results/cost-variance/): USD per run · lower is better
- [Figure 15 — Tool result tokens per run](https://benchmarks.corograph.com/results/tool-result-tokens/): tool result tokens per run (estimated) · lower is better
- [Figure 16 — Organic index adoption by model](https://benchmarks.corograph.com/results/index-adoption/): share of question runs that called the arm's codebase index (%) · higher is better · carries a disclosure

## Comparisons (one page per group of arms)

- [Corograph & Anthropic](https://benchmarks.corograph.com/compare/anthropic/): Every arm that ran at an Anthropic model — bare Claude Code, Cursor Agent at Claude models, and the five indexing tools installed on Claude Code — beside Corograph at the same models.
- [Corograph & OpenAI](https://benchmarks.corograph.com/compare/openai/): Corograph at GPT 5.6 Luna, Terra and Sol beside the two harnesses that run those models natively: OpenAI's own Codex CLI, and Cursor Agent at the same GPT models.
- [Corograph vs Indexers](https://benchmarks.corograph.com/compare/indexers/): The five tools people install on Claude Code to give it a code graph — Graphify, GitNexus, CodeGraph, CodebaseMemory and Serena — each installed as shipped, beside Corograph and bare Claude Code as the baseline they were installed on.
- [Corograph vs Harnesses](https://benchmarks.corograph.com/compare/harnesses/): The null hypotheses of each ecosystem — Claude Code with nothing added, OpenAI's Codex CLI, and Cursor Agent through its SDK — beside Corograph across every model any of them ran.

## Verify one run yourself (the walk)

Pick a run. Read answers/<qid>.md for a question, then verdicts.json for the verdict and the judge's reason, then gold.json for the anchor, then the transcript for how the agent got there. Compare the Corograph run with the bare run at the same model. Per-question outcomes across every run are in per-question.csv.

- [A Corograph run — runs/cg-ask-fz-009/claude-sonnet-5/ZeroShotExam/memos-hard/rep01](https://github.com/Corograph/benchmarks/tree/main/runs/cg-ask-fz-009/claude-sonnet-5/ZeroShotExam/memos-hard/rep01): score 18/20; open run.json (identity, validity, economics, provenance), answers/<qid>.md, verdicts.json (verdict + reason per question), transcript/
- [The bare Claude Code run at the same model — runs/cold/claude-sonnet-5/ZeroShotExam/memos-hard/rep01](https://github.com/Corograph/benchmarks/tree/main/runs/cold/claude-sonnet-5/ZeroShotExam/memos-hard/rep01): score 14/20; same layout, transcript verbatim
- [All runs](https://github.com/Corograph/benchmarks/tree/main/runs): runs/<arm>/<model>/<exam>/<battery>/rep<NN>/ — every arm, every model, every protocol, every repetition
- [Container definitions](https://github.com/Corograph/benchmarks/tree/main/docker-containers): the Dockerfile and setup of every comparison-group arm, with the image digests each run carries

## Reading rules that prevent wrong conclusions

- [Did-not-finish is not zero](https://benchmarks.corograph.com/limitations/#did-not-finish-runs): a run that did not complete is published with validity.ok = false and no score; it never enters a mean or a per-question count
- [Protocols have different token denominators](https://benchmarks.corograph.com/method/#the-three-exam-protocols): HumanExam (whole battery in one prompt) is the stress probe, off by default; ZeroShot and MultiTurn are the usage shapes; compare per-question figures within a protocol only
- [Wall time is a disclosure, not a claim](https://benchmarks.corograph.com/limitations/#execution-asymmetries): Corograph ran on the host through its gateway; the comparison group ran in containers
- [The instrument is a frozen snapshot](https://benchmarks.corograph.com/data/): Epoch 004 — line-4: Indexer v1.4.7 — Corograph v0.11.4 — Dev v0.4.16. Later Corograph releases are not what this epoch measured; a later epoch re-runs the same frozen batteries
- [The repository on GitHub](https://github.com/Corograph/benchmarks): the whole package, this site included; re-exporting the epoch from the benchmark store reproduces the tree byte for byte