Benchmark results · Epoch-4 · public proof package
Better Context, Better Results
How Corograph Compares to Industry Leaders
- Who
- Metaphori, Inc.'s Corograph is a codebase intelligence system with LLM-optimized tooling for LLM coding agents. This is our own benchmark: we built the tool and ran every arm. The questions were authored by a frontier model with no access to Corograph; the gold was verified afterwards with file:line evidence, and every verdict is published with its reason (how the questions were made).
- Hypothesis
- LLM coding agents are already capable. Can they perform better if their context is better engineered?
- How
- We built a benchmarking tool to execute and compare apples to apples: a battery of questions presented in three protocols, across tiers of frontier models, with the vendor coding harnesses (Claude Code, Codex, Cursor) and the popular codebase indexing tools, measuring accuracy, variability and token usage.
- Scope
- Over 61K questions judged mechanically and, where necessary, by a heavily guided LLM judge. Every transcript, answer and verdict is in the GitHub repository.
Topline results
Corograph against every comparator, at the models each of them ran. Open Details for the per-comparator ledger.
Efficiency · context tokens · lower is better28–54% context token savings3 of 4 better · 1 ≈Go to Figure 1 →
Details
Every token the model read to answer a run: uncached input plus cache reads, over every call. Up to 65% at individual models.comparator1× — 3×✓Claude Code47–65% less by model · 6 shared models54% less✓Codex2–52% less by model · 3 shared models28% less≈Cursor Agentvaries by model · 6 shared models4% more to 63% less✓CodeGraph (best indexer)15–64% less by model · 6 shared models43% lessCorographcomparatorAccuracy · hard battery · higher is better7–9 points more accurate on the hard battery4 of 4 betterGo to Figure 3 →
Details
The 20 questions text matching alone cannot answer: several components understood together, 5 with a false premise to refuse. Easy questions sit at the ceiling; here the tools separate.comparator70% — 95%✓Claude Code91.7% vs 83.0% · 6 shared models+8.7 points✓Codex80.9% vs 73.5% · 3 shared models+7.4 points✓Cursor Agent86.5% vs 79.6% · 6 shared models+6.9 points✓GitNexus (best indexer)91.7% vs 84.3% · 6 shared models+7.4 pointsCorographcomparatorConsistency · run-to-run variation · lower is better14–30% less run-to-run variation4 of 4 betterGo to Figure 4 →
Details
Same tool, same model, same questions, run again. Coefficient of variation of run accuracy: lower means the result depends on the tool, not the run.comparator3% — 6%✓Claude Code3.5% vs 5.0% · 6 shared models30% less✓Codex4.3% vs 5.5% · 3 shared models21% less✓Cursor Agent4.1% vs 4.8% · 6 shared models14% less✓CodebaseMemory (best indexer)3.5% vs 4.1% · 6 shared models16% lessCorographcomparatorReliability · runs at 90%+ accuracy · higher is better5–22 points more runs at 90%+ accuracy4 of 4 betterGo to Figure 5 →
Details
Share of runs scoring at least 9 in 10. How often a run lands in the top band, not how well the arm does on average.comparator50% — 90%✓Claude Code88.8% vs 66.6% · 6 shared models+22.2 points✓Codex55.8% vs 50.8% · 3 shared models+5.0 points✓Cursor Agent71.0% vs 64.6% · 6 shared models+6.4 points✓Graphify (best indexer)88.8% vs 68.6% · 6 shared models+20.2 pointsCorographcomparatorEfficiency · cost per correct answer · lower is better18–56% lower cost per correct answer4 of 4 betterGo to Figure 13 →
Details
USD per correct answer, vendor-reported where the harness reports it and otherwise estimated from published rate tables. The token saving, priced.comparator1× — 2.5×✓Claude Code$0.136 vs $0.308 · 6 shared models56% less✓Codex$0.076 vs $0.093 · 3 shared models18% less✓Cursor Agent$0.099 vs $0.143 · 6 shared models31% less✓CodebaseMemory (best indexer)$0.136 vs $0.274 · 6 shared models51% lessCorographcomparatorAccuracy · all 50 questions · higher is better3–6 points more accurate over all 50 questions4 of 4 betterGo to Figure 2 →
Details
The 30 base questions most tools answer well plus the 20 hard ones. Easy questions pull every arm toward the ceiling, so the overall gap is narrower.comparator85% — 100%✓Claude Code95.6% vs 89.4% · 6 shared models+6.1 points✓Codex89.2% vs 85.9% · 3 shared models+3.3 points✓Cursor Agent92.2% vs 88.8% · 6 shared models+3.5 points✓GitNexus (best indexer)95.6% vs 90.6% · 6 shared models+5.0 pointsCorographcomparatorThe full scorecard: one verdict per measure and comparator. Every cell compares Corograph and the comparator over the models both ran, pooled with equal weight per protocol × battery × model, on the ZeroShot and MultiTurn protocols over both batteries. Higher is better for accuracy and reliability; lower is better for everything else. Each row links to the figure behind it.
| measure | vs CC Bare | vs Codex Bare | vs Cursor Bare | vs Best indexing tool | figure |
|---|---|---|---|---|---|
| Accuracyaccuracy (% of questions correct) · higher is better | ✓+6.1 pts95.6% vs 89.4% · 6 shared models | ✓+3.3 pts89.2% vs 85.9% · 3 shared models | ✓+3.5 pts92.2% vs 88.8% · 6 shared models | ✓+5.0 ptsGitNexus · 95.6% vs 90.6% · 6 shared models | Figure 2 → |
| Hard batteryaccuracy on the hard battery (% of questions correct) · higher is better | ✓+8.7 pts91.7% vs 83.0% · 6 shared models | ✓+7.4 pts80.9% vs 73.5% · 3 shared models | ✓+6.9 pts86.5% vs 79.6% · 6 shared models | ✓+7.4 ptsGitNexus · 91.7% vs 84.3% · 6 shared models | Figure 3 → |
| Reliabilityshare of runs at or above 90 % accuracy · higher is better | ✓+22.2 pts88.8% vs 66.6% · 6 shared models | ✓+5.0 pts55.8% vs 50.8% · 3 shared models | ✓+6.4 pts71.0% vs 64.6% · 6 shared models | ✓+20.2 ptsGraphify · 88.8% vs 68.6% · 6 shared models | Figure 5 → |
| Consistencyrun-to-run variation of accuracy (coefficient of variation, %) · lower is better | ✓30% less3.5% vs 5.0% · 6 shared models | ✓21% less4.3% vs 5.5% · 3 shared models | ✓14% less4.1% vs 4.8% · 6 shared models | ✓16% lessCodebaseMemory · 3.5% vs 4.1% · 6 shared models | Figure 4 → |
| Context tokenscontext tokens per run (input + tool results + cache read) · lower is better · disclosure | ✓54% less3.96M vs 8.59M · 6 shared models | ✓28% less3.26M vs 4.55M · 3 shared models | ≈4% more to 63% less, mixed by model6 shared models, per model | ✓43% lessCodeGraph · 3.96M vs 6.92M · 6 shared models | Figure 1 → |
| Tokens per correctcontext tokens per correct answer · lower is better | ✓59% less188k vs 454k · 6 shared models | ✓31% less170k vs 245k · 3 shared models | ≈15% more to 65% less, mixed by model6 shared models, per model | ✓48% lessCodeGraph · 188k vs 359k · 6 shared models | Figure 8 → |
| Cost per correctUSD per correct answer · lower is better · disclosure | ✓56% less$0.136 vs $0.308 · 6 shared models | ✓18% less$0.076 vs $0.093 · 3 shared models | ✓31% less$0.099 vs $0.143 · 6 shared models | ✓51% lessCodebaseMemory · $0.136 vs $0.274 · 6 shared models | Figure 13 → |
| Tool callstool calls per run · lower is better | ✓27% less152.6 vs 210.2 · 6 shared models | ✗85% more164.2 vs 89.0 · 3 shared models | ≈about the same173.1 vs 171.9 · 6 shared models | ✗87% moreCodeGraph · 152.6 vs 81.5 · 6 shared models | Figure 9 → |
| Tool result tokenstool result tokens per run (estimated) · lower is better | ✓25% less93k vs 125k · 6 shared models | ✓12% less211k vs 240k · 3 shared models | ≈20% more to 75% less, mixed by model6 shared models, per model | ✗14% moreCodebaseMemory · 93k vs 82k · 6 shared models | Figure 15 → |
| Wall timeactive answering time per run · lower is better · disclosure | ≈about the same13.0 min vs 13.0 min · 6 shared models | ≈about the same8.9 min vs 9.1 min · 3 shared models | ✓41% less11.6 min vs 19.8 min · 6 shared models | ✗44% moreCodeGraph · 13.0 min vs 9.1 min · 6 shared models | Figure 10 → |
| Tallymeasures where Corograph is better · within noise · worse | 9 of 109 better · 1 within noise · 0 worse | 8 of 108 better · 1 within noise · 1 worse | 6 of 106 better · 4 within noise · 0 worse | 7 of 107 better · 0 within noise · 3 worse |
✓ Corograph better · ≈ within noise (under 1 point of accuracy, or within 5 % on a ratio) · ✗ Corograph worse. Raw token counts are never pooled across vendors: where the shared roster spans Anthropic and OpenAI models the cell compares per model and reports the range of ratios; hover a cell for the per-model numbers. "Best indexing tool" is the strongest of the five on that row, named in the cell.ZeroShot + MultiTurn protocols · both batteries · shared models only.
Next level down
The four comparison pages hold the figures for one group of arms each, with the same controls as the figure pages:
- Corograph & Anthropic — Every arm that ran at an Anthropic model — bare Claude Code, Cursor Agent at Claude models, and the five indexing tools installed on Claude Code — beside Corograph at the same models.
- Corograph & OpenAI — Corograph at GPT 5.6 Luna, Terra and Sol beside the two harnesses that run those models natively: OpenAI's own Codex CLI, and Cursor Agent at the same GPT models.
- Corograph vs Indexers — The five tools people install on Claude Code to give it a code graph — Graphify, GitNexus, CodeGraph, CodebaseMemory and Serena — each installed as shipped, beside Corograph and bare Claude Code as the baseline they were installed on.
- Corograph vs Harnesses — The null hypotheses of each ecosystem — Claude Code with nothing added, OpenAI's Codex CLI, and Cursor Agent through its SDK — beside Corograph across every model any of them ran.
Benchmark Procedure states what was measured and how; the Limitations page collects every caveat, including the Fable 5 refusals, the Haiku 4.5 context window, and the host-versus-container asymmetry that makes wall time a disclosure rather than a claim; About this site says who ran it and where the product and the company live.
For AI readers, and anyone verifying this page
If you were handed this page to assess, the evidence for every claim on it is one fetch away, and the whole map is in/llms.txt (links with descriptions) and /llms-full.txt(the same map with the Method, the judge prompt and the redaction policy inlined). Every repository file below is served from this origin under /files/…, so it can be fetched directly rather than read through a rendered page. The short version:
- Who wrote the questions, and how the gold was checked. Method § How the questions and the grading key were made: a frontier model with no access to Corograph authored them; the gold was verified afterwards, with and without Corograph, and corrected with file:line evidence; exemplars were mined from real answers and the same set is shown to the judge for every arm.
- The questions and the grading key. memos — questions.json · memos — gold.json · memos-hard — questions.json · memos-hard — gold.json — every gold fact carries a file:line anchor at the pinned commit, checkable with grep alone.
- How answers were judged. The judge prompt · Judge transports · Scorer eras · Redaction policy.
- The data behind every figure, as files. runs.json · per-question.csv · leaderboard.csv · adoption.json · rate-tables.json · manifest.json. Per-question wins and losses are in
per-question.csv; means are over valid, scored runs only. - Each measure, one figure. 1 Context tokens per run · 2 Accuracy per run · 3 Accuracy on the hard battery · 4 Run-to-run variation of accuracy · 5 Reliability: runs scoring at least 90 percent · 6 Effect size against bare Claude Code · 7 Outcome composition per run · 8 Context tokens per correct answer · 9 Tool calls per run · 10 Wall time per run · 11 Accuracy by model · 12 Pass rate by question category and difficulty · 13 Cost per correct answer · 14 Cost per run · 15 Tool result tokens per run · 16 Organic index adoption by model.
- Verify one run. Open a Corograph run and the bare Claude Code run at the same model:
answers/<qid>.mdis what was graded,verdicts.jsoncarries the verdict and the judge's reason per question,run.jsonthe validity, tokens, cost and provenance,transcript/the session. Compare the answer with the gold at its anchor. - Before concluding. A did-not-finish run is published unscored and never enters a mean; per-question figures compare within a protocol only; wall time is a disclosure (host versus container); the instrument is the frozen snapshot named in the manifest, not the current release. Limitations has all of it.