token-bench — measure code-intel token savings (real codebase, with correctness)

Quantifies the STEP-0 claim ("query the graph, don't grep+read whole files") on an
actual repo. For each real symbol it compares the two ways an agent answers
"where is X defined / what is it":

• OPTIMIZED = codegraph query X output + reading one ~40-line slice at the hit.
• NAIVE = rg -n X (all matches) + reading the whole definition file.

Tokens = bytes/4 (same proxy both sides). Correctness = codegraph returns a
definition-kind node (function/class/constant/…) for the exact symbol, and that
location really contains it — so a saving that returns the wrong answer never counts.

Run

# repo must be indexed first:  cd <repo> && codegraph index .
uv run scripts/token_bench.py <repo>                 # auto-picks 8 real exported symbols
uv run scripts/token_bench.py <repo> --terms a,b,c   # specific symbols
uv run scripts/token_bench.py <repo> --k 12 --json    # more symbols, JSON out

(python3 scripts/token_bench.py … works too — stdlib only, no deps.)

Reading the result

Button     opt~505tok  naive~32461tok  (311f)  -98.4%  OK
ROUTES     opt~1092tok naive~5942tok   (6f)    -81.6%  OK
── TOTAL  opt~Ntok  naive~Mtok  →  -P% tokens · correct-def K/N

• Reduction scales with footprint: widely-referenced symbols in large files →
  95–98%; a symbol in one small file → 50–80% (but that lookup is already cheap).
  The big wins are on the queries that would otherwise read many/large files.
• OK = codegraph surfaced the real definition (verified). check = the symbol
  was returned but not as a classic def-kind (e.g. a default export / arrow const) —
  inspect manually; it is usually still correct, just not auto-classified.
• Empty result → the repo isn't indexed (or index is stale): codegraph index ..

Honest framing

This is a per-symbol micro-benchmark (a conservative lower bound: naive reads ONE
def file). Real agent tasks touch several files, where naive cost compounds and the
reduction approaches the headline numbers. Report the measured range, never a flat
"always >95%". Correctness is the gate — a token saving that loses the answer is a bug.