Benchmarks

What we measured — including where we lose

Every row names the committed file that produced it, so you can re-run it. The eval suites use synthetic configs; the label gold set is 50 rows we wrote ourselves.

measured, reproducible where we lose or have no result not measured

01 Detection — 40 seeded configs

Ground truth known by construction: planted routes, laundering-only routes, clean configs, and tie cases. Each case is ingested into its own graph and scanned with the committed query, live against FalkorDB 6.0.2 on 2026-10-17.

case kindcasesfoundfailedsource
planted route16160detection.json
laundering-only route880detection.json
clean — expected silent880detection.json
tie — both cuts close it880detection.json
total40400detection.json

What this does not show. A single-server trifecta check is solved work, and we publish no head-to-head table: we did not run Snyk Agent Scan, triflow or mcptrustchecker ourselves, so any comparison row would be invented. The qualitative comparison, with citations, is in docs/RELATED-WORK.md. What this table claims is narrower and reproducible: the committed query found every planted route, and all eight clean configs produced zero rows.

02 The fix — the part nobody else ships

One CALL algo.maxFlow, two canonical cuts, the cheaper one chosen by grant count, then task cost, then the write side. Each case proves itself by re-running the leak query with the revoked grants excluded.

what the pass condition checkscasespassedfailedsource
cut matches the constructed answer40400fix-quality.json
route closes — VERIFY_FIX returns zero40400fix-quality.json
expected tasks keep working40400fix-quality.json
clean config → empty cut, nothing revoked880fix-quality.json

The first three rows are the three parts of the same pass condition over the same 40 cases, eight clean ones among them; the file's failures array is empty. On the default fs-git fixture the same code path printed one route, max-flow 2, revoke git.push, 0 of 1 routes remain, 3 of 4 tasks keep working — verbatim in docs/live-evidence/.

03 Label quality — the honest accounting

Labels come from a 35-row rules table aided by tool-set expansions for 11 known servers. The gold set is 50 hand-written rows, run in the package's own test.

artefactcountresultsource
labelling rules35—packages/labeler/src/rules.ts
server tool-set expansions11—packages/labeler/src/rules.ts
gold-set rows5050/50 matchpnpm --filter @lethalpath/labeler test

We publish no precision/recall table. The gold set is self-authored alongside the rules: it proves internal consistency and catches regressions, not external ground truth. The rules we trust least are named in the labeler README — the sink end (tools named generically, like execute or process) is the weak one, which makes a scan quieter, never noisier.

04 Runtime guard — demo session, simulated tool results

callresultsource
github.read(issue#42)allowed · taint: untrustedlive-evidence
filesystem.read(~/work/stats)allowed · taint: untrusted + privatelive-evidence
git.push(main)blocked · route named in the errorlive-evidence
git.push(main), fresh sessionallowed · taint: nonelive-evidence

This is a demo. The guard speaks real MCP over stdio, but allowed calls return simulated results; no real GitHub, filesystem or git server sits behind it yet. The same push is blocked in a tainted session and allowed in a clean one. The guard's own suite runs eight tests, including the stdio end-to-end. Added latency: not measured, so no milliseconds appear here. The audit log stores tool, decision, reason and a short argument hash — never argument contents.

05 What we did not measure

These have no number yet, so none is printed.

  • Head-to-head against other scanners — not run. Positioning is qualitative and cited in docs/RELATED-WORK.md, including where MSCC and CaMeL are ahead of us.
  • Retrieval accuracy on tool-selection datasets — not attempted. Retrieval is not the product; the path and the cut are.
  • External label ground truth — none exists yet. The 50-row gold set is self-authored, and says so.
  • Guard latency overhead — not measured. We print no milliseconds we did not time.

Reproduce any of it

# both eval suites, live — rewrites the JSON these tables read
pnpm --filter @lethalpath/eval build && pnpm --filter @lethalpath/eval evaluate

# the labeler gold run (section 03)
pnpm --filter @lethalpath/labeler test

# the guard transcript (section 04)
pnpm --filter @lethalpath/guard demo

# the committed files themselves
cat packages/eval/results/detection.json packages/eval/results/fix-quality.json