Controlled benchmark comparing naive (full history) vs ToolRecall (context dropping) on deepseek/deepseek-v4-flash via OpenRouter at the model's true 1,048,576 context limit. Three scripted workloads across a codebase of ~17 source files: review (read-only), bugfix (read-write), analysis (heavy reads). Separate OpenRouter API keys per arm — billed amounts are from the provider dashboard, not modeled.
N=1 seed per workload. These are not multi-seed averages. Every workload ran once per arm. Variance is not yet measured. The aggregate includes two workloads naive failed to complete — the cleanest head-to-head is bugfix (−31%, both arms completed 450 turns).
ToolRecall was 31% cheaper across all three workloads ($7.87 vs $11.36) and completed work naive could not finish. The cleanest matched comparison is bugfix (both arms completed 450 turns): ToolRecall cost 31% less. This benchmark measures token cost and session endurance, not task quality. See What NOT to Claim.
Measures: prompt tokens consumed, completion tokens produced, context window usage, session survival (did the workload finish before hitting the 1.05M cap?), and total billed cost from OpenRouter per arm.
Does NOT measure: answer quality, code correctness, task success rate, or whether ToolRecall's outputs are as good as naive's. Both arms follow the same scripted instructions and produce LLM responses every turn, but no evaluation framework validates quality. "Completed" means the harness finished without hitting the context cap — not that the task was solved correctly.
Probes (random token recall checks) are planted every 25 turns but have not been analyzed. See What's Missing.
The bugfix workload rotates through source files, asking the LLM to inspect and fix issues. 50 read-only turns, then 200 turns of alternating reads and writes, then 100 turns of error-handling additions, then 100 turns of cross-file tracing. Both arms completed all 450 turns — this is the cleanest matched comparison.
| Metric | Naive | ToolRecall | Δ |
|---|---|---|---|
| Total spend (billed) | $6.30 | $4.36 | −31% |
| Prompt tokens | 159,448,124 | 86,654,813 | −46% |
| Completion tokens | 455,406 | 521,302 | +14% |
| DeepSeek prefix cache hit | 87.9% | 79.8% | −8.1pp |
| Context at end | 1,006,089 | 682,688 | −32% |
| Status | alive (42K from cap) | alive (35% headroom) | — |
| Wall time | 231.8 min (3.9h) | 230.4 min (3.8h) | −1% |
| Avg latency per turn | 31.0s | 30.8s | −1% |
Why this is the headline number: Both arms completed the same 450-turn workload. ToolRecall sent 46% fewer prompt tokens for the same task and cost 31% less on this workload alone (derived from cumulative billing: $8.83 minus $2.53 for review). Naive's 87.9% prefix cache hit vs TR's 79.8%: TR drops old context, so DeepSeek's automatic prefix caching has less repeated prefix to match. Volume wins: 159M vs 87M prompt tokens.
Read-only workload: same 4 core files every turn (cache.py, daemon.py, config.py, client.py). ToolRecall's best case — all files are always clean, so file content is dropped every turn.
| Metric | Naive | ToolRecall | Δ |
|---|---|---|---|
| Total spend (billed) | $2.53 | $1.26 | −50% |
| Prompt tokens | 50,948,649 | 11,351,955 | −78% |
| Completion tokens | 79,642 | 110,226 | +38% |
| DeepSeek prefix cache hit | 82.2% | 66.0% | −16pp |
| Turns completed | 112 / 200 | 200 / 200 | ✅ |
| Context at end | 1,054,125 (capped) | 141,509 | −86% |
| Wall time | unavailable | 45.5 min | — |
| Avg latency per turn | unavailable | 13.7s | — |
Naive died at turn 112 (hit the 1.05M cap). ToolRecall completed all 200 turns at 142K — 86% smaller context. The 50% spend reduction understates the gap: running naive to completion would require a second run. Completion tokens (+38%) likely a workload artifact (same files every turn → shorter context → more verbose responses), not a quality signal.
Heavy-read workload: summarize files, compare modules, audit security. 3 files read per turn, rare writes. Naive's context grew fastest here — died by turn 145.
| Metric | Naive | ToolRecall | Δ |
|---|---|---|---|
| Total spend | $2.53 | $2.25 | −11% |
| Prompt tokens | 60,550,076 | 46,027,861 | −24% |
| Completion tokens | 201,192 | 503,336 | +150% |
| DeepSeek prefix cache hit | 81.9% | 82.8% | +0.9pp |
| Turns completed | 145 / 400 | 400 / 400 | ✅ |
| Context at end | 1,052,134 (capped) | 779,004 | −26% |
| Wall time | 115.3 min | 164.4 min | +43% |
| Avg latency per turn | 48.4s | 24.7s | −49% |
Per-workload cost derived from cumulative billing: $11.36 total minus $8.83 after review+bugfix. TR analysis cost more than review ($2.25 vs $1.26) despite fewer tokens — likely because TR completed 400 turns vs review's 200, and the longer context lowered the cache hit rate. The +150% completion spike is unexplained and warrants investigation. If short context causes the model to re-state information in completions (and completions are the expensive token at $0.55/M), this could offset some of TR's prompt-token savings.
Billed amounts per workload, derived from cumulative OpenRouter dashboard data. Each arm used one API key — the per-workload costs are obtained by subtracting the cumulative total at each checkpoint. The raw checkpoints are shown below so every derived number is auditable.
| Checkpoint | Naive key (cumulative) | TR key (cumulative) |
|---|---|---|
| After review | $2.53 · 53.1M tokens | $1.26 · 22.0M tokens |
| After review + bugfix | $8.83 · 228.0M tokens | $5.62 · 117.0M tokens |
| After all 3 workloads | $11.36 · 290.0M tokens | $7.87 · 168.0M tokens |
Token volumes are total billed (prompt + completion) from the OpenRouter dashboard. The 228.0M for naive at checkpoint 2 counts review's 53.1M + bugfix. Same for TR's 117.0M.
| Workload | Naive | ToolRecall | Δ | Naive billed tokens | TR billed tokens |
|---|---|---|---|---|---|
| Review (200t) | $2.53 | $1.26 | −50% | 53.1M | 22.0M |
| Bugfix (450t) | $6.30 | $4.36 | −31% | 174.9M | 95.0M |
| Analysis (400t) | $2.53 | $2.25 | −11% | 62.0M | 51.0M |
| TOTAL | $11.36 | $7.87 | −31% | 290.0M | 168.0M |
How per-workload is derived:
Naive bugfix = $8.83 − $2.53 = $6.30, 228.0M − 53.1M = 174.9M tokens.
Naive analysis = $11.36 − $8.83 = $2.53, 290.0M − 228.0M = 62.0M tokens.
Same arithmetic for TR: bugfix = $5.62 − $1.26 = $4.36, analysis = $7.87 − $5.62 = $2.25.
Analysis = review = $2.53 on the naive key is a coincidence of rounding, not a placeholder.
The raw checkpoints ($2.53, $8.83, $11.36) show analysis really added exactly $2.53.
The per-workload prompt-token-only tables below (50.9M, 159.4M, 60.6M) use different
counters (tiktoken on the request payload) — they will not match the billed token volumes
exactly, because OpenRouter counts differently (includes completion tokens, different
tokenizer, provider markup).
| Metric | Naive | ToolRecall | Δ |
|---|---|---|---|
| Total spend | $11.36 | $7.87 | −31% |
| Total prompt tokens | 270,946,849 | 144,034,629 | −47% |
| DeepSeek prefix cache hit | 85.6% | 78.9% | −6.7pp |
| Workloads completed | 1 / 3 | 3 / 3 | — |
| Total wall time | ~5.8h | ~7.4h | +28% |
Aggregate prompt tokens: review (50.9M + 11.4M) + bugfix (159.4M + 86.7M) + analysis (60.6M + 46.0M). Aggregate cache hit rate: weighted by token volume. Spend is the total per-key billed amount from OpenRouter covering all runs on that key. The aggregate includes two workloads naive failed to complete — the cleanest head-to-head is bugfix (−31%).
Using DeepSeek's list prices ($0.14/1M uncached prompt, ~$0.01/1M cached prompt, $0.55/1M completion) on the bugfix token counts gives ~$4.62 naive / ~$3.72 TR — roughly half the actual billed amounts ($6.30 / $4.36 for bugfix alone). The gap likely comes from OpenRouter's routing markup, provider selection (DeepInfra frequently routes DeepSeek traffic), and the complexity of how prefix cache hits are applied across providers. The comparison between arms is valid (same routing, same model). The absolute dollar amounts should not be used to estimate costs on other providers or routing configurations.
Average prompt tokens per turn by decile. ToolRecall's advantage compounds as clean file content is dropped turn after turn.
| Turn range | Naive avg/turn | ToolRecall avg/turn | TR vs naive |
|---|---|---|---|
| 001–050 | 8,359 | 6,320 | −24% |
| 051–100 | 20,756 | 12,810 | −38% |
| 101–150 | 33,449 | 18,262 | −45% |
| 151–200 | 46,435 | 22,833 | −51% |
| 201–250 | 56,339 | 27,679 | −51% |
| 251–300 | 66,116 | 31,933 | −52% |
| 301–350 | 76,439 | 36,265 | −53% |
| 351–400 | 87,311 | 41,087 | −53% |
| 401–450 | 99,246 | 46,186 | −53% |
ToolRecall's advantage grows from −24% in the first decile to −53% by the last. This is the compounding effect of dropping clean content: each turn's savings carry forward. On the review workload (read-only, max drop rate), the end gap was −86%.
✗ "ToolRecall saves 31% on any task."
These are scripted benchmarks on one model (DeepSeek V4 Flash). Real LLM-agent tasks involve different files, write patterns, and decision-making that may change the savings.
✗ "Lower completion tokens = better quality."
Completion counts varied inconsistently across workloads (−4% to +150%). This benchmark does not measure answer quality, task success rate, or code correctness.
✗ "Bounded context growth."
Context still grows linearly — just slower. TR grows at ~100–120 tokens/turn on steady-state vs naive's ~200–300 tokens/turn. It is not bounded.
✗ "ToolRecall is cheaper than provider prefix caching."
Both arms receive DeepSeek's automatic prefix caching. TR's lower cache hit rate (77% vs 87%) partially offsets its volume savings. The advantage comes from sending fewer total tokens, not from more efficient caching.
✗ "TR completed the work, naive didn't."
Both arms follow the same scripted instructions, but no task-evaluation framework validated code quality. "Completed" means the harness didn't hit the context cap — not that the task was solved correctly.
✗ "Benchmark predicts production cost savings."
Scripted workloads rotate through a fixed file pool. Real-world workloads have longer file lists, larger files, and inconsistent reuse patterns. Results may not generalize.
✗ "Completion token differences are meaningful."
Variance across workloads (−4% to +150%) suggests workload-specific artifacts, not a consistent property of either arm.
This benchmark measures token counts and context survival — not whether TR's outputs are as good as naive's. A SWE-bench-style evaluation or even a manual diff of outputs on matched turns would close the biggest gap. Until then, "cheaper but possibly worse" is the honest framing. Probes (random token recall checks every 25 turns) are planted in every run but have not been analyzed.
There is no third arm that just drops tool outputs older than N turns (no daemon, no dirty tracking, no file content awareness). That would be ~20 lines of Python. Until it loses, the benchmark shows the idea works, not that this specific implementation is needed.
Each workload ran once per arm (N=1). Adding 3-5 seeds would show variance and whether results hold across random file rotations and probe token placements.
The +150% completion spike on analysis is unexplained. If short context causes the model to re-state information in its responses, and completions cost $0.55/M vs prompts at ~$0.01–0.14/M, this could significantly offset savings. Needs investigation.
Billed amounts are 1.5–1.9× higher than list-price estimates. The gap needs explanation before absolute cost numbers are used for budgeting.
toolrecall/bench/run_arm.py — scripted workloads with per-run SQLite turn logs.
Each arm runs make_agent_turn() injected with the arm-specific strategy:
naive keeps full history, toolrecall calls the daemon's context tracker and strips
clean file blocks after each turn. Both read files via toolrecall.client.cached_read().
Both use the same file cache (no pre-warming between arms).
The harness enforces CONTEXT_LIMIT = 1,048,576 tokens — DeepSeek V4 Flash's
advertised maximum. Before each turn, request_tokens() (tiktoken cl100k_base)
counts the full message payload. Runs exceeding the limit are terminated with
context_exhausted.
Each arm used a dedicated OpenRouter API key. The billed amounts are from OpenRouter's usage dashboard for each key, covering all requests. The billed figures are not modeled — they are what OpenRouter actually charged. The pricing section below documents the gap between billed and list-price estimates.
All workloads are scripted turns over the ToolRecall codebase (~17 source files, 10 documentation files):
| Workload | Turns | Pattern | Characteristics |
|---|---|---|---|
| review | 200 | Same 4 core files every turn | Read-only, max cache overlap |
| bugfix | 450 | Rotating files, read → fix → write | Reads + writes, dirty tracking |
| analysis | 400 | Summarize, compare, audit | Heavy reads, rare writes |
# Repro command
NAIVE_API_KEY="sk-or-..." \
python3 bench/run_arm.py naive bugfix --max-turns 450 \
--provider openrouter --model deepseek/deepseek-v4-flash
Five sanitized SQLite databases from the DeepSeek V4 Flash runs above.
Timestamps zeroed. Each contains the full per-turn turn_log table
and probe_result table.
The databases are shipped with the repo at bench/data/.
Clone and query them locally:
$ git clone https://github.com/whiskybeer/toolrecall.git && cd toolrecall/bench/data
$ sqlite3 deepseek-v4-flash_naive_bugfix_450t.db "SELECT arm, workload_id, SUM(prompt_tokens) FROM turn_log;"
Files: naive_bugfix_450t (108K), toolrecall_bugfix_450t (116K), naive_analysis_145t (48K), toolrecall_analysis_400t (104K), toolrecall_review_200t (68K). README with schema & query examples included in the repo.
Sanitization: ts (timestamp), ttft_s, and
api_latency_s are zeroed. All token and cost metrics are untouched.
No API keys or secrets are stored in these databases — error messages are
JSON-parsing failures only.
sqlite3 deepseek-v4-flash_naive_bugfix_450t.db "SELECT arm, workload_id, SUM(prompt_tokens) FROM turn_log;"
→ naive|bugfix|159448124 (matches the table above)
⚠ Superseded by the DeepSeek V4 Flash benchmarks above.
These gpt-4o-mini results used a 128K harness cap (not the model's true limit),
a smaller context window model, and different billing. The pattern (TR drops content,
naive accumulates) is consistent, but the absolute ratios and endurance numbers
(7.4×, "128K is the ceiling") should not be carried forward. The 3-seed confirmation
of the direction is still valid — the exact multipliers are not.
The following benchmarks were run with gpt-4o-mini at a 128K harness cap. (the model's true limit is 128K). These are preserved for reference. The same harness, methodology, and workload definitions were used. The key difference is the model (smaller context window) and the harness cap (128K vs 1.05M).
Same review workload: 4 core files every turn. gpt-4o-mini via OpenRouter. 128K harness cap. The pattern is the same — ToolRecall drops clean content, naive accumulates everything — but the absolute numbers differ because the model and context limit are smaller.
| Turn | Prefix prompt_tokens | ToolRecall prompt_tokens | Dropped (cumulative) | Ratio |
|---|---|---|---|---|
| 1 | 7,609 | 7,609 | 41,699 | 1.0× |
| 5 | 40,101 | 10,477 | 208,495 | 3.8× |
| 10 | 82,887 | 13,460 | 416,990 | 6.2× |
| 15 | 124,606 | 16,243 | 625,485 | 7.7× |
| 17 (prefix exhausted) | 134,570 | 17,356 | 708,883 | 7.8× |
| 50 | — | 36,304 | 2,084,950 | — |
| 100 | — | 92,238 | 4,324,345 | — |
| 140 (TR exhausted) | — | 128,006 | 4,375,315 | 7.4× longer |
Prefix: 134,570 prompt_tokens — full history, dead at 128K cap.
ToolRecall: 17,356 prompt_tokens — 87% less context at the same turn. Survives to turn 140 before hitting the 128K ceiling.
gpt-4o-mini via OpenRouter. Separate API keys. Billed amounts show actual provider cost.
| Arm | Turns | Avg $/turn (list) | Billed $ (actual) |
|---|---|---|---|
| Prefix caching | 17 | $0.0103 | $0.054 |
| ToolRecall | 140 | $0.0087 | $0.049 (29 turns)* |
* ToolRecall's 140-turn run billing data was collected for a 29-turn segment. Per-turn: 16% cheaper. Full 140-turn billing pending provider invoice.
Confirmed across 3 seeds (42, 43, 44). gpt-4o-mini (128K context). Naive exhausted at turn 28. ToolRecall completed all 200 turns.
| Turn | Naive req_tok (median, n=3) |
ToolRecall req_tok (median, n=3) |
Ratio |
|---|---|---|---|
| 10 | 37,170 | 4,898 | 7.6× |
| 20 | 72,976 | 10,311 | 7.1× |
| 28 (naive exhausted) | 106,111 | 14,668 | 7.2× |
| 30 | 113,258 | 15,748 | 7.2× |
| 50 | — | 26,559 | — |
| 100 | — | 55,623 | — |
| 200 | — | 110,318 | — |
Naive exhausted at turn 28 (128K cap). ToolRecall completed all 200 turns at only 110K request tokens — still 14% of context budget left.
Measured during the gpt-4o-mini benchmark runs. All file reads went through
toolrecall.client.cached_read().
| Arm | Hits | Misses | Hit Rate |
|---|---|---|---|
| naive | 124 | 0 | 100% |
| prefix | 140 | 0 | 100% |
| toolrecall | 717 | 5 | 99.3% |
| ALL ARMS | 981 | 5 | 99.5% |
ToolRecall's 5 misses happened during warm-up (first read of each file after daemon start).
Full benchmark harness, workloads, and per-run databases are below. Download the raw SQLite databases and query them yourself. Every claim above is traceable to a per-turn log.
View bench/ on GitHub →