Skip to content
cogniphicSuperintelligenceDownload Forge

ResearchBenchmarks and decisions

About half the cost per task,
and cheaper on every task.

Our first study asked one question: with the same model, what do a context engine and a prompt cache change? 312 live runs with Claude Opus 5.5, judged by hidden tests the agent never saw. Every number on this page comes straight from the run logs.

Explore every runHow we measured

Specimen 0120 tasks · 240 runs

Project · context engine

Claude Opus 5.5 · hidden testsBaseline$0.0405Mean cost per run

One blob per task

2026

  • 312 live runs
  • −49% cost per task
  • 34 decisions on record
−0%Cost per accepted taskA $0.0405 → C $0.0205 · main set, 240 runs
−0%The same, on the harder setA $0.0543 → C $0.0284 · 72 runs
−0%Turns with the context engine3.05 → 2.50 on average · B against A
−0%A warm, cached request$0.0248 cold → $0.0012 warm

Explore

312 runs.
Compare them any way you like.

Pick a task set, a measure and the arms to compare. Each row is one task; the large dot is its average over three repeats and the small dots are the single runs. Hover a row for its numbers.

Cost, task by task: main set

Measured cost of each run, from the provider’s own cost field. Every run was accepted, so this is also the cost per accepted task.

Computed here from the run log: 240 runs, 20 tasks, three repeats each. Large dots are each task’s mean; small dots are the single runs.

Data table
TaskSetAC
h9_py_long_repairheld-out$0.0641$0.0422
h11_py_roundingheld-out$0.0605$0.0344
h12_node_format_bytesheld-out$0.0524$0.0245
h3_node_empty_romanheld-out$0.0481$0.0415
h5_py_word_countheld-out$0.0474$0.0224
h8_node_missing_testheld-out$0.0470$0.0259
h6_py_public_typeheld-out$0.0469$0.0205
h2_py_large_repoheld-out$0.0424$0.0121
t7_node_featuretuning$0.0400$0.0226
h4_node_medianheld-out$0.0395$0.0163
t8_py_missing_testtuning$0.0389$0.0183
t4_py_kvtuning$0.0355$0.0149
t6_py_empty_fizzbuzztuning$0.0343$0.0241
h1_py_leapheld-out$0.0340$0.0092
t2_py_discounttuning$0.0324$0.0207
t1_node_slugtuning$0.0322$0.0164
h7_node_configheld-out$0.0302$0.0174
t3_py_configtuning$0.0297$0.0099
h10_py_concurrent_editheld-out$0.0282$0.0089
t5_py_addtuning$0.0267$0.0083

Cost

Half the cost,
on both sets.

Arm C, the context engine plus the prompt cache, costs about half of the baseline on both sets, with every run accepted.

Cost per accepted task, by arm

Total measured cost ÷ runs accepted by hidden tests. Same model, same machine, same tasks.

Main set: 240 runs, $8.37 total. Harder set: 72 runs, $3.22 total. Arm R was not run on the harder set.

Data table
ArmMain setHarder set
A · Baseline$0.0405$0.0543
B · + context engine$0.0376$0.0513
C · + context engine + cache$0.0205$0.0284
R · + research helpers$0.0409not run

Every task got cheaper, all 20 on the main set

Mean cost per run of each task (three repeats), baseline A against arm C. Sorted by baseline cost.

  • A · Baseline
  • C · context engine + cache
Data table
TaskSetA meanC meanChange
h9_py_long_repairheld-out$0.0641$0.0422-34%
h11_py_roundingheld-out$0.0605$0.0344-43%
h12_node_format_bytesheld-out$0.0524$0.0245-53%
h3_node_empty_romanheld-out$0.0481$0.0415-14%
h5_py_word_countheld-out$0.0474$0.0224-53%
h8_node_missing_testheld-out$0.0470$0.0259-45%
h6_py_public_typeheld-out$0.0469$0.0205-56%
h2_py_large_repoheld-out$0.0424$0.0121-71%
t7_node_featuretuning$0.0400$0.0226-44%
h4_node_medianheld-out$0.0395$0.0163-59%
t8_py_missing_testtuning$0.0389$0.0183-53%
t4_py_kvtuning$0.0355$0.0149-58%
t6_py_empty_fizzbuzztuning$0.0343$0.0241-30%
h1_py_leapheld-out$0.0340$0.0092-73%
t2_py_discounttuning$0.0324$0.0207-36%
t1_node_slugtuning$0.0322$0.0164-49%
h7_node_configheld-out$0.0302$0.0174-42%
t3_py_configtuning$0.0297$0.0099-67%
h10_py_concurrent_editheld-out$0.0282$0.0089-69%
t5_py_addtuning$0.0267$0.0083-69%

Why it’s cheaper

The same prompt size,
most of it read back.

The context engine keeps the prompt small; the cache means the stable part is paid for once and read back at a twentieth of the price.

Where each run's input tokens come from

Mean input tokens per run, split into fresh input, tokens written to the cache, and tokens read back from it.

  • Fresh input
  • Cache write (billed 1.25×)
  • Cache read (billed 0.05×)

Prompt tokens include cached reads and writes; they are split here, never added twice. Billing ratios from ADR 0010.

Data table
Set · armInputFreshCache writeCache read
Main · A6,5846,58400
Main · B6,3396,33900
Main · C6,34171,3784,955
Main · R6,8036,80300
Harder · A8,6488,64800
Harder · B8,3908,39000
Harder · C8,86681,6577,200

What one request costs, cache by cache

Six live requests to anthropic/claude-opus-5.5 via OpenRouter (2026-09-26), measured with the provider's own cost field.

A cache marker is required; there is no implicit cache. Writes bill at 1.25×, reads at 0.05×; the measured costs match those ratios.

Data table
ScenarioPrompt tokensCache readCache writeCost
No cache marker (same request twice)3,62500$0.0146
Marked, cold4,93104,927$0.0248
Marked, warm (same request)4,9314,9270$0.0012
Later content changed, prefix the same4,9311,7673,160$0.0163
Prefix changed (full miss)4,93804,934$0.0248
Breakpoint on a tool result, then continue6,0015,9820$0.0014

Quality and speed

Nothing got worse.
Fewer turns, the same time.

Every run was accepted by the hidden tests. The context engine needed fewer model turns, and wall-clock time stayed within a few seconds.

Accepted by hidden tests: 312 of 312 runs

Each square is one run (three repeats per task and arm). Filled = the hidden tests passed on the final tree.

  • Accepted

Every arm passed every task, so these sets cannot separate the arms on quality. We don't claim the arms are equal in general, only that none was worse than the baseline here.

Data table
GroupArmAccepted
Main set · tuning (8 tasks)A24/24
Main set · tuning (8 tasks)B24/24
Main set · tuning (8 tasks)C24/24
Main set · tuning (8 tasks)R24/24
Main set · held-out (12 tasks)A36/36
Main set · held-out (12 tasks)B36/36
Main set · held-out (12 tasks)C36/36
Main set · held-out (12 tasks)R36/36
Harder held-out set (8 tasks)A24/24
Harder held-out set (8 tasks)B24/24
Harder held-out set (8 tasks)C24/24

Turns per run: the context engine needs fewer

Share of the 60 main-set runs per arm that finished in 1, 2, 3 or 4 model turns. Mean on the right.

  • 1 turn
  • 2 turns
  • 3 turns
  • 4 turns

Published: average turns 3.05 → 2.50 with the context engine (B vs A, −18%). Means here are computed from the evidence log.

Data table
Arm1234Mean turns
A · Baseline064593.05
B · + context engine0361862.50
C · + context engine + cache3321872.48
R · + research helpers0104192.98

Time to finish, every run

Wall-clock seconds per run, one dot per run (jittered vertically). Ticks show the published median and p90.

  • One run (blue: arm C)
  • Median
  • p90

Speed was not the goal of the context engine or the cache; medians stay within a few seconds across arms.

Data table
Set · armRunsMedian sp90 sFastest sSlowest s
Main · A6017338.846.6
Main · B6016316.838.0
Main · C6017306.937.5
Main · R6017347.842.0
Harder · A24194111.357.1
Harder · B2417358.839.1
Harder · C2416408.841.6

Method

Written down first,
then run.

The protocol and its thresholds were fixed before any result was seen. Each arm adds one thing to the one before it, so any difference has one cause.

  1. ABaseline

    The agent with its plain tools: read, write, run. No context engine, no cache markers.

  2. B+ context engine

    The Rust engine parses the project, builds a symbol graph and sends the model only the definitions the task needs.

  3. C+ context engine + cache

    B, plus cache markers on the stable prefix and a moving breakpoint on the newest message.

  4. R+ research helpers

    A, plus research tools the model may call. It never called them in 60 runs, so no benefit could be measured; they stay off.

  • Model

    Claude Opus 5.5 through OpenRouter, in every arm.

  • Machine

    One Mac (macOS 26.6, arm64); the same tools and verifier for every run.

  • Main set

    20 tasks, 8 tuning and 12 held-out, × 4 arms × 3 repeats = 240 runs, on 27 Sep 2026.

  • Harder set

    8 new held-out tasks (LRU eviction, RFC 4180 CSV, SemVer precedence, a symptom-only bug in a 200-file repo and more) × 3 arms × 3 repeats = 72 runs.

  • Acceptance

    Hidden tests the agent never sees, run on the final tree in a container with no network.

  • Task validity

    The hidden tests fail on every original project and pass on every reference solution.

  • Cost

    Total measured cost ÷ accepted runs, from the provider’s own cost field.

  • Fixed in advance

    An arm can’t become the default if it is worse than A on more than one held-out task.

  • Spend

    $8.37 for the main set, $3.22 for the harder set.

Accepted by hidden tests312 / 312 runs

A B C R · One square per run

Limits

What this
does not show.

A benchmark is only as honest as its edges. These are ours.

  1. No other product

    The only comparison is our own baseline. Nothing here ranks us against another tool.

  2. No other model

    Every run used Claude Opus 5.5. Another model may behave differently.

  3. No quality gap

    Every arm passed every task, 312 of 312, so these sets can’t tell the arms apart on quality.

  4. Small and medium tasks

    A much larger set would be needed to see quality gaps on big, multi-step work.

  5. No live resource use

    Only the core engine’s memory was measured, in a local soak with no model calls.

  6. No confidence interval

    The statistical confidence of the success rate is undefined at this sample size.

Decisions

Written down
before it was built.

Thirty-four architecture decision records so far. Each one says what we chose, what we turned down, and why. A dozen of them:

  1. ADR 0039Pentest module: fixed, scope-confirmed tools
  2. ADR 0031Forge Cloud: one sandbox account behind a gateway
  3. ADR 0030Daytona as an optional cloud sandbox
  4. ADR 0028The agent works in the project folder itself
  5. ADR 0024Conversation threads, auto-apply and undo
  6. ADR 0022Studio: image and video generation
  7. ADR 0013Reliability, security and the P12 benchmark
  8. ADR 0011Verifier, impact analysis and repair
  9. ADR 0010Provider cache and long conversations
  10. ADR 0009The Structural Context Engine
  11. ADR 0005Sandbox, operation ledger and recovery
  12. ADR 0001Platform, toolchain and the process boundary

Questions

Why compare against your own baseline instead of other tools?
Because it isolates one thing: what the context engine and the cache change, with the same model, machine, tasks and verifier. A cross-product comparison would mix in different models, prompts and interfaces. We haven’t run one, so we don’t claim one.
Is the 49% saving on every task, or an average?
It is total cost divided by accepted runs. Per task, arm C’s mean cost was lower than the baseline’s on all 20 main-set tasks and all 8 harder tasks. The explorer and the per-task chart show each one.
Does the cache change the answers?
No. Cache markers don’t change the request content; a test checks the text is identical. A prefix change simply misses the cache once and the answer stays correct: that is the full-miss row in the ladder.
Why is arm R here if it was never used?
It was part of the protocol fixed in advance, so its result is reported as it came out: the model never called the research helpers in 60 runs. They stay off by default.
What does accepted mean?
A run counts only if hidden tests pass: tests the agent never saw, run on the final tree in a container with no network.
Can I check the numbers?
Every chart has its data table one click below it, and the explorer computes everything in your browser from the 312 run records. See how the engine works.
Next divisionRuntimeContext, sandbox, tests and cache: what a model sees and what it may do.