Scope clarification · . The phrases “for this kind of work” and “cheapest proven model” below should be read narrowly: these results cover the named eight tasks, fixed comparison runs and pre-registered rule. They do not establish a general routing policy or savings reproduced by an outside party. The original account remains below. Publication standard.
EXP 018 compared a frozen table copied from our routing rule with two fixed choices on eight real coding tasks: always Claude Opus 5.5, and always GPT-6 Luna. Its answer was "incomplete": one of the three routed runs lost its last task to its own spend stop, and EXP 018 allows no re-runs. EXP 019 is a new experiment. It ran the same routed arm three more times, with the same table, tasks, agent and sandbox. Those three runs were compared with EXP 018's fixed arms exactly as published, under EXP 018's decision rule.
All three runs completed. Under the rule we wrote down in advance, routing beats always-Opus and does not beat always-Luna. Routing paid $0.143 per case fixed, against $0.247 for Opus and $0.043 for Luna. That is about 42% less than Opus and about three times Luna. It fixed 56.9% of the target cases, against 64.7% for Opus and 52.2% for Luna.
Read the Opus win carefully. Routing fixed 7.8 points fewer cases than Opus. The rule allows a shortfall up to the larger spread between replicates, and routing's own three runs varied by 11.3 points (from 50.4% to 61.8%). So the rule calls this "not lower beyond the spread". A steadier routed arm, or a tighter allowance, could have ruled the other way. The result is "cheaper, and not measurably worse by this rule". It is not "as good".
What was fixed in advance
The pre-registration was merged on 23 October 2026 at 18:41:50 UTC, and the first counted cell started at 18:42:27 UTC. It pins the same private harness as EXP 018 (same commit, same file hashes). It also pins EXP 018's published results file and EXP 018's analysis script by hash. This experiment imports that analysis rather than copying it, and refuses to run if either file changed. Nothing was amended.
- Not pooled: EXP 018's two complete routed runs are not combined with these three. Only EXP 018's fixed arms are reused.
- Spend: at most $78, so the replay as a whole (EXP 018 plus 019) stays within its $275 cap. The three runs went one after another. Each ran on its own key, minted just before it started, with a limit of $78 minus what the earlier keys had billed: $78.00, then $58.08, then $39.84.
- Time gap: the fixed arms ran 13:27–16:49 UTC, these runs 18:42–21:53 UTC the same day. Changes behind the model ids in between would not be detected.
The result
| Arm | Share, per run | Mean share | $ per case fixed | Billed, per run | Out-of-sample |
|---|---|---|---|---|---|
| Routed, EXP 019 (3 fresh runs) | 58.4% / 61.8% / 50.4% | 56.9% [42.4%–71.3%] | $0.143 [$0.095–$0.191] | $19.92 / $18.24 / $19.52 | 23/114 |
| Always Claude Opus 5.5, EXP 018 | 64.7% / 65.5% / 63.9% | 64.7% [62.6%–66.8%] | $0.247 [$0.083–$0.411] | $44.14 / $26.61 / $43.00 | 55/114 |
| Always GPT-6 Luna, EXP 018 | 51.3% / 53.4% / 52.1% | 52.2% [49.6%–54.9%] | $0.043 [$0.020–$0.067] | $6.23 / $4.17 / $5.76 | 46/114 |
What else the runs show
The cheap part of the table is the shaky part. On one of the two tasks the table's picks were never chosen on, GPT-6 Luna fixed 0, 12 and 0 of 16 cases across these runs. In EXP 018 the routed arm fixed 13 of 16 there in every run. The routed arm's out-of-sample share fell from 51.1% in EXP 018 to 20.2% here. Out-of-sample is exactly where a routing table's choices are not backed by measurement.
Cost is steadier than in EXP 018. The three runs billed $19.92, $18.24 and $19.52. In EXP 018 the routed runs billed $20.61, $18.27 and $27.87, the last including one $12.95 cell. No cell here hit a stop, and no cell timed out.
One sandbox refusal inside a cell's own files. In one GPT-6 Sol cell, the sandbox refused one access to a test file in the cell's own workspace. The cell was graded as usual (34 of 48 cases). EXP 018 had one such cell in 72.
No silent model swaps. Every cell was served by the model it asked for.
What it means
Measured, under a rule fixed in advance: on these eight tasks, routing by our written rule cost about 42% less per case fixed than always using Opus 5.5. It fixed fewer cases, and the rule allowed the difference only because the routed runs varied as much as they did. Always using GPT-6 Luna was about a third of routing's cost per case and fixed 4.6 points fewer cases. So for this kind of work, "route by the rule" is a real saving against "always the premium model", and "always the cheapest proven model" is cheaper still. The out-of-sample tasks are a warning: where the table's picks were not measured, its cheapest pick was unreliable.
Evidence and its boundary
The results are recomputed in CI by the frozen analysis from every cell and every key's bill. There is also a spend ledger. Tasks appear only as t1 to t8. The corpus is private, so you can check the hashes, the method, the counts and the bills, but you cannot rerun the tasks. This run billed $57.67 of its $78, and the replay as a whole $254.33 of $275. The run covered 24 cells, between 18:42:27 and 21:52:49 UTC on 23 October 2026.
Pre-registered experiment. Eight tasks from one repository, three runs of one arm compared with another experiment's published arms, one broker. The routed arm is in-sample on six of the eight tasks.