Scope clarification · . The phrase “cheapest proven model” below refers only to the model’s observed results on this experiment’s tasks and scoring rule. It does not establish general reliability, deployment readiness or the cheapest choice on other work. The original account remains below. Publication standard.
Odin's routing rule (rule 27) says which model tier a task deserves. EXP 018 turned the rule, as it is written today, into a frozen table that picks one model per task. Single-file fixes go to GPT-6 Luna, multi-file fixes to GPT-6 Sol, and a change to authentication or credential code goes to Claude Opus 5.5. We compared that table with two fixed choices: always Opus 5.5, and always Luna. The test was eight real changes from our own repository, each replayed from its parent commit and graded by the change's own hidden tests. Each arm did every task three times in an OS sandbox whose only network was OpenRouter. The headline is what each arm paid per hidden test case it made pass.
Under the rule we wrote down in advance, the answer is "incomplete". One routed cell, the eighth task of the third replicate, was cut when its lane's own spend stop ran out. That left two complete routed replicates where the rule needs three. The counts the run did produce are below. They are not a verdict.
- The routed arm paid 13.3 cents per case fixed and fixed 61.3% of the target cases (two complete replicates).
- Always Opus 5.5 paid 24.7 cents and fixed 64.7%.
- Always Luna paid 4.3 cents and fixed 52.2%.
So routing cost about half as much per case as Opus, for a share 3 points lower. Luna cost about a third as much as routing, for a share 9 points lower. That is the direction we predicted before the run ("beats Opus, not Luna"). We do not claim it: the record says an incomplete comparison stays incomplete.
What was fixed in advance
The question, the three arms, the routing table, the corpus, the metric, the analysis and the decision rule were published before the first counted cell (merged 23 October 2026, 13:27:12 UTC). The first cell started at 13:27:56 UTC. The private harness is pinned there by hash. Nothing was amended.
- Routing table: copied from our rule as fixed on 23 October, before the run, after an earlier version sent half the tasks to the most expensive tier because of keywords in task titles. The table reads only which files a task changes. Its three model picks come from a run on 24 September that measured 6 of these 8 tasks. On those 6 tasks the routed arm is in-sample. The two tasks that run never saw are reported separately below.
- Metric: per replicate, the arm's OpenRouter key usage after minus before, divided by the target cases fixed. A target case is a hidden test case that failed before the original change and passed after it; there are 238 across the 8 tasks. Per arm: the mean over complete replicates with a 95% t-interval.
- Decision rule: routing beats fixed X if its mean $ per case fixed is lower AND its mean share is not lower than X's by more than the replicate spread. Either arm needs 3 complete replicates.
- Spend: a hard cap of $275 over nine dedicated keys, one per arm and replicate, plus a per-lane stop in the driver.
The result
| Arm | Complete replicates | Share, r1 / r2 / r3 | Mean share | $ per case fixed | Pooled $ per case | Billed | Out-of-sample tasks |
|---|---|---|---|---|---|---|---|
| Routed (frozen rule-27 table) | 2 of 3 | 65.5% / 57.1% / 61.6% | 61.3% [8.0%–114.7%] | $0.133 [$0.119–$0.147] | $0.133 | $66.75 | 47/92 (51.1%) |
| Always Claude Opus 5.5 | 3 of 3 | 64.7% / 65.5% / 63.9% | 64.7% [62.6%–66.8%] | $0.247 [$0.083–$0.411] | $0.246 | $113.74 | 55/114 (48.2%) |
| Always GPT-6 Luna | 3 of 3 | 51.3% / 53.4% / 52.1% | 52.2% [49.6%–54.9%] | $0.043 [$0.020–$0.067] | $0.043 | $16.17 | 46/114 (40.4%) |
Verdict, routing vs always Opus: incomplete (router 2, Opus 3 complete replicates). Routing vs always Luna: incomplete (router 2, Luna 3).
What else the runs show
The cut was the routed arm's own spend stop. Its third replicate drew a costly cell earlier in the lane: one multi-file task cost about $12.95 on GPT-6 Sol, against about $3.49 in the first replicate (both by the driver's own reading). The pre-registration set each lane's driver stop $1 under its key limit, so $29 for a routed lane (spend.runnerStop). By the driver's own readings, the lane's first seven cells came to $27.41, which left $1.59 when the last task came up. The agent ended that cell at its session budget. The key itself had not reached its limit: the lane billed $27.87 of $30. The stop worked as written. It was set too tight for the cost variance of a single cell. The cut cell is reported as an error and counts toward nothing, and nothing was re-run.
No arm fully solved any task. Every one of the 72 cells passed only part of its hidden tests. That is consistent with the 24 September run, where one model solved one task out of 72 cells.
Cost varies more than quality. The fixed-Opus arm's three replicates fixed 64.7%, 65.5% and 63.9% of the cases, but billed $44.14, $26.61 and $43.00. One replicate cost 40% less for the same work. The cost interval is wide for that reason, not because quality moved.
A built-in consistency check. On the one task the table sends to Opus, both arms ran the same model on the same task. Fixed Opus fixed 4 of 22 cases in all three replicates; the routed arm fixed 4 of 22 in both replicates that finished.
Timeouts are Luna's. Five of Luna's 24 cells and two of the routed arm's Luna cells hit the 25-minute limit; no Opus or Sol cell did. A timed-out cell is graded on what it had written.
No silent model swaps. Every cell that got a response was served by the model it asked for.
What it means
Measured, but not decided by the rule: on these eight tasks, a table copied from our routing rule fixed about as many cases as always using Opus, for about half the money per case. Always using the cheapest proven model was cheaper still, and fixed fewer cases. A clean verdict needs a fourth run of the routed arm, or a new pre-registration with a lane stop sized to per-cell variance. Either would be a new record, published first. This experiment stays incomplete.
Evidence and its boundary
The results are recomputed in CI by the frozen analysis from every cell and every key's bill, and there is a spend ledger. Tasks appear only as t1 to t8. The corpus is private, so you can check the hashes, the method, the counts and the bills, but you cannot rerun the tasks. The run billed $196.66 in total, under the $275 cap, between 13:27:56 and 16:49:28 UTC on 23 October 2026 (72 cells).
Pre-registered experiment. Eight tasks from one repository, three replicates, one broker. The routed arm is in-sample on six of the eight tasks.