Scope clarification · . The phrase “cheapest proven model” below refers only to the model’s observed results on this experiment’s tasks and scoring rule. It does not establish general reliability, deployment readiness or the cheapest choice on other work. The original account remains below. Publication standard.

Odin's routing rule (rule 27) says which model tier a task deserves. EXP 018 turned the rule, as it is written today, into a frozen table that picks one model per task. Single-file fixes go to GPT-6 Luna, multi-file fixes to GPT-6 Sol, and a change to authentication or credential code goes to Claude Opus 5.5. We compared that table with two fixed choices: always Opus 5.5, and always Luna. The test was eight real changes from our own repository, each replayed from its parent commit and graded by the change's own hidden tests. Each arm did every task three times in an OS sandbox whose only network was OpenRouter. The headline is what each arm paid per hidden test case it made pass.

Under the rule we wrote down in advance, the answer is "incomplete". One routed cell, the eighth task of the third replicate, was cut when its lane's own spend stop ran out. That left two complete routed replicates where the rule needs three. The counts the run did produce are below. They are not a verdict.

So routing cost about half as much per case as Opus, for a share 3 points lower. Luna cost about a third as much as routing, for a share 9 points lower. That is the direction we predicted before the run ("beats Opus, not Luna"). We do not claim it: the record says an incomplete comparison stays incomplete.

What was fixed in advance

The question, the three arms, the routing table, the corpus, the metric, the analysis and the decision rule were published before the first counted cell (merged 23 October 2026, 13:27:12 UTC). The first cell started at 13:27:56 UTC. The private harness is pinned there by hash. Nothing was amended.

The result

Per arm. Share = target cases fixed / target cases graded. Intervals are 95% Student-t over complete replicates. The routed arm's intervals have one degree of freedom. Its $ interval is narrow only because its two complete replicates happened to land within 0.2 cents of each other. Its share interval runs past 100%, which says it carries no information. Billed = OpenRouter's own key counters.
ArmComplete replicatesShare, r1 / r2 / r3Mean share$ per case fixedPooled $ per caseBilledOut-of-sample tasks
Routed (frozen rule-27 table)2 of 365.5% / 57.1% / 61.6%61.3% [8.0%–114.7%]$0.133 [$0.119–$0.147]$0.133$66.7547/92 (51.1%)
Always Claude Opus 5.53 of 364.7% / 65.5% / 63.9%64.7% [62.6%–66.8%]$0.247 [$0.083–$0.411]$0.246$113.7455/114 (48.2%)
Always GPT-6 Luna3 of 351.3% / 53.4% / 52.1%52.2% [49.6%–54.9%]$0.043 [$0.020–$0.067]$0.043$16.1746/114 (40.4%)

Verdict, routing vs always Opus: incomplete (router 2, Opus 3 complete replicates). Routing vs always Luna: incomplete (router 2, Luna 3).

What else the runs show

The cut was the routed arm's own spend stop. Its third replicate drew a costly cell earlier in the lane: one multi-file task cost about $12.95 on GPT-6 Sol, against about $3.49 in the first replicate (both by the driver's own reading). The pre-registration set each lane's driver stop $1 under its key limit, so $29 for a routed lane (spend.runnerStop). By the driver's own readings, the lane's first seven cells came to $27.41, which left $1.59 when the last task came up. The agent ended that cell at its session budget. The key itself had not reached its limit: the lane billed $27.87 of $30. The stop worked as written. It was set too tight for the cost variance of a single cell. The cut cell is reported as an error and counts toward nothing, and nothing was re-run.

No arm fully solved any task. Every one of the 72 cells passed only part of its hidden tests. That is consistent with the 24 September run, where one model solved one task out of 72 cells.

Cost varies more than quality. The fixed-Opus arm's three replicates fixed 64.7%, 65.5% and 63.9% of the cases, but billed $44.14, $26.61 and $43.00. One replicate cost 40% less for the same work. The cost interval is wide for that reason, not because quality moved.

A built-in consistency check. On the one task the table sends to Opus, both arms ran the same model on the same task. Fixed Opus fixed 4 of 22 cases in all three replicates; the routed arm fixed 4 of 22 in both replicates that finished.

Timeouts are Luna's. Five of Luna's 24 cells and two of the routed arm's Luna cells hit the 25-minute limit; no Opus or Sol cell did. A timed-out cell is graded on what it had written.

No silent model swaps. Every cell that got a response was served by the model it asked for.

What it means

Measured, but not decided by the rule: on these eight tasks, a table copied from our routing rule fixed about as many cases as always using Opus, for about half the money per case. Always using the cheapest proven model was cheaper still, and fixed fewer cases. A clean verdict needs a fourth run of the routed arm, or a new pre-registration with a lane stop sized to per-cell variance. Either would be a new record, published first. This experiment stays incomplete.

Evidence and its boundary

The results are recomputed in CI by the frozen analysis from every cell and every key's bill, and there is a spend ledger. Tasks appear only as t1 to t8. The corpus is private, so you can check the hashes, the method, the counts and the bills, but you cannot rerun the tasks. The run billed $196.66 in total, under the $275 cap, between 13:27:56 and 16:49:28 UTC on 23 October 2026 (72 cells).

Pre-registered experiment. Eight tasks from one repository, three replicates, one broker. The routed arm is in-sample on six of the eight tasks.