Odin can reach the same model through several brokers. If they bill the same for it, the choice between them can rest on speed, reliability and jurisdiction alone. EXP 016 sent the identical request, for the same model, through three brokers at the same moment, for each of 60 small code changes, twice: 720 calls.
Not quite the same. Two brokers billed exactly the list price of each call's own tokens, on every successful call. The third billed exactly 10% above list, on every successful call, for both models. Under the pre-registered rule (equivalent only within 5%), both arms show a measured price spread of 1.10. No broker ever served a different model than the one requested.
What was fixed in advance
The question, the two models, the 60 changes, the request, the runner, the analysis and the decision rule were published before the first counted call and frozen by sha256; the record was merged at 05:02 UTC and the counted run started at 08:00:44 UTC, after its notBefore of 08:00. It was reviewed once, corrected (the runner's default spend cap and a failure bias in one metric) and reviewed again before it merged.
- Models: Claude Sonnet 5.5 and GPT-4o mini, each under each broker's own id. Two brokers were pinned to the model vendor's own API; the third offers no pin and served both models through other cloud providers. So this compares the brokers as a buyer meets them, not one identical upstream.
- Changes: the 60 public code changes of EXP 005, each sent with the same review instruction, at most 400 tokens back.
- Decision rule: per arm, the billed-to-list ratio of each call (the cost the broker reports, divided by the list price of that call's own tokens); price-equivalent if the largest broker median is at most 1.05 times the smallest.
The result
| Model · broker | Billed ÷ list (median) | Answered | p50 / p95 latency | Other model served |
|---|---|---|---|---|
| Claude Sonnet 5.5 · Broker A | 1.00 | 114 of 120 | 3,198 / 5,145 ms | 0 |
| Claude Sonnet 5.5 · Broker B | 1.10 | 113 of 120 | 2,967 / 5,059 ms | 0 |
| Claude Sonnet 5.5 · Broker C | 1.00 | 116 of 120 | 3,381 / 10,080 ms | 0 |
| GPT-4o mini · Broker A | 1.00 | 120 of 120 | 1,594 / 2,239 ms | 0 |
| GPT-4o mini · Broker B | 1.10 | 120 of 120 | 1,626 / 2,891 ms | 0 |
| GPT-4o mini · Broker C | 1.00 | 120 of 120 | 1,375 / 1,739 ms | 0 |
Latency is wall-clock time to the full response from one laptop in the Netherlands, with the median's 95% interval in the data. Only on GPT-4o mini are two medians' intervals apart: Broker C answered faster than both others there. On Sonnet the intervals overlap, so no broker is called faster.
Where the calls failed
All 360 GPT-4o mini calls were answered. On Sonnet, 17 of 360 were not, in two ways. Broker B returned HTTP 503 "no upstream provider is currently available" 7 times. Brokers C and A returned HTTP 200 with no answer text 4 and 6 times: the model spent the whole 400-token budget on its own reasoning (finish reason "length"), and those calls were still billed. Under the record a call without an answer is a failure, so the cost of these empty answers, $0.049 in all, is in the spend ledger but outside the price ratio. Our pre-registered prediction of error rates below 2% was wrong for Sonnet: 5.0%, 5.8% and 3.3% for Brokers A, B and C.
The second price test, billed dollars per 60 changes, came out "not measurable" for Sonnet: every broker failed at least once in each replicate, and the record counts only complete replicates so that failures cannot make a broker look cheaper. For GPT-4o mini it was $0.0051, $0.0057 and $0.0052 per 60 changes for Brokers A, B and C, a spread of 1.12 that includes the noise of answer length.
What it means
Measured: for these two models, on list-priced uncached calls, two brokers pass the vendor's list price through unchanged and one adds 10%; that markup is stable call by call, so it can be priced into a routing decision exactly. Measured: none of the three silently swapped the model. Not measured: discounts, caching, long contexts or answer quality.
Evidence and its boundary
Every call is published: the results, recomputed in CI from the run's rows, and a per-call spend ledger. The brokers are shown as letters by Amendment 01, written after the run and before publication: two of the three have terms we have not yet cleared for naming them next to a number. The amendment changes labels only; the frozen analysis scores the published rows. The pre-registration, published before that review, lists the three brokers and says which one billed 10% above list in calibration, so a careful reader can match at least one letter. The run cost $1.40 as the brokers reported it.
Pre-registered experiment. One client, one day, two models, three brokers.