The weekly
broker probe.
Once a week a scheduled job sends 20 identical tiny calls to each of 2 fixed models through every broker we hold a key for, and fetches each broker's model catalogue. It records calls that fail, calls answered by a different model than the one asked for, latency, and models added, removed or re-priced since the week before.
Week by week.
| Week | Broker | Model | Answered | 5xx | Other model served | p50 / p95 ms | Catalogue change |
|---|---|---|---|---|---|---|---|
| 2026-W46 | OpenRouter | claude-haiku-4-5 | 20/20 | 0 | 0 | 636 / 776 | +21 −12 ~36 |
| 2026-W46 | OpenRouter | gpt-4o-mini | 20/20 | 0 | 0 | 568 / 717 | +21 −12 ~36 |
| 2026-W46 | Broker B | not probed: its terms are under review before any measurement is published | |||||
| 2026-W46 | Broker C | not probed: its terms are under review before any measurement is published | |||||
| 2026-W45 | OpenRouter | claude-haiku-4-5 | 20/20 | 0 | 0 | 649 / 830 | +6 −57 ~36 |
| 2026-W45 | OpenRouter | gpt-4o-mini | 20/20 | 0 | 0 | 1074 / 1480 | +6 −57 ~36 |
| 2026-W45 | Broker B | not probed: its terms are under review before any measurement is published | |||||
| 2026-W45 | Broker C | not probed: its terms are under review before any measurement is published | |||||
| 2026-W44 | OpenRouter | claude-haiku-4-5 | 20/20 | 0 | 0 | 737 / 1322 | +2 −3 ~17 |
| 2026-W44 | OpenRouter | gpt-4o-mini | 20/20 | 0 | 0 | 578 / 1209 | +2 −3 ~17 |
| 2026-W44 | Broker B | not probed: its terms are under review before any measurement is published | |||||
| 2026-W44 | Broker C | not probed: its terms are under review before any measurement is published | |||||
| 2026-W43 | OpenRouter | claude-haiku-4-5 | 20/20 | 0 | 0 | 935 / 1050 | first snapshot |
| 2026-W43 | OpenRouter | gpt-4o-mini | 20/20 | 0 | 0 | 677 / 1357 | first snapshot |
| 2026-W43 | Broker B | not probed: its terms are under review before any measurement is published | |||||
| 2026-W43 | Broker C | not probed: its terms are under review before any measurement is published | |||||
How it is measured.
The prompt is Reply with the single word: pong
, at most 16 tokens back, temperature 0. A call counts as answered when it returns HTTP 200 with text. A served model counts as the requested one when the model id the broker returns names the same model family. The job runs on a GitHub-hosted runner under the lab's own GitHub identity, with dedicated, spend-capped keys, and only on its schedule or by hand: never on a pull request. The script is scripts/broker-probe.mjs; each week's record is site/data/broker-probe/ plus the week, as JSON.
2 of 3 brokers are shown by letter: a broker is named next to a number only when its published terms, as recorded in site/data/provider-terms.json, allow it. A broker shown as "not keyed" had no dedicated capped key that week; one shown as "not probed" has terms that are still under review, so no call is made to it at all.
Twenty calls a week cannot show a small error rate, and one runner location is not every location. This page reports what happened; it does not rank brokers.