The weekly
broker probe.

Once a week a scheduled job sends 20 identical tiny calls to each of 2 fixed models through every broker we hold a key for, and fetches each broker's model catalogue. It records calls that fail, calls answered by a different model than the one asked for, latency, and models added, removed or re-priced since the week before.

Week by week.

Each row: one broker, one model, one week. Latency is measured from a GitHub-hosted runner.
WeekBrokerModelAnswered5xxOther model servedp50 / p95 msCatalogue change
2026-W46OpenRouterclaude-haiku-4-520/2000636 / 776+21 −12 ~36
2026-W46OpenRoutergpt-4o-mini20/2000568 / 717+21 −12 ~36
2026-W46Broker Bnot probed: its terms are under review before any measurement is published
2026-W46Broker Cnot probed: its terms are under review before any measurement is published
2026-W45OpenRouterclaude-haiku-4-520/2000649 / 830+6 −57 ~36
2026-W45OpenRoutergpt-4o-mini20/20001074 / 1480+6 −57 ~36
2026-W45Broker Bnot probed: its terms are under review before any measurement is published
2026-W45Broker Cnot probed: its terms are under review before any measurement is published
2026-W44OpenRouterclaude-haiku-4-520/2000737 / 1322+2 −3 ~17
2026-W44OpenRoutergpt-4o-mini20/2000578 / 1209+2 −3 ~17
2026-W44Broker Bnot probed: its terms are under review before any measurement is published
2026-W44Broker Cnot probed: its terms are under review before any measurement is published
2026-W43OpenRouterclaude-haiku-4-520/2000935 / 1050first snapshot
2026-W43OpenRoutergpt-4o-mini20/2000677 / 1357first snapshot
2026-W43Broker Bnot probed: its terms are under review before any measurement is published
2026-W43Broker Cnot probed: its terms are under review before any measurement is published

How it is measured.

The prompt is Reply with the single word: pong, at most 16 tokens back, temperature 0. A call counts as answered when it returns HTTP 200 with text. A served model counts as the requested one when the model id the broker returns names the same model family. The job runs on a GitHub-hosted runner under the lab's own GitHub identity, with dedicated, spend-capped keys, and only on its schedule or by hand: never on a pull request. The script is scripts/broker-probe.mjs; each week's record is site/data/broker-probe/ plus the week, as JSON.

2 of 3 brokers are shown by letter: a broker is named next to a number only when its published terms, as recorded in site/data/provider-terms.json, allow it. A broker shown as "not keyed" had no dedicated capped key that week; one shown as "not probed" has terms that are still under review, so no call is made to it at all.

Twenty calls a week cannot show a small error rate, and one runner location is not every location. This page reports what happened; it does not rank brokers.