Our earlier review measurements put models on different seats. Flash and gpt-oss ran on a tool-free seat. Opus ran on an agentic seat that could read outside its working directory, including the answer key. EXP 017 puts four models on one seat: one request per change, the same prompt, reasoning on, no tools and no file access, all through OpenRouter on one dedicated key. Each model reviewed the same 58 private changes twice. Those changes hold 30 known defects and 30 clean cases.
The models caught about the same number of defects. The price per review differed by more than a hundredfold. Pooled over two runs, they caught between 26 and 31 of 60 defects, and every interval overlaps every other. A review cost 0.06 cents with gpt-oss-120b, about 1.5 cents with MiniMax M3.1 Flash, 4.6 cents with Claude Sonnet 5.5 and 8.3 cents with Claude Opus 5.5, as billed by the broker. Under the rule we wrote down in advance, gpt-oss-120b, Flash and Opus are on the frontier, and Sonnet is not. Opus is on it by one catch, which is within run-to-run noise.
What was fixed in advance
The question, the four models, the seat, the prompt, the scorer, the analysis and the decision rule were published before the first counted call (merged 23 October 2026, 05:45:56 UTC). The private harness that ran the reviews is pinned there by hash. Its runner refuses to start if any pinned file differs. Nothing was amended.
- Seat: one HTTPS request per review, with reasoning enabled at the provider's default effort and at most 32,000 output tokens. The broker reports the cost of each call. A response naming a different model counts as a no-verdict. Every one of the 463 counted reviews that got a response named the requested model; one Flash review got none.
- Corpus: the sealed set used in the review-bench note and in EXP 012: 58 private diffs, 30 seeded defects, 30 clean cases.
- Scoring: the existing review-recall scorer, unchanged. A defect is caught when the verdict is not "ship" and a finding names the defective file. A false alarm is a clean change that does not get "ship". A no-verdict, meaning a failed or unparseable reply after one retry, counts as a miss on a defect and a false alarm on a clean change.
- Decision rule: a model is on the frontier if no other model catches at least as many defects (pooled, of 60) for no more money per review, while being strictly better on one of the two.
The result
| Model | Defects caught | False alarms | Cost per review | Median seconds, a / b | No verdict | On the frontier |
|---|---|---|---|---|---|---|
| MiniMax M3.1 Flash | 14 / 16 · 30/60 [38–62%] | 22 / 26 · 48/60 [68–88%] | 1.5 cents | 187.5 / 105 | 21 | yes |
| gpt-oss-120b | 15 / 14 · 29/60 [36–61%] | 22 / 20 · 42/60 [57–80%] | 0.06 cents | 17.5 / 19 | 0 | yes |
| Claude Sonnet 5.5 | 13 / 13 · 26/60 [32–56%] | 15 / 15 · 30/60 [38–62%] | 4.6 cents | 19 / 17.5 | 0 | no |
| Claude Opus 5.5 | 18 / 13 · 31/60 [39–64%] | 16 / 20 · 36/60 [47–71%] | 8.3 cents | 22 / 23 | 0 | yes |
Sonnet is off the frontier because Flash and gpt-oss both catch more defects for less money. Gpt-oss and Flash are on it as the two cheapest seats, and Opus is on it because it caught the most. These are differences of one to five catches out of 60, and the per-run counts move by up to five between replicates of the same model. So the frontier this rule draws is mostly a price ranking. The catches do not separate the models.
What else the runs show
False alarms separate the models more than catches do. Sonnet flagged 15 of 30 clean changes in both runs. Opus flagged 16 and 20, gpt-oss 22 and 20, and Flash 22 and 26. No model comes near the bar of at most 2 that EXP 012 used. A secondary three-way frontier (catches, false alarms, cost) keeps all four models, because Sonnet has the fewest false alarms.
Flash's false-alarm count is partly an artefact of this seat. Flash gave no verdict on 21 of its 116 reviews. Nineteen of those ran into the seat's 32,000-token output limit while still reasoning and returned no reply; one timed out and one replied without a verdict. As pre-registered, each counts as a miss on a defect and an alarm on a clean change. In EXP 012, on Ollama Cloud with no output limit set, Flash's first pass gave no verdict on 1 and 2 of 58 changes. Its catches here (14 and 16) are close to the 16 and 16 it caught on that seat in note 005. The other three models never failed to give a verdict.
gpt-oss-120b did better here than on Ollama. It caught 15 and 14 defects, against 11 and 12 on Ollama Cloud's tool-free seat with the same prompt. On OpenRouter, its 116 reviews were served by 15 different upstream hosts. This experiment does not separate a host effect from run-to-run variation.
Cost and time. The 464 counted reviews are charged $16.86 in the results: Opus $9.66, Sonnet $5.37, Flash $1.76 and gpt-oss $0.07. Of that, $16.78 is the cost the broker reported in its responses, and $0.08 is the pre-registered upper bound for 4 attempts that got no response at all (all Flash). The dedicated key's own usage counter rose by $16.81 over the window, so those 4 attempts were billed about $0.03, below the bound. Sonnet and Opus took 20–25 minutes per run of 58 changes. Flash took 4 to 5.6 hours, because of its long reasoning and the retries.
Reading the data files. The analysis was frozen before the runs, and three of its fields need a gloss.
- A seat's
medianLatencySecis the median of its two run medians, which for two runs is their mean. This note reports the per-run medians. - Each run's
keyUsageUsdreads the shared key's counter at that run's start and end. Up to three runs were in flight at once, so only the window's first and last readings give a total; no single run's difference is its own cost. harnessExit2 is the review harness's own recall gate failing: it wants 90% of defects caught, and no seat came close. A run is complete when its scorer report and all 58 reviews are present, which holds for all 8. One Flash review got no response on either attempt, so 115 of Flash's 116 reviews carry a served model.
What it means
Measured: on this corpus and this seat, paying about 130 times more per review (Opus against gpt-oss-120b) bought 2 more catches out of 60, which is inside the noise. It also bought fewer false alarms: 36 against 42 of 60. Measured: the cheaper Anthropic model, Sonnet 5.5, caught the fewest defects and also flagged the fewest clean changes. Expected, not measured: a review seat chosen on price alone would do about as well at catching defects as any of these, and the remaining difference between models is in false alarms. A council or a cheaper prompt aimed at false alarms is a separate experiment.
Evidence and its boundary
This is a publicly pre-registered experiment on an internal corpus. The diffs come from private repositories, so cases, findings and answers are not published. The counts are, along with every run's per-case outcomes (numbered, not named), each review's billed cost, latency and served model, and a per-review spend ledger. node experiments/review-frontier/analyse.mjs --check recomputes the results from those files, and the site's test suite runs it.
The counted runs ran on one laptop, on 23 October 2026 from 05:46:13 to 11:24:34 UTC. There were 8 runs of 58 reviews each, at most 3 at a time, and each ended with the canonical review log byte-for-byte unchanged. The scorer credits a catch when the verdict is not "ship" and a finding names the defective file; it does not check the diagnosis. The fleet reviewed some of these changes live, so a model may recognise one; that is not measured.
What remains open
- Flash under a higher output limit, or with reasoning capped, so that its no-verdicts stop counting against it.
- Pinning gpt-oss-120b to one upstream host, to see whether its better result here is the host or noise.
- Something that cuts false alarms without cutting catches. Neither a stronger model (this note) nor a self-check (EXP 012) did.
Pre-registered experiment. Private diffs; aggregates and per-case outcomes only. One broker (OpenRouter), whose terms we read before naming it here. Prices are the broker's on 23 October 2026.