The first study showed that MiniMax M3.1 Flash can point near the broken line in private code it is unlikely to have seen. This follow-on asks the question a reviewer actually faces: given a whole change, does it say "fix this" when there is a defect and "ship it" when there is not?
On our sealed review corpus, Flash's counts sit in the same range as Claude Opus's. It was credited with 16 of 30 defects in both of its standard-prompt runs and flagged 21 and 19 of 30 clean changes. Opus, in an earlier calibration through a reviewer seat with tools, scored 16 and 19 with 20 and 18. The two were run differently, so this is a description, not an equivalence test. At list price that is about 1.5 cents a review. No reviewer we have measured, Flash included, flags few enough clean changes to be given a vote.
The corpus and the bar
The corpus was assembled before this trial and sealed: 30 real defects from our own review history, 30 clean changes that were merged and never fixed afterwards, 58 diffs in all. A reviewer sees one diff and a neutral description, nothing else. The existing scorer, unchanged, credits a catch when the verdict is not "ship" and a finding names the defective file. It checks the file, not whether the finding describes the actual defect, so a catch is an upper bound on correct diagnoses. A clean change that does not get "ship" is a false alarm.
The bar we set for a reviewer that may vote: at most 2 of 30 clean changes flagged in a run, so that the 95% upper bound stays at or below 25%. In the earlier calibration, no seat met it; the strictest run flagged 6.
What we ran
Flash and gpt-oss:120b, our current default for offloaded work, each ran twice per prompt (replicates a and b). Prompt P1 is the live review contract. P3 adds an enumerated fault model: a blocking finding must name one of seven fault classes and a concrete failing input. Both models ran as a tool-free reviewer: one direct call to the model API, with no tools and no file access. Why that matters is the second half of this note.
| Reviewer | Prompt · run | Defects credited | False alarms | No verdict |
|---|---|---|---|---|
| MiniMax M3.1 Flash | P1 · a | 16 | 21 | 0 |
| MiniMax M3.1 Flash | P1 · b | 16 | 19 | 1 |
| MiniMax M3.1 Flash | P3 · a | 15 | 20 | 1 |
| MiniMax M3.1 Flash | P3 · b | 17 | 19 | 0 |
| gpt-oss:120b | P1 · a | 11 | 23 | 1 |
| gpt-oss:120b | P1 · b | 12 | 21 | 1 |
| gpt-oss:120b | P3 · a | 7 | 14 | 1 |
| gpt-oss:120b | P3 · b | 5 | 12 | 0 |
| Claude Opus (earlier calibration) | P1 · a / b | 16 / 19 | 20 / 18 | 0 / 0 |
A no-verdict (a timeout, or an HTTP 429 or 500 while the account was saturated) counts as a miss on a defect and as a false alarm on a clean change, so these figures are conservative. Three of the four planned Flash runs were complete when this note was first published. The fourth, P3 · b, was restarted after an accounting abort and finished on 22 October 2026 at 00:24 UTC; its row and pairing were added then.
Flash against our default, case by case. In each paired run, Flash was credited with more defects that gpt-oss:120b missed than the reverse: 8 against 3 (P1 a), 7 against 3 (P1 b), 11 against 3 (P3 a) and 12 against 0 (P3 b). The direction is consistent. Only the last is significant (exact paired p = 0.23, 0.34, 0.057 and 0.0005), and that gpt-oss:120b run answered in a six-second median against about three minutes for its other runs, so it may have been degraded. Replicates replay the same cases, so they are not pooled. On clean changes the two are level on P1. On P3, gpt-oss:120b got more clean changes right because it ships more of everything.
Flash repeats itself. Between its two P1 runs, only 2 of the 30 defect outcomes flipped. In the earlier calibration, seats flipped between 4 and 9. A reviewer that gives the same answer twice is one you can calibrate.
Unlike the seats in the earlier calibration, Flash did not shed false alarms under P3: 20 and 19 against 21 and 19. It kept its credited recall (15 and 17 of 30), though. Between its two P3 runs, 6 of the 30 defect outcomes flipped.
The finding that matters more than the score
We first ran Flash through our general agent runtime, the way earlier reviewer seats were run, from an empty scratch directory. It did not stay there. With a shell tool available, it left the directory, found the developer checkout on the machine and searched it with git log and grep. That checkout contains the sealed answer key and the real fix history. Every command it ran was read-only, but nothing stopped it.
- An empty working directory is a convention, not a sandbox. Any reviewer with tools can, in principle, look up the answer. Our earlier calibration seats also ran with tools; their numbers may be flattered, and we have not re-measured them yet.
- “No tools” was not tool-free. With tools switched off, the runtime still exposed agent-spawning and dispatch tools. Flash used them: 15 context loads, a spawned agent and 3 dispatches. It never finished within the 900-second limit. The same prompt as one direct API call answered in 69 seconds.
- A silent model swap. During a brief failure to reach the model catalogue, the runtime quietly answered with a different model from another provider under Flash's name, and that call was billed. Our scorer checks the model named in each receipt, so that answer would have been discarded, but the money was already spent.
- Tools may flatter a reviewer. gpt-oss:120b was credited with 17 and 14 defects through the agent runtime but 11 and 12 as a tool-free reviewer. Some of that gap could be answer-key access. This data cannot separate the two.
We now measure reviewers through a tool-free seat. It makes one HTTPS request to a fixed endpoint and has no tools, no filesystem and no configuration lookup. Its model identity is checked against the response, and its cost is computed exactly from the receipt.
Why it matters
Measured: a free open-weight model's credited catches on our own changes fall in the same range as a frontier model's from an earlier, differently run calibration, at a small fraction of the cost, and it is unusually consistent between runs. Whether it matches that model under identical conditions is unresolved until the earlier seats are re-measured tool-free. Measured: no reviewer we have tested yet raises few enough false alarms to vote on its own. Expected, not measured: that a cheap, high-recall reviewer paired with a strict one could beat either alone.
Evidence and its boundary
This is an internally evidenced experiment. The diffs come from private repositories, so cases, findings and answers are not published; the counts are. The corpus, scorer and prompts were fixed before this trial; this study's own run plan was locally frozen, not publicly pre-registered. Runs took place on 21 October 2026, starting 11:51 UTC, on one laptop. Every run ended with the canonical review log byte-for-byte unchanged. The Claude Opus rows come from an earlier calibration on the same corpus (14 October 2026) and were measured through a seat with tools.
What remains open
- Re-measure the earlier seats through the tool-free path before any voting rule is written. That is also the like-for-like comparison this note does not have.
- A diagnosis check: whether each credited finding names the actual defect, not only its file.
- Whether a reviewer can talk itself out of its false alarms. That is the next pre-registered experiment.
- Latency outside peak hours. Up to eight runs shared one account at peak times, so Flash's five-to-seven-minute medians are an upper bound.
Local experiment. The run plan was locally frozen, not publicly pre-registered. Private diffs; aggregates only. Flash was free on Ollama Cloud until 22 October 2026 23:59 UTC, then $0.15 / $0.60 per million input / output tokens.