Every model reviewer we have measured on our sealed corpus catches about half of 30 real defects and also flags about two-thirds of 30 clean changes. A reviewer like that cannot block a merge on its own. EXP 012 tested the cheapest fix we could think of: let the same model check its own alarms. In a second pass it had to name a concrete failing input for every finding, or drop it.
It did not work. MiniMax M3.1 Flash kept 249 of the 269 findings it had raised. False alarms ended at 20 and 22 of 30 clean changes in the two counted runs. The bar, written down before any counted call, was at most 2. Defects caught stayed above the floor of 12, at 13 and 14 of 30. Under the published decision rule, the answer is no.
What was fixed in advance
The question, the model, the corpus, both prompts, the scorer and the decision rule were published before the first counted call. An independent review of that record found two defects in the runner: the published layout could not run, and a malformed second-pass reply could silently clear an alarm, which would have biased the result toward the hypothesis. A first counted attempt had already started; it was stopped after 4 of 58 changes per run, excluded from every result, and disclosed in Amendment 01, which was reviewed again before the counted runs restarted. Nothing else changed.
- Model: MiniMax M3.1 Flash on Ollama Cloud, one direct call per pass, no tools and no file access. No other model was called.
- Corpus: the same sealed set used in the review-bench note: 30 real defects and 30 clean changes in 58 private diffs.
- Arm A1, review then verify: pass 1 is the ordinary review. If it is not "ship" and lists findings, pass 2 sees the change, the diff and the numbered findings, and keeps a finding only with a non-empty failing input. If none survive, the verdict becomes "ship". Two runs.
- Arm A0, single review: already measured before the record was written, and disclosed in it: 16 and 16 defects caught, 21 and 19 false alarms. Not re-run.
- Decision rule: "earns its keep" if both A1 runs have at most 2 false alarms and at least 12 catches; "partial" if both have at most 12 false alarms and at least 12 catches; otherwise "no".
The result
| Arm · run | Defects caught | False alarms | No verdict |
|---|---|---|---|
| A1 review, then verify · a | 13 [27–61%] | 20 [49–81%] | 1 |
| A1 review, then verify · b | 14 [30–64%] | 22 [56–86%] | 2 |
| A0 single review · a / b (earlier) | 16 / 16 | 21 / 19 | 0 / 1 |
A no-verdict is a pass-1 timeout or transport error after one retry. As pre-registered, it counts as a miss on a defect and a false alarm on a clean change. The second pass never failed in either run.
Why it failed
The model almost never withdrew an alarm. Across both runs, pass 1 raised 269 findings and pass 2 dropped 20 of them, about 7%. The drops pointed the right way: 13 of the 17 changes that lost a finding were clean. But a clean change usually carried several findings, and one surviving finding is enough to keep the verdict at "fix needed". In each run, the second pass turned exactly one clean change to "ship". That is the whole of its effect on the false alarms.
The other movements are noise, not the second pass. Compared with the earlier A0 runs, the A1 runs caught 3 and 2 fewer defects and flagged 1 fewer and 3 more clean changes. Pass 1 is sampled afresh in every run, so most of that difference is the first review answering differently, not the check. The check itself turned no flagged defect into "ship". We cannot rule out that a dropped finding was the one naming a defect's file, because the runner records how many findings were dropped, not which.
Cost: about 1.8 and 2.0 cents a review at list price, against about 1.5 cents for a single review. Median latency 260 and 317 seconds, on a shared account.
What it means
Measured: asking this model to justify each alarm with a failing input does not cut its false alarms; it can produce a failing input for nearly everything it raised. Measured: the gap between a noisy reviewer and one that may vote is not closed by a self-check prompt on this model. Expected, not measured: a different second opinion (another model, or a narrower question per finding) may do better. That is a separate experiment, and nothing here supports it yet.
Evidence and its boundary
This is a publicly pre-registered experiment on an internal corpus. The diffs come from private repositories, so cases, findings and answers are not published; the counts are. Counted runs ran on one laptop from 21 October 2026, 20:51 UTC, to 22 October, 04:02 UTC. Both ended with the canonical review log byte-for-byte unchanged. The scorer credits a catch when the verdict is not "ship" and a finding names the defective file; it does not check the diagnosis.
What remains open
- A different verifier: a second model, or one question per finding instead of all findings at once.
- A verdict rule that weighs surviving findings by severity rather than letting any one finding block.
- Re-measuring the earlier reviewer seats tool-free, as the review-bench note recommends.
Pre-registered experiment. Private diffs; aggregates only. Flash was free on Ollama Cloud until 22 October 2026 23:59 UTC, then $0.15 / $0.60 per million input / output tokens.