The question: per model, how many of 30 known defects does a review catch, and how many of 30 clean cases does it flag anyway? Each model reviewed all 60 cases twice, so each count is of 60.
AN AUTHORED FIXTURE, WRITTEN BY US
The 60 cases are 58 changes from our own private repositories: 30 with a known defect, 30 clean. The changes are not published; their sealed hashes are. Every model got the same prompt, one request per change, no tools and no file access.
| Model | Defects caught, of 60 (each run) | Clean cases flagged, of 60 (each run) | No verdict | On the frontier |
|---|
| MiniMax M3.1 Flash | 30 of 60 (14 and 16) | 48 of 60 (22 and 26) | 21 | on |
|---|
| gpt-oss 120B | 29 of 60 (15 and 14) | 42 of 60 (22 and 20) | 0 | on |
|---|
| Claude Sonnet 5.5 | 26 of 60 (13 and 13) | 30 of 60 (15 and 15) | 0 | off |
|---|
| Claude Opus 5.5 | 31 of 60 (18 and 13) | 36 of 60 (16 and 20) | 0 | on |
|---|
The models caught between 26 and 31 of 60, a spread of 5, while one model's two runs differed by up to 5: on catches, none is clearly ahead. False alarms spread wider: Claude Sonnet 5.5 flagged fewest clean changes (30), MiniMax M3.1 Flash the most (48). MiniMax M3.1 Flash gave no verdict 21 times, each counted as a miss or a false alarm.
The decision the experiment registered weighs catches against what one review costs. A model is on the frontier when no other model catches at least as many defects for no more per review. On it: MiniMax M3.1 Flash, gpt-oss 120B, Claude Opus 5.5. Off it: Claude Sonnet 5.5, because another model caught at least as many for no more per review. The price of a review is in the experiment's record, not on this page; this site does not show money figures.
From EXP 017's results record, last written by commit 9c379579, sha256 fb01a382cf425e3619f861b8435e84242b183e3e8b4d7ef2fe60e9a05bb086e0. Re-derived from that record at every build of this page; in CI, the experiment's own frozen analysis, run in its check mode, recomputes the record from the committed runs and must find it unchanged.