LAB 06 · WHICH MODEL SHOULD REVIEW YOUR CODE?

Which model should review your code?

We gave four models the same 60 cases to review, twice each, and counted what they caught and what they flagged that was fine. We measured a second reviewer on 60 more changes, and a third set-up that reviews and then checks its own findings. And below are pull requests on this site where an AI reviewer said no.

What it is.

The lab in plain words, and how it runs.

Before a change to code is merged, someone should check it. This lab asks which model should do it: a good reviewer catches the real problems and leaves the good changes alone. It is made of three experiments.

In the first, we gave 4 models the same 60 cases to review, 30 with a known defect and 30 clean, and each model reviewed every case twice. We counted what each caught and what it flagged that was fine. In the second, a reviewer that runs as a tool checked 60 other changes we wrote to keep or break a set of rules, not the known defects. In the third, one model reviewed and then checked its own findings.

What came out: on catches, no model was clearly ahead: the models were up to 5 catches apart, and a single model's own runs were up to 5 apart. Each caught 26 to 31 of 60 tries (30 known defects, each reviewed twice). On false alarms, the models were further apart, up to 18: each flagged 30 to 48 of 60 tries (30 clean cases, each reviewed twice). The tool reviewer met the bar set for it beforehand. Checking its own findings did not get that reviewer out of its false alarms. Every reviewer here is an AI, not a person, and every case in these experiments was chosen or written by us. 8 of this site's own pull requests record a FIX-NEEDED from the AI reviewer in their description; those receipts are further down.

How it works: three experiments

EXP 017: 4 models, each reviews every case twice

  1. 60 cases to review, chosen by us30 with a known defect, 30 clean; reviewed by MiniMax M3.1 Flash, gpt-oss 120B, Claude Sonnet 5.5, Claude Opus 5.5
  2. Defects caught, per model26 to 31 of 60 tries (30 known defects, each reviewed twice)
  3. Clean cases flagged, per model30 to 48 of 60 tries (30 clean cases, each reviewed twice)
  4. Weighed against what a review costson the frontier: MiniMax M3.1 Flash, gpt-oss 120B, Claude Opus 5.5

EXP 006: a reviewer that runs as a tool, 3 runs per change

  1. 60 other changes we wroteeach written to keep or break a set of rules; not the known defects
  2. 180 runsblind runs: 3, each scored as an error
  3. Scored against the known answerrule-breaking changes accepted: 1 of 90 runs; good changes rejected: 2 of 90 runs
  4. The registered bar: metthe check that it read the changes passed

EXP 012: review, then check its own findings (summary only)

  1. One model reviews, then checks its own findingssummary only: its record holds the counts, not the cases or the reviews, so they are shown only in the box below
  2. Dropped its false alarms: nothe decision registered beforehand

Four models, the same 60 cases.

What each caught, and what it flagged that was fine.

The question: per model, how many of 30 known defects does a review catch, and how many of 30 clean cases does it flag anyway? Each model reviewed all 60 cases twice, so each count is of 60.

AN AUTHORED FIXTURE, WRITTEN BY US

The 60 cases are 58 changes from our own private repositories: 30 with a known defect, 30 clean. The changes are not published; their sealed hashes are. Every model got the same prompt, one request per change, no tools and no file access.

ModelDefects caught, of 60 (each run)Clean cases flagged, of 60 (each run)No verdictOn the frontier
MiniMax M3.1 Flash30 of 60 (14 and 16)48 of 60 (22 and 26)21on
gpt-oss 120B29 of 60 (15 and 14)42 of 60 (22 and 20)0on
Claude Sonnet 5.526 of 60 (13 and 13)30 of 60 (15 and 15)0off
Claude Opus 5.531 of 60 (18 and 13)36 of 60 (16 and 20)0on

The models caught between 26 and 31 of 60, a spread of 5, while one model's two runs differed by up to 5: on catches, none is clearly ahead. False alarms spread wider: Claude Sonnet 5.5 flagged fewest clean changes (30), MiniMax M3.1 Flash the most (48). MiniMax M3.1 Flash gave no verdict 21 times, each counted as a miss or a false alarm.

The decision the experiment registered weighs catches against what one review costs. A model is on the frontier when no other model catches at least as many defects for no more per review. On it: MiniMax M3.1 Flash, gpt-oss 120B, Claude Opus 5.5. Off it: Claude Sonnet 5.5, because another model caught at least as many for no more per review. The price of a review is in the experiment's record, not on this page; this site does not show money figures.

From EXP 017's results record, last written by commit 9c379579, sha256 fb01a382cf425e3619f861b8435e84242b183e3e8b4d7ef2fe60e9a05bb086e0. Re-derived from that record at every build of this page; in CI, the experiment's own frozen analysis, run in its check mode, recomputes the record from the committed runs and must find it unchanged.

A reviewer that runs as a tool.

Does it reject what breaks the rules, and only that?

A different question: does a reviewer that runs as a tool, nina, reject the changes that break the rules and accept the ones that keep them? 3 runs per change, 180 runs. Every run is counted. A run whose own tool calls do not show it read the change is blind, and a blind run is scored as an error, whatever it decided: an earlier measurement had found its reviewer auditing the unchanged code instead.

AN AUTHORED FIXTURE, WRITTEN BY US

The 60 changes are authored: we wrote each one to keep to, or break, a set of architecture rules, so the right answer is known for every one.

  • Rule-breaking changes it accepted: 1 of 90 runs.
  • Good changes it rejected: 2 of 90 runs.
  • Changes given the same verdict in every one of their runs: 57 of 60.
  • Runs where the harness failed: 0 of 180. Blind runs, each scored as an error above: 3 of 180; the registered check allowed at most 18.

Verdict on the registered bar: met; the check that it read the changes: passed. These are rule-checking changes in one codebase, not the known defects above; the two are not one league table.

From EXP 006's results record, last written by commit a4c01980, sha256 3ef7bcf6d0317b2095df953b404e86059f9b84ffc4af6b4cb476ddaa7d9c916c. Re-derived at every build; in CI, the experiment's own frozen scorer, run in its check mode, recomputes the record from the committed run and must match it byte for byte.

Review, then check yourself.

A summary: the counts are published, the reviews are not.

SUMMARY ONLY, NOT REPRODUCIBLE HERE

Can a reviewer talk itself out of its false alarms?

EXP 012 had one model review each change and then check its own findings, to drop the wrong ones. It did not work: the decision registered beforehand was no.

  • Run a: caught 13 of 30 defects, flagged 20 of 30 clean changes.
  • Run b: caught 14 of 30 defects, flagged 22 of 30 clean changes.

This is a summary. The experiment's committed record holds these counts, not the reviews they come from, so this page, and anyone reading this repository, cannot re-run them. From EXP 012's results record, commit 3d870da3, sha256 f2e9d313681d0576c4bf1ced44e3147eadec34fcab8dfdb946bfbb4fac86b652.

Re-run this.

Three commands recompute the two records above. You need a clone and Node, nothing else.

Clone this site's repository, use Node 22, and from its root run these commands. Each one needs nothing installed and nothing from the network, and prints exactly the line under it. On macOS, shasum -a 256 in place of sha256sum prints the same line.

  1. node experiments/review-frontier/analyse.mjs --check

    prints

    EXP 017: results.json matches the frozen analysis

    EXP 017's frozen analysis recomputes its results record from the committed runs and finds it unchanged.

  2. sha256sum experiments/review-frontier/results.json

    prints

    fb01a382cf425e3619f861b8435e84242b183e3e8b4d7ef2fe60e9a05bb086e0  experiments/review-frontier/results.json

    That record is the one this page cites: the same sha256.

  3. node experiments/nina-changes/write-results6.mjs --check

    prints

    PASS experiments/nina-changes/results/results.json recomputes byte for byte (sha256 3ef7bcf6d0317b2095df953b404e86059f9b84ffc4af6b4cb476ddaa7d9c916c)

    EXP 006's frozen scorer recomputes its results record byte for byte, and prints the sha256 this page cites.

These are the commands and the lines a CI job of this repository runs and checks, word for word, on every change and every day (the lab kits workflow). The expected hashes are the ones this page's record cites; when an experiment record changes, its new hash comes from the record, not from this text.

When the reviewer said no.

FIX-NEEDED receipts on this site's own pull requests.

This repository's rule is that an independent review examines the final change before it merges. On the pull requests below, that reviewer was an AI, run apart from the agent that wrote the change, and it first answered FIX-NEEDED; the change was then fixed and reviewed again. Here are those receipts, as the pull requests themselves record them.

A SEPARATE AI REVIEWER, NOT A HUMAN

Every verdict below is a separate AI reviewer's, not a human's. A FIX-NEEDED is what it asked for, not proof that the code was wrong.

Four more pull requests, #56, #54, #53, #50, name the review step in their landing plan but record no FIX-NEEDED of their own, so they are not shown as catches.

What this does not show.

  • Each experiment is one corpus, one prompt, one moment. A model's next version, or a different prompt, can change the counts.
  • We chose the cases with known defects and wrote the rule-checking changes ourselves. Neither is a random sample of the bugs a team meets.
  • The review experiments measure catches and false alarms, not whether a team would act on a finding.
  • The receipts are the reviewer's verdicts as the pull requests record them; this page does not re-judge them.

Its runs.

The runs of this lab's check that its record holds, each with its own record hash. The org machines' runs re-run this lab's kit at the pinned, reviewed commit of this repository, not at its moving main; moving them to a newer commit is a reviewed change to the lab runner. A day whose org run would check other results than this page cites is not published here: it is a red day, recorded in the runner's own run.

LATEST RUN · github-hosted · commit 88779efe · record 04fa0d5ba8de

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

Each day on the org machines

One point per day the record holds org runs for: how many org machines reproduced the committed state hash. A value holds until the next day's runs; a day with no run is no point.

org machines that reproduced the hash, held until each change: 2026-10-29: 2; 2026-10-30: 2; 2026-10-31: 2; 2026-11-01: 2; 2026-11-02: 2; 2026-11-03: 2; 2026-11-04: 2; 2026-11-05: 2; 2026-11-06: 2; 2026-11-07: 2; 2026-11-08: 2; 2026-11-09: 2; 2026-11-10: 2; 2026-11-11: 2; 2026-11-12: 2; 2026-11-13: 2; 2026-11-14: 2; 2026-11-15: 2; 2026-11-16: 2; 2026-11-17: 22026-10-292026-11-1722222222222222222222
  1. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  2. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  3. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  4. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  5. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  6. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f93aa888409b; org-ci-heavy, ubuntu-latest
  7. : same hash on both org machines, 2 of 2 reproduced the committed hash; record a43b8f02a211; org-ci-heavy, ubuntu-latest
  8. : same hash on both org machines, 2 of 2 reproduced the committed hash; record a43b8f02a211; ubuntu-latest, org-ci-heavy
  9. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; ubuntu-latest, org-ci-heavy
  10. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; ubuntu-latest, org-ci-heavy
  11. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  12. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; ubuntu-latest, org-ci-heavy
  13. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  14. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  15. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  16. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  17. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; ubuntu-latest, org-ci-heavy
  18. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; ubuntu-latest, org-ci-heavy
  19. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest
  20. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 04fa0d5ba8de; org-ci-heavy, ubuntu-latest