LABS · HOW WE DECIDE

How we decide.

Before we run an experiment, we write down the question, how it will be scored and what we expect. When it compares two systems, we also let two AI model advocates try to break the plan, one arguing for each side. This page is the record of every plan we froze, every veto, every change after the freeze and every gap we admitted.

Why a veto makes a result worth trusting.

A result is only as fair as its plan. If the plan quietly favours one side, the numbers come out favouring that side, however carefully they are measured.

So when a plan compares two systems, it gets two AI model advocates before it is frozen, one for each side of the comparison. Each AI model advocate is told to make sure its side gets its best realistic design, and may veto anything that would handicap it. A veto is not a suggestion: the plan changes, or the veto is answered in writing.

After the freeze, nothing may change except by a dated amendment, published before the result it affects. When we break that rule, the amendment says so in plain words, and the gap stays listed below.

Every row on this page links the published file it comes from, and every count is computed from those files when the site is built, so a number here cannot drift from its source.

The ledger.

Each experiment in its own order, from the first round to the last amendment. Every row links the file it comes from.

6 pre-registrations; 5 rounds of review by AI model advocates (10 reviews); 37 vetoes, 37 accepted; 7 amendments; 16 disclosed gaps. Every number here is counted from the files linked below when the site is built.

EXP 020: MongoDB on its home ground: where does it fit Odin?

Frozen ; the record is published here (sha256 128087dbba8b…), merged to main as 0af2c486 before any counted run. The plain-words note.

Rounds of review by AI model advocates, one for each store, each with the power to veto.
RoundWhat each AI model advocate vetoedWhat happened
The AI model advocates' design review, before the freeze

MongoDB advocate AI model: Claude, fresh context read the review
5 vetoes: S1 decode and placement; S3 grace; S2 retention; Storage path; Undefined gates

PostgreSQL advocate AI model: Claude, fresh context read the review
6 vetoes: S1 "json loses indexing"; S2 outbox unspecified; S3 rule asymmetry; No primary metric or margin, and no component definition; Image defaults; Extension policy

11 of 11 accepted. Every veto was accepted. source
The AI model advocates' review of amendment 02, before S2's verdict

MongoDB advocate AI model: Claude, fresh context read the review
1 veto: the paragraph. Accepted: the rule

PostgreSQL advocate AI model: Claude, fresh context read the review
1 veto: the paragraph. Accepted: the rule

2 of 2 accepted. This amendment records both reviews and applies what they require. (every reviewer and advocate named here is an AI model) source
The AI model advocates' review of nine readings of the S2 rules (amendment 04), before S2's verdict

MongoDB advocate AI model: Claude, fresh context read the review
1 veto: item 8. Accepted: item 1; item 2; item 3; item 4; item 5; item 6; item 7; item 9

PostgreSQL advocate AI model: Claude, fresh context read the review
1 veto: item 8. Accepted: item 1; item 2; item 3; item 4; item 5; item 6; item 7; item 9

2 of 2 accepted. Both vetoes are accepted: the report checks the disturbances itself and trusts no directory. source
Amendments, each dated and published before the result it affects.
AmendmentWhat it changedWhy
Amendment 01

S2's runner changes, the rule for which cells count, and the nine discarded cells are published the record

  • The pgoutput cursor is the commit LSN
  • Every arm applies in batches
  • The producer's schedule starts after its payloads and connections exist
  • The MongoDB consumer applies once its locally buffered changes are exhausted
  • The disturbance rule decides which counted cells count
  • One cell at a time
EXP 020's S2 runner (the change-feed scenario) was changed four times, and given a rule for which counted cells are kept, after this record was published and after S2's first counted cells ran.
Amendment 02

S2 compares MongoDB with one PostgreSQL variant per scale, chosen once the record

  • One PostgreSQL variant per scale decides the S2 verdict (primary analysis)
  • Per-repetition best is a labelled secondary analysis
Amendment 01 recorded, and left open, a conflict between S2's runner protocol and this record.
Amendment 03

Both advocates accept the S2 pairing rule; the decision paragraph is rewritten as they required (every reviewer and advocate named here is an AI model) the record

  • Amendment 02's decision paragraph is replaced
  • Ties in the variant choice
  • Symmetry of the gate
  • The secondary analysis, extended
Amendment 02 asked both EXP 020 advocates to review its decision paragraph and rule, with veto power, before S2's verdict. (every reviewer and advocate named here is an AI model)
Amendment 04

Both advocates settle nine readings of the S2 rules before the verdict; both veto item 8, so the report checks the disturbance windows itself (every reviewer and advocate named here is an AI model) the record

  • "Same direction" on seed 2
  • "Beats by at least 20 %"
  • Component counts for the tie-break
  • Item 4: the old pairing's gate and components
  • Item 5: invalid vs missing files
  • Item 6: which file names count
  • Item 7: excluded variants
  • Which counted cells a disturbance removes: the report checks it, instead of trusting the discarded/ directory
  • Item 9: the seed-2 selection's candidate set
Writing the S2 report code showed nine places where the published rules (the pre-registration and amendments 01 to 03) leave a reading open.
Amendment 05

S2's run record: the pause, all 13 discards and the report's hashes the record

  • The pause of 13 days and 6 hours
  • The stack-stop incident, with no discard
  • All 13 discarded cell runs, kept in the runner's discarded/ directory and all re-run
  • The disturbance-window refusal rule, as it ran (amendment 04 item 8)
  • The report's outputs, the files it read, and the 60 counted cells, by sha256
  • The verdict, as the report states it
S2 is complete, and its report was computed under the published rules.

Gaps we disclosed

  • These changes were made, and run under, before they were published here, so they were not pre-registered in the strict sense: a reader has the runner owner's account of when each was made, with the commit time, not an earlier public record. source
  • Amendments 02 and 04 move the primary metric a long way, and 04 applies to one arm only, in its favour. The S2 report should be read with them in mind. source
  • Whether a disturbance was heavy is the runner owner's judgement, written in the log for each row; the light rows that were kept are not listed cell by cell here. (every reviewer and advocate named here is an AI model) source
  • The protocol pairs MongoDB, per repetition, with the better passing PostgreSQL variant of that repetition. The pre-registration says 'the other store's best variant'. EXP 021's reviewers rejected a per-repetition best for EXP 021 (it is a best-of-two that tilts toward PostgreSQL). This amendment records the protocol as written and does not decide S2's pairing; any change to it is a further dated amendment, published before S2's report. (every reviewer and advocate named here is an AI model) source
  • Not blind: interim seed-1 results had been seen when this rule was chosen, and the selection uses seed-1 results. source
  • The rule was transferred from EXP 021, a different experiment whose advocates agreed it; EXP 020's own advocates have not yet reviewed it (see review). (every reviewer and advocate named here is an AI model) source
  • Both advocates note, and accept, that choosing the variant on seed 1 and comparing on seed 1 leans slightly toward PostgreSQL, and both call seed 2 out-of-sample. That holds for seed 2's lag values, which never enter the choice; seed 2's gate outcomes do enter it, since a candidate must pass every counted cell on both seeds. (every reviewer and advocate named here is an AI model) source
  • The nine readings were proposed by the runner owner, who also implements them; the advocates reviewed a brief that is not published. Each review says which files it read. (every reviewer and advocate named here is an AI model) source
  • Which code clause implements which of items 4 to 7 and 9 is the publisher's reading: the brief is unpublished, so each item is matched by the runner owner's summary, in its order, and by each review's own words for that item. (every reviewer and advocate named here is an AI model) source
  • The tests were run by the publisher on a copy of the pinned files, not by an independent party. source
  • Every behaviour of the report code that this record describes (refusing, writing nothing, --allow-incomplete, the light-overlap list, the old pairing) is the runner owner's statement and the pinned code's own docstring, checked only by the publisher's run of the private tests. It is not independently verifiable from this repository. source
  • The PostgreSQL advocate's own alternative for item 8 was not adopted; the stricter MongoDB-advocate version was. The thread records no second review of the adopted version by the PostgreSQL advocate. (every reviewer and advocate named here is an AI model) source
  • Whether a disturbance window is heavy or light is the runner owner's transcription of the log. The runner owner's post says six of the calls were judgements; the file as fetched has five windows with a non-empty 'judgement' field. The file, not the post, is what the report reads. (every reviewer and advocate named here is an AI model) source
  • The report code and the windows file are in a private repository and are cited here by commit and sha256, not published; a reader can check a hash but not open them. source

EXP 021: Kill the database mid-session, does anyone notice?

Frozen ; the record is published here (sha256 7e2f31594842…), merged to main as b36684ce before any counted run. The plain-words note.

Earlier version: It is history only: never published, never counted, not the plan. source

Rounds of review by AI model advocates, one for each store, each with the power to veto.
RoundWhat each AI model advocate vetoedWhat happened
The AI model advocates' design review, before the v1 freeze

MongoDB advocate AI model: Claude, fresh context read the review
8 vetoes: topology; client timeouts; the kill; tuned-cell parity; component counting; F2b; the feed; the host

PostgreSQL advocate AI model: Claude, fresh context read the review
8 vetoes: Undefined kill; F2 cannot show split-brain; Unmatched client timeouts and retries; Patroni `etcd:` (v2) would break on modern etcd; The quorum sync mode is missing; The tuned cell is underspecified; The PostgreSQL feed's best variant is missing; No rejoin hygiene

16 of 16 accepted. All 16 vetoes were accepted. source
The AI model advocates' review of the runner's open questions, before the v1.1 freeze

MongoDB advocate AI model: Claude, fresh context read the review
4 vetoes: the per-repetition variant choice (A11); the change stream consumer; false failovers are not in any verdict; F3 under the literal rule (A1)

PostgreSQL advocate AI model: Claude, fresh context read the review
2 vetoes: the tuned cell at 20 / 1 / 3 (A2); the redundancy check does not cover PostgreSQL's rejoin state

6 of 6 accepted. Every veto is accepted. source
Amendments, each dated and published before the result it affects.
AmendmentWhat it changedWhy
Amendment 01

The runner protocol and the design draft are published the record

  • The runner protocol is published
  • The design draft is published
EXP 021's frozen record cites two private inputs by sha256 only: the runner protocol (PROTOCOL-LAB02.md) and the design draft (DESIGN.md).
Amendment 02

The pre-registered single-copy rule fired, so the compared PostgreSQL variants are strict; ten runner fixes are recorded the record

  • The pre-registered single-copy rule fired: the compared PostgreSQL variants are the synchronous_mode_strict ones
  • Ten runner fixes, found by the smoke run (1 to 8) and by the first counted cells (9 and 10)
  • Fix 8 is reviewed: the PostgreSQL advocate accepts it, and its tightening is adopted (every reviewer and advocate named here is an AI model)
  • Fix 9: the first PostgreSQL block's run manifest was written late
  • Fix 10: the verdict script was checked on synthetic cells
  • The counted plan, and its pauses and disturbances so far
The smoke run, which is never counted, did what the record says it is for.

Gaps we disclosed

  • The frozen text binds the feed to 'EXP 020 S2's runner rules (amendments 01-04)', and FREEZE-NOTES.md M7 cites 'amendment 04's buffer rule'. Those are runner amendments in the private repository. EXP 020's published record carries no amendment, so a reader cannot check those four rules here. source
  • The runner protocol and the runner code are private. A reader can check the frozen plan, the reviews, the freeze notes and the hashes here, but not the runner. (every reviewer and advocate named here is an AI model) source

Lab 02: every brain on the same tasks

The brain bench's plans are frozen in a private repository and cited here by commit and sha256, so you can check a hash but not open the plan. They had no AI model advocate round; each prediction was written before its run and scored by a rule fixed in advance.

Each pre-registration is cited by hash: its commit and sha256, from the published results record.
ExperimentPre-registrationPredictions
EXP-W01
Which small local model makes the best agent brain?
commit 3d7137a3, sha256 7191ed201db3…
3 predictions: 1 holds on the point estimate, no detectable difference; 1 fails; 1 not supported. 5 caveats on the lab page.
EXP-W02
Do cloud models make better agent brains than our tuned local model?
commit c3fa1656, sha256 09d4adcd58c5…
4 predictions: 4 holds. 4 caveats on the lab page.
EXP-W03
Are the cloud models really better at following a person's instruction?
commit 1f75de87, sha256 2357a516ac3b…
4 predictions: 4 holds. 2 caveats on the lab page.
EXP-W04
After fixing the bench, is the gap still there?
commit 5a01a950, sha256 cd25acbc7ff1…
5 predictions: 4 holds; 1 holds as registered, not claimed. 6 caveats on the lab page.

This page covers the pre-registrations this site publishes with their AI model review rounds, plus the brain bench's pre-registrations, which are cited by hash. Older experiments have their own records and amendments on their pages; they are not in this ledger yet.