EXP 020: MongoDB on its home ground: where does it fit Odin?
Frozen ; the record is published here (sha256 128087dbba8b…), merged to main as 0af2c486 before any counted run. The plain-words note.
| Round | What each AI model advocate vetoed | What happened |
|---|---|---|
| The AI model advocates' design review, before the freeze | MongoDB advocate AI model: Claude, fresh context read the review PostgreSQL advocate AI model: Claude, fresh context read the review | 11 of 11 accepted. Every veto was accepted.source |
| The AI model advocates' review of amendment 02, before S2's verdict | MongoDB advocate AI model: Claude, fresh context read the review PostgreSQL advocate AI model: Claude, fresh context read the review | 2 of 2 accepted. This amendment records both reviews and applies what they require.(every reviewer and advocate named here is an AI model) source |
| The AI model advocates' review of nine readings of the S2 rules (amendment 04), before S2's verdict | MongoDB advocate AI model: Claude, fresh context read the review PostgreSQL advocate AI model: Claude, fresh context read the review | 2 of 2 accepted. Both vetoes are accepted: the report checks the disturbances itself and trusts no directory.source |
| Amendment | What it changed | Why |
|---|---|---|
| Amendment 01 | S2's runner changes, the rule for which cells count, and the nine discarded cells are published the record
| EXP 020's S2 runner (the change-feed scenario) was changed four times, and given a rule for which counted cells are kept, after this record was published and after S2's first counted cells ran. |
| Amendment 02 | S2 compares MongoDB with one PostgreSQL variant per scale, chosen once the record
| Amendment 01 recorded, and left open, a conflict between S2's runner protocol and this record. |
| Amendment 03 | Both advocates accept the S2 pairing rule; the decision paragraph is rewritten as they required (every reviewer and advocate named here is an AI model) the record
| Amendment 02 asked both EXP 020 advocates to review its decision paragraph and rule, with veto power, before S2's verdict. (every reviewer and advocate named here is an AI model) |
| Amendment 04 | Both advocates settle nine readings of the S2 rules before the verdict; both veto item 8, so the report checks the disturbance windows itself (every reviewer and advocate named here is an AI model) the record
| Writing the S2 report code showed nine places where the published rules (the pre-registration and amendments 01 to 03) leave a reading open. |
| Amendment 05 | S2's run record: the pause, all 13 discards and the report's hashes the record
| S2 is complete, and its report was computed under the published rules. |
Gaps we disclosed
These changes were made, and run under, before they were published here, so they were not pre-registered in the strict sense: a reader has the runner owner's account of when each was made, with the commit time, not an earlier public record.
sourceAmendments 02 and 04 move the primary metric a long way, and 04 applies to one arm only, in its favour. The S2 report should be read with them in mind.
sourceWhether a disturbance was heavy is the runner owner's judgement, written in the log for each row; the light rows that were kept are not listed cell by cell here.
(every reviewer and advocate named here is an AI model) sourceThe protocol pairs MongoDB, per repetition, with the better passing PostgreSQL variant of that repetition. The pre-registration says 'the other store's best variant'. EXP 021's reviewers rejected a per-repetition best for EXP 021 (it is a best-of-two that tilts toward PostgreSQL). This amendment records the protocol as written and does not decide S2's pairing; any change to it is a further dated amendment, published before S2's report.
(every reviewer and advocate named here is an AI model) sourceNot blind: interim seed-1 results had been seen when this rule was chosen, and the selection uses seed-1 results.
sourceThe rule was transferred from EXP 021, a different experiment whose advocates agreed it; EXP 020's own advocates have not yet reviewed it (see review).
(every reviewer and advocate named here is an AI model) sourceBoth advocates note, and accept, that choosing the variant on seed 1 and comparing on seed 1 leans slightly toward PostgreSQL, and both call seed 2 out-of-sample. That holds for seed 2's lag values, which never enter the choice; seed 2's gate outcomes do enter it, since a candidate must pass every counted cell on both seeds.
(every reviewer and advocate named here is an AI model) sourceThe nine readings were proposed by the runner owner, who also implements them; the advocates reviewed a brief that is not published. Each review says which files it read.
(every reviewer and advocate named here is an AI model) sourceWhich code clause implements which of items 4 to 7 and 9 is the publisher's reading: the brief is unpublished, so each item is matched by the runner owner's summary, in its order, and by each review's own words for that item.
(every reviewer and advocate named here is an AI model) sourceThe tests were run by the publisher on a copy of the pinned files, not by an independent party.
sourceEvery behaviour of the report code that this record describes (refusing, writing nothing, --allow-incomplete, the light-overlap list, the old pairing) is the runner owner's statement and the pinned code's own docstring, checked only by the publisher's run of the private tests. It is not independently verifiable from this repository.
sourceThe PostgreSQL advocate's own alternative for item 8 was not adopted; the stricter MongoDB-advocate version was. The thread records no second review of the adopted version by the PostgreSQL advocate.
(every reviewer and advocate named here is an AI model) sourceWhether a disturbance window is heavy or light is the runner owner's transcription of the log. The runner owner's post says six of the calls were judgements; the file as fetched has five windows with a non-empty 'judgement' field. The file, not the post, is what the report reads.
(every reviewer and advocate named here is an AI model) sourceThe report code and the windows file are in a private repository and are cited here by commit and sha256, not published; a reader can check a hash but not open them.
source