Updates from
the floor.
Everything we publish here, newest first. We write each experiment down before it runs, publish the result whatever it shows, and correct the record in the open when we get something wrong.
Latest lab runs
One latest run per lab. These are checks of committed records. A matching hash does not establish a new experiment or independent validation.
-
Lab 06: Which model should review your code?
The check printed SAME with 0 breaches. Record
04fa0d5ba8de. -
Lab 05: Can an agent go rogue?
The check printed SAME with 0 breaches. Record
f2c48fc2a28a. -
Lab 13: Does an agent company get better, measured blind?
The check printed SAME with 0 breaches. Record
f419d9215e2b.
NOVEMBER 2026
- RESULTS · EXP 021Kill the database mid-session, does anyone notice?Single host: MongoDB fits four write-outage and two acknowledgement-gap scenarios; PostgreSQL two by feed gate. No acknowledged writes lost; no overall winner. Strict-ack and disturbance uncertainties remain.
- AMENDMENT · EXP 021Kill the database mid-session, does anyone notice?The counted run is complete: 48 discards reconciled; strict acknowledgements and one disturbance window remain unresolved.
- AMENDMENT · EXP 020MongoDB on its home ground: where does it fit Odin?S2's run record: the pause, all 13 discards and the report's hashes.
- AMENDMENT · EXP 021Kill the database mid-session, does anyone notice?The pre-registered single-copy rule fired, so the compared PostgreSQL variants are strict; ten runner fixes are recorded.
OCTOBER 2026
- AMENDMENT · EXP 020MongoDB on its home ground: where does it fit Odin?Both advocates settle nine readings of the S2 rules before the verdict; both veto item 8, so the report checks the disturbance windows itself.
- AMENDMENT · EXP 021Kill the database mid-session, does anyone notice?The runner protocol and the design draft are published.
- AMENDMENT · EXP 020MongoDB on its home ground: where does it fit Odin?S2's runner changes, the rule for which cells count, and the nine discarded cells are published.
- AMENDMENT · EXP 020MongoDB on its home ground: where does it fit Odin?S2 compares MongoDB with one PostgreSQL variant per scale, chosen once.
- AMENDMENT · EXP 020MongoDB on its home ground: where does it fit Odin?Both advocates accept the S2 pairing rule; the decision paragraph is rewritten as they required.
- PRE-REGISTERED · EXP 021Kill the database mid-session, does anyone notice?The question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 020MongoDB on its home ground: where does it fit Odin?The question and how it will be scored were published before any counted run.
- FIELD NOTEWhen a cached yes answers the opposite questionWe changed the rules behind saved AI answers and reworded questions so yes meant the opposite. Versions kept old answers out, but similar wording still caused wrong replies.
- FIELD NOTEMongoDB beyond JSON: agents, memory and modelsWe tested MongoDB for agents and memory with real Voyage embeddings. Both databases found the same nearest results; tools can repeat after a crash and cached answers can outlive a policy change.
- RESULTS · EXP 019Routing by our own rulebook, measured again: three fresh runs of the routed armPre-registered verdict: routing beats always-Opus, within a wide replicate allowance, and does not beat always-Luna.
- RESULTS · EXP 018Does routing by our own rulebook beat one fixed model on real tasks?Eight real coding tasks, three arms, three runs each, scored by the share of hidden test cases fixed (no task was fully solved). Per case fixed, a table copied from our routing rule paid 13.3 cents (61% of cases), always Opus 5.5 paid 24.7 (65%) and always GPT-6 Luna paid 4.3 (52%). The pre-registered verdict is incomplete: one routed cell hit its spend stop.
- RESULTS · EXP 017What does a defect caught in review cost, model by model?Four models on one tool-free review seat caught between 26 and 31 of 60 defects, all within noise, at 0.06 to 8.3 cents a review. gpt-oss-120b, MiniMax M3.1 Flash and Claude Opus 5.5 are on the pre-registered frontier; Claude Sonnet 5.5 is not.
- RESULTS · EXP 016Does the same model cost the same through every broker?Measured on 720 identical calls: two brokers billed exactly list price for Claude Sonnet 5.5 and GPT-4o mini, the third 10% above it on both, a spread of 1.10 against a pre-registered bar of 1.05. No broker served a different model.
- AMENDMENT · EXP 016Does the same model cost the same through every broker?EXP 016's results are published with the three brokers shown as letters, not names. After this record was frozen, the lab recorded each broker's published terms (site/data/provider-terms.json, 2026-10-23). Two of the three brokers have a clause the lab owner has not yet ruled on: one bars use of its trade names without consent, the other bars systematically collecting benchmarks for competitive purposes. The lab owner decided on 2026-10-23 to publish EXP 016's results without saying which broker is which. This amendment changes only how the results are labelled. Nothing measured, counted or decided changes.
- PRE-REGISTERED · EXP 019Routing by our own rulebook, measured again: three fresh runs of the routed armThe question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 018Does routing by our own rulebook beat one fixed model on real tasks?The question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 017What does a defect caught in review cost, model by model?The question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 016Does the same model cost the same through every broker?The question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 015Does Odin's cost estimate match what the brokers bill?The question and how it will be scored were published before any counted run.
- FIELD NOTEEXP 016: Does the same model cost the same through every broker?Almost. Measured on 720 identical calls: two brokers billed exactly list price for Claude Sonnet 5.5 and GPT-4o mini, the third 10% above it on both, a spread of 1.10 against a pre-registered bar of 1.05. No broker served a different model.
- FIELD NOTEEXP 017: What does a defect caught in review cost, model by model?Four models on one tool-free review seat caught between 26 and 31 of 60 defects, all within noise, at 0.06 to 8.3 cents a review. gpt-oss-120b, MiniMax M3.1 Flash and Claude Opus 5.5 are on the pre-registered frontier; Claude Sonnet 5.5 is not.
- FIELD NOTEEXP 019: Routing by our own rulebook, measured againThree fresh, complete runs of the routed arm on the same eight tasks, scored by the share of hidden test cases fixed. Routing paid 14.3 cents per case fixed (57% of cases), against EXP 018's 24.7 for always-Opus (65%) and 4.3 for always-Luna (52%). Pre-registered verdict: routing beats always-Opus, within a wide replicate allowance, and does not beat always-Luna.
- FIELD NOTEEXP 018: Does routing by our own rulebook beat one fixed model on real tasks?Eight real coding tasks, three arms, three runs each, scored by the share of hidden test cases fixed (no task was fully solved). Per case fixed, a table copied from our routing rule paid 13.3 cents (61% of cases), always Opus 5.5 paid 24.7 (65%) and always GPT-6 Luna paid 4.3 (52%). The pre-registered verdict is incomplete: one routed cell hit its spend stop.
- RESULTS · EXP 012Can a reviewer talk itself out of its false alarms?No. Asked to name a concrete failing input for every finding or drop it, MiniMax M3.1 Flash kept 249 of 269 of its own findings. False alarms stayed at 20 and 22 of 30 clean changes, against a pre-registered bar of 2.
- FIELD NOTEEXP 012: Can a reviewer talk itself out of its false alarms?No. Asked to name a concrete failing input for every finding or drop it, MiniMax M3.1 Flash kept 249 of 269 of its own findings. False alarms stayed at 20 and 22 of 30 clean changes, against a pre-registered bar of 2.
- AMENDMENT · EXP 012Can a reviewer talk itself out of its false alarms?The independent review of the frozen commit (FIX-NEEDED, 2026-10-21 20:43Z) found two defects in the frozen runner. (1) The wrapper reads verify-prompt.txt from its own directory, but the record published the prompt one level up, so the published layout could not run. (2) A malformed verify reply such as {"keep":[null]} passed the list check, became an empty keep list and turned the verdict into SHIP with no retry and no verifyFailed mark. That contradicts the record's own failure rule, which says a verify failure leaves the alarm standing, and it would bias the result toward the hypothesis.
- PRE-REGISTERED · EXP 012Can a reviewer talk itself out of its false alarms?The question and how it will be scored were published before any counted run.
- FIELD NOTEA free model on our review benchOn a sealed corpus of 30 real defects and 30 clean changes, MiniMax M3.1 Flash's counts sat in Claude Opus's range at about 1.5 cents a review. No reviewer flags few enough clean changes to vote, and an agent-run reviewer can read its own answer key.
- FIELD NOTECan a free model find real bugs in our private code?On bugs in our private fix history that can be found from the code alone, MiniMax M3.1 Flash's first guess landed within two lines of the fix 63% of the time, against 23% for our current default. Only about 1 in 10 of the small fixes we sampled could be found that way.
- FIELD NOTEWhere would MongoDB earn a place in Odin?99,000 requests and a separate 19,800-request repeat found two advantages over monolithic JSONB, but none over the stronger PostgreSQL alternatives, for these workloads on one machine. All seventeen outcomes and the durability amendment are retained.
- FIELD NOTEDoes flexible workflow data need MongoDB?Both stores passed 20,000 operations, 42 correctness checks and ten evolution suites behind Temporal activities. For this workload, a MongoDB advantage remains unresolved and flexible JSON alone gave no demonstrated reason to switch.
- ERRATUM · EXP 005Jev as a fast gateErratum 01: the DOI link for Newcombe (1998) in the pre-registration does not resolve.
- PRE-REGISTERED · EXP 011Where does a model reviewer earn its keep?The question and how it will be scored were published before any counted run.
- RESULTS · EXP 009Unload leaves no traceP1 refuted: 1 of 400 H1 trials diverged from R0 (95% Wilson upper bound 1.40%). P2 supported: 0 silent-inert and 0 false-inactive kernel configurations of 120. All 5 validity gates passed, so the record is informative.
- AMENDMENT · EXP 008Hand over the cache, not the textAmendment 02: the zero-prefill counter fix and a fresh counted run.
- RESULTS · EXP 007Which of your gate's rules need a model at all?Pre-registered; measured 7 Oct 2026: the premise is refuted.
- AMENDMENT · EXP 007Which of your gate's rules need a model at all?Amendment 01: one re-call for the translator calls the client's spend limit refused.
- PRE-REGISTERED · EXP 009Unload leaves no traceThe question and how it will be scored were published before any counted run.
- PRE-REGISTERED · EXP 007Which of your gate's rules need a model at all?The question and how it will be scored were published before any counted run.
- AMENDMENT · EXP 008Hand over the cache, not the textAmendment 01: the confirmatory analysis script.
- PRE-REGISTERED · EXP 008Hand over the cache, not the textThe question and how it will be scored were published before any counted run.
- RESULTS · EXP 006nina reviews the changePre-registered; measured 01 OCT 2026: bar met, nina in the spotlight. Amended 01 Oct 2026, before any counted run.
- AMENDMENT · EXP 006nina reviews the changeAmendment 01: a digest is not a restricted term, a compound git call counts, and the client's built-in reads are disclosed.
SEPTEMBER 2026
- PRE-REGISTERED · EXP 006nina reviews the changeThe question and how it will be scored were published before any counted run.
- RESULTS · EXP 005Jev as a fast gatePre-registered; measured 29 Sep 2026: the claim is refuted.
- AMENDMENT · EXP 005Jev as a fast gateAmendment 01: nina 0.34.0 and a pre-registered spotlight bar.
- AMENDMENT · EXP 005Jev as a fast gateAmendment 02: the reviewer's file and git tools are fenced to its workspace, in two layers.
- PRE-REGISTERED · EXP 005Jev as a fast gateThe question and how it will be scored were published before any counted run.
- TOOLLaya in your browserRun Laya, an open-weight typed-decision model, in your browser after a model download. Read its probabilities for a typed question. Three hand-written cases checked parity with the reference; they did not measure accuracy or calibration. Inference runs on your device without an account or API key.
- MEASURED · EXP 004System-1 decisions without lock-in: Jev vs open-weight LayaMeasured on one laptop, 48 hand-written rows.
- RECORDED · EXP 001–003Three blueprint recipesEach recipe applies one unchanged architectural rule to a conforming tree and to a deliberately drifted tree; the engine scored every conforming tree GREEN and every drifted tree RED.
- TOOLCI WitnessTurn an environment-sensitive failure into an inspectable comparison: the failing assertion, a controlled reproduction and a correction that keeps the agreed checks.
- TOOLMigration WitnessMake the acceptance decision explicit: compare the old and new component against examples you approve, with intentional changes recorded separately.
- TOOLTest WitnessA passing suite can miss a broken calculation. Inspect whether tests expose specified defects before you rely on their green result.
- FIELD NOTEA passing pipeline was not yet a verified release.What a missing coverage report taught us about making “done” explainable.
- FIELD NOTEWhy open the factory floor?Tools, experiments and a place to show work while it is still becoming something.