11 November 2026 update: The counted result and amendment 03 are now available, with unresolved strict-ack and disturbance limits. The plan and earlier amendments below are historical records.

EXP 020 left one question for later: what happens when the database's main server dies? This is that study, Lab 03 (failover). While the agent space writes its proof records and a feed turns them into marks on the lens, the primary fails. We ask which database keeps every write it confirmed and the feed intact, with the shortest write outage the client can see and the fewest kinds of moving parts, when each gets its best realistic high-availability design.

There is no overall winner. Each fault gets its own verdict, and the moving parts get theirs. The two contenders are a MongoDB 8.0 replica set of three data members, and PostgreSQL 16 with Patroni 4.x and etcd 3.5.x on three nodes. The plan went through two rounds of review, each with one reviewer arguing for MongoDB's best design and one for PostgreSQL's. All 22 of their vetoes were accepted. All four reviews are published below.

Status at the 25 October pre-registration: not yet run. This plan was written, hashed and merged to main before any counted run. Counted runs start only after the commit that publishes it. Nothing in it may change afterwards, except by a dated amendment that is published first.

The five faults

Each fault strikes at a seeded instant, and each waits for full recovery before the next.

F1b is a labelled line: it is published and never decides a verdict.
FaultWhat happens
F1, node deathdocker kill -s KILL of the primary's node container
F1b, process death (labelled)Only mongod / the postmaster is killed
F2a, full isolationThe primary is disconnected from both networks, app (the client) and ha (peers and replication)
F2b, peer-only partitionThe primary is disconnected from ha only: the client still reaches it
F3, planned switchoverreplSetStepDown against patronictl switchover --candidate <sync standby> --force

What is measured

An even setup

How a verdict is decided

Predictions, written down so the result can contradict them

The v1 lines are kept as frozen; v1.1 adds both advocates' updates. The design author's are unchanged in v1.1.
WhoF1 / F2F3FeedMoving parts
The design authormongodb-fits at the default and the tuned cells; the gate passes on both armsboth-fitboth-fit (outbox)mongodb-fits
The MongoDB advocate, v1F1 mongodb-fits at defaults (about 10–13 s against 30–40 s); tuned both-fit, leaning MongoDB, low confidence; F2 as F1both-fit—mongodb-fits, high (about 5 against 14–15)
The MongoDB advocate, v1.1F1 default mongodb-fits, high (about 11–16 s against 30–42 s); F1 tuned mongodb-fits, moderate-high; F2a / F2b as F1both-fit, leaning MongoDB, low—mongodb-fits, high
The PostgreSQL advocate, v1mongodb-fits, moderate-high (defaults about 11–14 s against 22–33 s; tuned about 4–7 s against 13–20 s)both-fit, leaning MongoDBboth-fit (outbox)mongodb-fits, high
The PostgreSQL advocate, v1.1F1 / F2a / F2b mongodb-fits, high (tuned about 5.5–7 s against 19–23 s)mongodb-fits about 55% / both-fit about 45%, lowboth-fitmongodb-fits, high

The four reviews

Each reviewer was an independent model reviewer (Claude, fresh context), briefed to give its store its best realistic HA design and to veto strawmen. The first round reviewed the design draft and filed 16 vetoes: among them, the topology (three data members, no arbiter), matched client timeouts and one shared retry loop, a node-level kill on both arms, a second network so a partition can show split-brain, quorum sync, and rejoin hygiene. The second round reviewed 12 open questions the runner surfaced and filed 6 more: among them, PostgreSQL's tuned timing (20 / 4 / 8 rather than the validator floor), F3's compared number, one compared PostgreSQL variant instead of a best-of-two per repetition, and false failovers counting against the arm that has them.

What this does not establish

Named next, not judged here: multi-host networks, sharding, Kubernetes operators, and HAProxy or pg_auto_failover as primary arms. Everything runs on one Docker host, so it says nothing about failover across machines.

Amendment 01: how it will be run

A pre-registration should show how it will be run, not only what it predicts. Amendment 01 (also on this site) publishes the runner protocol: the gates before any cell, the phases of one cell, the faults, the oracle, the plan's size and order, and every runner decision. It also publishes the design draft. The draft is not the plan, but its own predictions and its plan for this page are written nowhere else, and the round-1 reviews answer it. Both files carry the same kind of neutral substitutions as the rest, each listed with the file's source and published sha256. The amendment adds documents only: no rule, measure or prediction changes.

Amendment 02: the strict switch and the runner fixes

The smoke run, which is never counted, made one pre-registered rule fire. In the non-strict quorum cell for a killed process, the new primary briefly named no synchronous standby, and writes were acknowledged with one copy. The frozen rule for that case applies: the compared PostgreSQL variants are the synchronous_mode_strict ones, and the non-strict numbers are context only. Amendment 02 (also on this site) records that, and ten runner fixes, each cited by its private commit. Nine are mechanics, recording or analysis code. One changes a definition, how full redundancy is told in quorum and classic mode; the PostgreSQL advocate's review of it is published verbatim, and its tightening, which favours MongoDB, is adopted. It was published after the counted runs started, and it says so; it changes no rule they run under.

Provenance

The pre-registration (also published on this site, byte for byte) carries the frozen text in full: v1.1, source commit 5d056765 in a private repository; PREREG sha256 f3630380…. Machine names, the private repository's name and one internal file path were replaced with neutral wording in five places, each listed in the record; nothing else in the text differs. The first-round reviews and freeze notes are published byte for byte; the second-round reviews and freeze notes each differ from their private copies by one listed substitution, the private repository's name. The runner protocol and the design draft were cited by sha256 at first and are published by amendment 01. An earlier v1 (8e55a4bb) was superseded the same day before any cell ran; it is history only, never counted.

The private files call this lab "Lab 02". On this site it is Lab 03 (failover), because our Lab 02 is the brain bench.

Historical pre-registration; the result is linked above. Two databases, three nodes each, on one Docker host; the runner's protocol is published, but its code is private, so the hashes, the plan, the reviews and the protocol can be checked here, but the runner itself cannot be rerun.