11 November 2026 update: The counted result and amendment 03 are now available, with unresolved strict-ack and disturbance limits. The plan and earlier amendments below are historical records.
EXP 020 left one question for later: what happens when the database's main server dies? This is that study, Lab 03 (failover). While the agent space writes its proof records and a feed turns them into marks on the lens, the primary fails. We ask which database keeps every write it confirmed and the feed intact, with the shortest write outage the client can see and the fewest kinds of moving parts, when each gets its best realistic high-availability design.
There is no overall winner. Each fault gets its own verdict, and the moving parts get theirs. The two contenders are a MongoDB 8.0 replica set of three data members, and PostgreSQL 16 with Patroni 4.x and etcd 3.5.x on three nodes. The plan went through two rounds of review, each with one reviewer arguing for MongoDB's best design and one for PostgreSQL's. All 22 of their vetoes were accepted. All four reviews are published below.
Status at the 25 October pre-registration: not yet run. This plan was written, hashed and merged to main before any counted run. Counted runs start only after the commit that publishes it. Nothing in it may change afterwards, except by a dated amendment that is published first.
The five faults
Each fault strikes at a seeded instant, and each waits for full recovery before the next.
| Fault | What happens |
|---|---|
| F1, node death | docker kill -s KILL of the primary's node container |
| F1b, process death (labelled) | Only mongod / the postmaster is killed |
| F2a, full isolation | The primary is disconnected from both networks, app (the client) and ha (peers and replication) |
| F2b, peer-only partition | The primary is disconnected from ha only: the client still reaches it |
| F3, planned switchover | replSetStepDown against patronictl switchover --candidate <sync standby> --force |
What is measured
- The gate: 0 acknowledged writes lost (every acknowledged id is read back from the final primary), 0 duplicated or missing feed effects, and ambiguous commits resolved by the oracle. Writes accepted but never acknowledged in F2a / F2b are reported as their own number, never as a loss.
- The primary metric: the client-visible write outage, from the fault to the first acknowledged write after it, per writer thread; the median over the 8 threads is used, and the max is reported. For F3, a planned switchover with no fault instant that stops writes, the compared number is the median over the threads of the longest gap between acknowledged writes ending in (the fault, the end of the hold]; the literal number is published too.
- Acknowledged means the database confirmed durability: for MongoDB, no write-concern error; for PostgreSQL, the transaction waited for the synchronous standby.
- Also reported: detection, promotion and rediscovery where observable; the longest single write; the time to a resumed feed; the time to full redundancy; false failovers in a 10-minute steady phase; etcd fsync latency.
- Moving parts: EXP 020's component checklist, counted in kinds for the verdict, with processes published too. Data members cancel on both sides, and defaults are not components.
An even setup
- MongoDB: 3 data members, priority 1 / votes 1, no arbiter; keyFile + auth; 2 GB and 2 CPU per member; writes
{w: "majority", j: true}; PyMongo 4.10 with retryable writes and reads,timeoutMS5000. Default timing:electionTimeoutMillis10000 /heartbeatIntervalMillis2000; tuned: 5000 / 1000. - PostgreSQL: 3 nodes, each running PG16, Patroni (
etcd3:API) and an etcd member; 2 GB and 2 CPU per node;synchronous_mode: quorum(ANY 1), with classic sync mode as a variant; data checksums +use_pg_rewind; libpq multi-host withtarget_session_attrs=read-write; writes idempotent by proof id. Default timing: ttl 30 / loop_wait 10 / retry_timeout 10; tuned: 20 / 4 / 8, the most aggressive documented, supported setting. Patroni's validator floor, 20 / 1 / 3, runs as a labelled line. - The workload: proof records of about 2 KB at 20 a second from 8 writer threads, with ids chosen by the client; one shared retry loop for both arms (the same id, 0.5 s backoff, a 90 s budget).
- The feed: effect and cursor in one transaction on both arms; a MongoDB change stream against the PostgreSQL commit-order-safe outbox, with pgoutput on Patroni's permanent logical slots as a second variant. One variant is compared per scenario.
- The host: Windows + WSL2 Docker, two networks, named volumes, run only when no other benchmark measures.
How a verdict is decided
- A fault verdict uses the gate and the outage only:
mongodb-fitsorpostgres-fitswhen the gate passes and the median outage is at least 20% shorter than the other arm's compared variant in at least 4 of 5 paired repetitions, with the same direction in the reproduction on a second seed. - The compared PostgreSQL variant is one per (timing, fault): among the variants that pass the gate in every cell on both seeds, the one with the lowest median of its seed-1 repetition medians.
- The moving-parts verdict is its own line: at least 2 fewer component kinds.
- Only one arm passes the gate: that arm fits, labelled
gate-only. Otherwiseboth-fit,neitherorinvalid. A false failover in an arm's steady cell at a timing means that arm cannot get a-fitsverdict at that timing. - Repetitions: 5 per (arm, variant, cell, fault) on seed 1, plus a reproduction of every cell on seed 2. Every cell, variant and outcome is published.
Predictions, written down so the result can contradict them
| Who | F1 / F2 | F3 | Feed | Moving parts |
|---|---|---|---|---|
| The design author | mongodb-fits at the default and the tuned cells; the gate passes on both arms | both-fit | both-fit (outbox) | mongodb-fits |
| The MongoDB advocate, v1 | F1 mongodb-fits at defaults (about 10–13 s against 30–40 s); tuned both-fit, leaning MongoDB, low confidence; F2 as F1 | both-fit | — | mongodb-fits, high (about 5 against 14–15) |
| The MongoDB advocate, v1.1 | F1 default mongodb-fits, high (about 11–16 s against 30–42 s); F1 tuned mongodb-fits, moderate-high; F2a / F2b as F1 | both-fit, leaning MongoDB, low | — | mongodb-fits, high |
| The PostgreSQL advocate, v1 | mongodb-fits, moderate-high (defaults about 11–14 s against 22–33 s; tuned about 4–7 s against 13–20 s) | both-fit, leaning MongoDB | both-fit (outbox) | mongodb-fits, high |
| The PostgreSQL advocate, v1.1 | F1 / F2a / F2b mongodb-fits, high (tuned about 5.5–7 s against 19–23 s) | mongodb-fits about 55% / both-fit about 45%, low | both-fit | mongodb-fits, high |
The four reviews
Each reviewer was an independent model reviewer (Claude, fresh context), briefed to give its store its best realistic HA design and to veto strawmen. The first round reviewed the design draft and filed 16 vetoes: among them, the topology (three data members, no arbiter), matched client timeouts and one shared retry loop, a node-level kill on both arms, a second network so a partition can show split-brain, quorum sync, and rejoin hygiene. The second round reviewed 12 open questions the runner surfaced and filed 6 more: among them, PostgreSQL's tuned timing (20 / 4 / 8 rather than the validator floor), F3's compared number, one compared PostgreSQL variant instead of a best-of-two per repetition, and false failovers counting against the arm that has them.
- Round 1: the MongoDB advocate, the PostgreSQL advocate, and the freeze notes
- Round 2 (v1.1): the MongoDB advocate, the PostgreSQL advocate, and the freeze notes
What this does not establish
Named next, not judged here: multi-host networks, sharding, Kubernetes operators, and HAProxy or pg_auto_failover as primary arms. Everything runs on one Docker host, so it says nothing about failover across machines.
Amendment 01: how it will be run
A pre-registration should show how it will be run, not only what it predicts. Amendment 01 (also on this site) publishes the runner protocol: the gates before any cell, the phases of one cell, the faults, the oracle, the plan's size and order, and every runner decision. It also publishes the design draft. The draft is not the plan, but its own predictions and its plan for this page are written nowhere else, and the round-1 reviews answer it. Both files carry the same kind of neutral substitutions as the rest, each listed with the file's source and published sha256. The amendment adds documents only: no rule, measure or prediction changes.
Amendment 02: the strict switch and the runner fixes
The smoke run, which is never counted, made one pre-registered rule fire. In the non-strict quorum cell for a killed process, the new primary briefly named no synchronous standby, and writes were acknowledged with one copy. The frozen rule for that case applies: the compared PostgreSQL variants are the synchronous_mode_strict ones, and the non-strict numbers are context only. Amendment 02 (also on this site) records that, and ten runner fixes, each cited by its private commit. Nine are mechanics, recording or analysis code. One changes a definition, how full redundancy is told in quorum and classic mode; the PostgreSQL advocate's review of it is published verbatim, and its tightening, which favours MongoDB, is adopted. It was published after the counted runs started, and it says so; it changes no rule they run under.
Provenance
The pre-registration (also published on this site, byte for byte) carries the frozen text in full: v1.1, source commit 5d056765 in a private repository; PREREG sha256 f3630380…. Machine names, the private repository's name and one internal file path were replaced with neutral wording in five places, each listed in the record; nothing else in the text differs. The first-round reviews and freeze notes are published byte for byte; the second-round reviews and freeze notes each differ from their private copies by one listed substitution, the private repository's name. The runner protocol and the design draft were cited by sha256 at first and are published by amendment 01. An earlier v1 (8e55a4bb) was superseded the same day before any cell ran; it is history only, never counted.
The private files call this lab "Lab 02". On this site it is Lab 03 (failover), because our Lab 02 is the brain bench.
Historical pre-registration; the result is linked above. Two databases, three nodes each, on one Docker host; the runner's protocol is published, but its code is private, so the hashes, the plan, the reviews and the protocol can be checked here, but the runner itself cannot be rerun.