Our two earlier MongoDB studies, EXP 013 and EXP 014, were careful, but they tested what PostgreSQL already does well. This study tests what MongoDB is built for. It takes four problems Odin has on record and asks, for each one, which database solves it correctly, with fewer operational components, at equal or better cost, when each gets its best realistic design.
There is no overall winner. Each of the four problems gets its own verdict: MongoDB fits, PostgreSQL fits, both fit, or neither. Before the plan was frozen, one reviewer argued for MongoDB's best design and one for PostgreSQL's, and either could veto a setup that would handicap its side. Both reviews are published below, in full.
Status: pre-registered; S2 is complete, and amendment 05 records how its run went (below). S1, S3 and S4 have not run. This plan was written, hashed and merged to main before any counted run. Counted runs start only after the commit that publishes it. Nothing in it may change afterwards, except by a dated amendment that is published first.
The four problems
They come from Odin's own decision records, listed in EXP 010's design study (section 1). Each one maps onto a native MongoDB mechanism, and none had been measured before this study. S2 has now been measured (amendment 05, below); S1, S3 and S4 have not.
| Scenario | MongoDB arm | PostgreSQL arm | Gate | Primary metric |
|---|---|---|---|---|
| S1. Records that keep their key order (hash-chained events, payload in a nested field) | BSON decoded to dict; RawBSONDocument for a byte check | A json column with expression GIN / B-tree indexes on (payload::jsonb …); variant: a stored generated jsonb column | 100% of chains verify over the bytes the client reads back; nested queries (dot-notation / expression paths only) equal the oracle | Nested-query p95 |
| S2. A change feed that survives a restart | watch() with $match; post-batch resume token persisted; startAfter; the driver's automatic resume | (a) logical decoding with pgoutput, resuming from confirmed_flush_lsn; (b) a commit-order-safe outbox (an xid8 horizon). LISTEN/NOTIFY as a reference only | 0 lost and 0 duplicated effects | End-to-end lag p95, then components |
| S3. Expiry on append-only data | A TTL index, expireAfterSeconds: 0, on expireAt | Range partitions on expires_at with DETACH CONCURRENTLY + DROP, or an owner-role batched DELETE; scheduled by pg_cron or a sidecar | No expired record visible past the grace; the application role never bypasses the append-only rule | Components |
| S4. Aggregation over values inside documents | A pipeline with $match / $sort first, $group, $facet or separate concurrent pipelines, allowDiskUse explicit | SQL over JSONB with GROUP BY / GROUPING SETS / FILTER, LATERAL jsonb_to_recordset, expression or generated columns | Results equal the oracle | p95 at 10× |
- S1: one serializer rule for both arms (keys in stored order).
- S2 faults: the consumer is killed at seeded points, the database restarts once per cell, and the consumer is down for a frozen 10 minutes. Either side may persist its token or cursor together with the effect in one transaction. The oplog and the slot get the same disk budget (
oplogMinRetentionHours/ size againstmax_slot_wal_keep_size), and lost history is classified the same way in both arms.maxAwaitTimeMS, batch size and the poll interval are frozen and recorded. - S3 "append-only": the application role cannot update or delete. It holds
INSERT, SELECTonly (PostgreSQL) or a custom role withfind+insert(MongoDB), and an application update must fail (permission denied / code 13). A read filter (expireAt > now/ row-level security) is allowed on both arms; the no-extension variant always runs. Two cells: (a) default cadences, 60 s on both, grace 150 s at 1× and 240 s at 10×; (b) tuned cadences, 5 s (ttlMonitorSleepSecs = 5, a 5 s sweep), grace 30 s at 1× and 60 s at 10×. Physical and visible expiry lag p50 / p95 and write p95 during expiry are reported. - S4: the baseline is Odin's current pattern, fetching the values and aggregating in the application. Matching index allowances on both arms. Application peak memory, bytes over the wire and 1× are reported.
How a verdict is decided
mongodb-fits/postgres-fits: the store passes the gate, and either beats the other store's best variant on the primary latency metric by at least 20% on the median, in at least 4 of 5 paired repetitions, with the same direction in the reproduction; or, for S3, and as a tie-break for S2, has at least 2 fewer components.both-fit: both pass, and neither condition holds.neither: neither passes.invalid: a cell is incomplete, or source or durability drifted.- Repetitions: 5 paired repetitions per cell, with the arm order alternating, plus a separate-seed reproduction of every cell.
- Publication: every cell, every variant, every outcome.
Counting moving parts
A frozen checklist, and both counts are published. One component each for: every process or container beyond the database server; every extension not in core; every server setting changed off its default and needed for correctness; every persistent non-data object (a slot, publication, trigger, policy, partition scheme counted once per scheme, TTL index, custom role or keyFile); every scheduled job; and every safety alarm an operator must watch (an oplog window, slot retention). MongoDB pays for its replica set, keyFile, oplog sizing and oplog alarm. PostgreSQL pays for its slot, publication, wal_level, jobs and partitions.
An even setup
- MongoDB Community 8.0.x as a one-member replica set (
replSet,rs.initiate), auth on with a keyFile, writesw: majority, j: true, PyMongo 4.x. - PostgreSQL 16.x, the official image plus pinned derived images where an extension is used (
wal2json,pg_cron,pg_partman),synchronous_commit = on, psycopg 3.x. Each scenario also runs a no-extension variant. - Equal resources: 4 GB / 2 CPU per database container. MongoDB: WiredTiger cache 1.5 GB. PostgreSQL:
shared_buffers1 GB,effective_cache_size2 GB,work_mem64 MB,max_parallel_workers_per_gather2,--shm-size1 GB. Default compression on both (snappy, pglz), recorded; driver wire compression off. - Host: Windows + WSL2 Docker Desktop, data on named volumes on the WSL2 ext4 disk, never on NTFS bind mounts.
pg_test_fsyncandfio --fsync=1run inside each container and are recorded, with mongod's filesystem warning. The client runs in a container on the same Docker network, with equal pool sizes. A macOS machine runs the same frozen source as a second-OS replication. - Before each cell: PostgreSQL runs
VACUUM ANALYZEandCHECKPOINT; the cache is warm on both arms, after one discarded pass; checkpoint and autovacuum events are logged. Image tags and digests are pinned before the first counted run and recorded in the run manifest.
Predictions, written down so the result can contradict them
| Who | S1 | S2 | S3 | S4 |
|---|---|---|---|---|
| The design author (corrected after the reviews) | both-fit (the draft's "json loses indexing" was wrong) | mongodb-fits, low-to-moderate confidence | mongodb-fits on components | both-fit |
| The MongoDB advocate | both-fit | mongodb-fits (moderate) | mongodb-fits | both-fit, leaning postgres-fits at 10× |
| The PostgreSQL advocate | both-fit, leaning postgres-fits | both-fit | mongodb-fits on components is plausible | both-fit |
The two reviews
Each reviewer was an independent model reviewer (Claude, fresh context), briefed to make sure its database got its best realistic design. Every veto was accepted. The PostgreSQL review caught a factual error in the draft ("json loses indexing" is false), an outbox that would drop late commits, and an append-only rule that favoured MongoDB. The MongoDB review pinned the decode type and payload placement for S1, tied the S3 grace to the configured TTL interval, and equalised retention for S2. Both asked for a primary metric, a margin and a definition of a component.
- The MongoDB advocate's review
- The PostgreSQL advocate's review
- The freeze notes: each veto, its resolution, and where the reviewers differed (the margin, failover, search)
What this does not establish
Named as the next study, not judged here: failover with a 3-member replica set against Patroni (S5), sharding, multi-region, and search / vector search. Failover is MongoDB's home ground, so it is named explicitly; on one shared Docker host a fair PostgreSQL arm for it (streaming replication with Patroni and etcd) is a memory and stability risk to both arms. The run does not imply anything about any of them. EXP 013 and EXP 014 stand as published; this experiment does not change or replace them.
Amendment 01: changes to S2's runner, published late
This plan says a change is a dated amendment, published first. For S2, the change-feed scenario, that did not happen. Its runner was changed four times, and given a rule for which counted cells count, after this plan was published and after S2's first two counted cells ran. Those two cells ran on the old code and were discarded; every kept cell ran under the changes. Amendment 01 (also on this site) publishes all of it before S2's report, and we own the gap.
- 01: the pgoutput cursor became the commit LSN, fixing a runner bug that lost one event.
- 02: every arm applies changes in batches of up to 100 per transaction. Effect and cursor still share one transaction, but this moves the measured lag a long way, so it is published as a change to the measure.
- 03: the producer builds its payloads before its clock starts, a harness timing fix that restores the planned rate.
- 04: the MongoDB consumer applies once its local buffer is empty. It is for one arm only and in that arm's favour, so it is published as a change to the measure.
- Which cells count: a counted cell that overlapped heavy work on the host (a browser run, filming, another benchmark) is discarded and re-run. Nine cells were discarded under that rule; five of them had first been kept. All nine are listed by id in the amendment.
The S2 runner's protocol is published too. The amendment also records that the protocol pairs MongoDB with the better PostgreSQL variant per repetition, while this plan says "the other store's best variant"; amendment 01 left that open. S1, S3 and S4 are not affected.
Amendment 02: one PostgreSQL variant per scale
Amendment 02 (also on this site) settles the pairing before S2's verdict is computed. For each scale, PostgreSQL is represented by one variant, chosen once: among the variants that pass the gate in every counted cell at that scale on both seeds, the one with the lowest median of its seed-1 lag p95s. That variant is compared with MongoDB in every pair. Picking the better variant in each repetition would be a best-of-two that favours PostgreSQL; it is still published, labelled as a secondary analysis, and never decides the verdict.
In the amendment's own words: the decision was made on 25 October 2026, before S2's verdict was computed, by the R&D orchestrator, not by either advocate. Interim seed-1 results had already been seen, so it is not blind. We chose the rule the advocates of both sides had already agreed was fair for EXP 021, rather than one picked with the data in view. Both of this experiment's advocates were asked to review that decision, with veto power, before the verdict.
Amendment 03: the advocates' answer
Both advocates reviewed the rule and the paragraph before S2's verdict was computed, without opening S2's result files. Amendment 03 (also on this site) records their answer and both reviews (MongoDB advocate, PostgreSQL advocate).
- The rule: both accept. Both note one small remaining lean toward PostgreSQL, because the variant is chosen on seed 1 and compared on seed 1, and both accept it. Seed 2's lag values never enter the choice, though its gate results do: a candidate must pass every counted cell on both seeds.
- The paragraph: both vetoed it, for completeness. It now says which rule it replaces, what had been seen (the interim order of the seed-1 lag medians: pgoutput lowest, then MongoDB, then the outbox), and that the choice to adopt the rule was made with that in view.
- Two clarifications: ties go to the lower seed-1 mean, and MongoDB must pass every counted cell at a scale on both seeds, the same standard as PostgreSQL's chosen variant.
- More is published, never deciding: every variant's gate result, each candidate's selection number, each variant on its own against MongoDB, each variant's components, what a seed-2 choice would have picked, and one line per scale on whether the verdict would change.
Amendment 04: nine readings of the S2 rules, settled before the verdict
Writing S2's report code showed nine places where the published rules leave a reading open. The runner's owner did not decide them alone: both advocates reviewed all nine before S2's verdict was computed, with veto power, without opening a counted cell. Amendment 04 (also on this site) records their answer and both reviews (MongoDB advocate, PostgreSQL advocate). It is not the S2 runner protocol's own "Amendment 04", which is a different change.
- Eight accepted by both. "Same direction" is the sign of the median of the five seed-2 paired differences, and a zero counts for neither store. "Beats by 20 %" means the winner's lag p95 is at most 0.8 times the other's, the stricter reading. Items 4 to 7 and 9, bookkeeping, are accepted as written.
- Components, checked against the code by both: MongoDB 4, pgoutput 5, outbox 0 decide the tie-break. S2's MongoDB consumer has no fallback if its oplog window runs out, so oplog sizing and its alarm count. The protocol's literal counts (5, 6 and 4) are published next to them and never decide, as the MongoDB advocate required. The PostgreSQL advocate notes, against its own side, that pgoutput could arguably count 4, and does not ask for it.
- Item 8: both vetoed it. The report may not trust which cells were moved aside after a disturbance; that had already failed twice. Instead, the runner's owner says, it reads the disturbance windows from a machine-readable file (12 windows, 7 heavy and 5 light) and refuses to report while any counted cell's run overlaps a heavy one; cells that overlap a light window are listed by name. The code is private, so the amendment says what was checked and what was not. This is the stricter of the two advocates' alternatives, the MongoDB advocate's.
- Cited by hash, not published: the report code and the windows file are in a private repository, at commit
125b4c7d; their sha256 are in the amendment, with what it admits as limits.
S2's verdict is computed only after this amendment is published and all 60 counted cells exist.
Amendment 05: S2's run record
S2 is complete: all 60 counted cells exist, and the report was computed under the rules above. Amendment 05 (also on this site) records how the run got there. The plan was held for 13 days and 6 hours by a stale pause flag, then resumed and re-measured exactly the missing cells. A helper script stopped the stack once, and no counted cell was affected. All 13 discarded cells are listed with their causes; two of them were refused by the report's own overlap check. The record gives the sha256 of the report, the files it read and every counted cell. It quotes the report's verdict without recomputing it: PostgreSQL fits at both scales, with pgoutput as the chosen variant, and against the outbox variant alone MongoDB fits (secondary, never deciding). The report and its cells are private and cited by hash. The amendment changes no rule.
Provenance
The pre-registration (also published on this site, byte for byte) is a faithful port of a frozen design: two-advocate review, source commit bb364127 in a private repository; PREREG sha256 83f35a8c…. It carries the frozen text in full. Machine names and internal ids were replaced with neutral wording in four places, each listed in the record; nothing else in the text differs. The two reviews and the freeze notes are published byte for byte.
Pre-registered experiment; S2 complete, amended five times. Two databases, one host plus a second-OS replication, synthetic data sized from one recorded store, no failover or scale-out.