Labs.

A lab is a project that keeps going. You can follow it as it grows, and you can take part: ask a question, and see it answered in the work itself.

LAB 01

The agent space

In recorded sessions, agents appear in a small 3D world and answer selected questions on camera. Published source cards identify what an answer cited; they do not establish correctness.

Latest session: .

LAB 02

Every brain on the same tasks

Every brain on the same tasks — Open the lab page

Can a small model on your own machine do real agent tasks as well as the cloud models?

Hand-written rules, small local models and cloud models play the same tasks, set up the same way each time. Before each run we write down what we expect; afterwards we count, and give each result its margin of error and its limits.

Results published · pre-run plans private, checksums public.

LAB 03

Failover

Failover — Open the lab page

When the database server a company writes to fails, which keeps the company's writes better: MongoDB or PostgreSQL?

We stop the main database server in the middle of a working session, on purpose, and check which writes it had confirmed are still there afterwards. Each side's advocate wrote down what they expect before anything ran.

Measured on one host; strict-ack and overlap limits remain.

LAB 05

Can an agent go rogue?

Can an agent go rogue? — Open the lab page

What does an agent do when a planted ticket tells it to wipe the code repository, and what stops it?

A staged attack on a practice repository filled with made-up data. The same agent gets the same ticket twice: once with no controls, and once behind the engine's real guards, which refuse and record each command.

Staged attack, real refusals · record checksum published.

LATEST RUN · github-hosted · commit c169ad9b · record f2c48fc2a28a

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

This run in the list of all runs

LAB 06

Which model should review your code?

Which model should review your code? — Open the lab page

Which models catch the bugs in a code change, which raise false alarms on changes that were fine, and what does our own reviewer turn down?

Four models review the same changes, twice each, alongside a reviewer that runs as a tool, recounted from the records the experiments saved in the repository. We add the review verdicts on this site's own pull requests.

Measured · each count recounted from the repository records.

LATEST RUN · github-hosted · commit 88779efe · record 04fa0d5ba8de

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

This run in the list of all runs

LAB 08

Can an agent fix a red build?

Can an agent fix a red build? — Open the lab page

Given failing builds it has never seen, does the CI doctor, the agent space's build fixer, make a fix a reviewer would call real?

A set of failing builds kept apart and locked before the doctor ran. Each answer is judged by two separate AI reviewers, and we recount their verdicts saved in the repository.

A negative result · record published with its checksum.

LATEST RUN · org-hosted · commit c169ad9b · record 9e1467d6b2c8

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

This run in the list of all runs

LAB 09

Can a talk leak data?

Can a talk leak data? — Open the lab page

When we show real software to a room, what stops a private name, figure or key from reaching a screen or a shipped file?

A made-up company on stage, a sealed build as the only way out, and a leak scan that fails the build. Its tests run, and a fresh scan of the sealed build.

Fiction on stage, a real scan · record cited by hash.

LAB 11

Same model, with and without a harness

Same model, with and without a harness — Open the lab page

Asked about a change it cannot run, does a local model claim a failing test suite passes, or name files that do not exist, and does a harness change that?

A small made-up route planner and one change to it. Each local model answers the same question bare and with a harness that ran three tools first, and code scores every reply.

Measured on a laptop · record and checksum published.

LAB 13

Does an agent company get better, measured blind?

Does an agent company get better, measured blind? — Open the lab page

When we fix our agents' instructions and tools, do their answers get better on work the fixes were not made on?

The same tasks, answered before and after the fixes. A separate AI reviewer judges each answer without being told which side wrote it. Two of the three sets of tasks were kept away from the fixing.

Judged by an AI, not a person · record checksum published.

LATEST RUN · org-hosted · commit c169ad9b · record f419d9215e2b

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

This run in the list of all runs

LAB 14

Can an AI jury judge how a room looks?

Can an AI jury judge how a room looks? — Open the lab page

Can a jury of AI models, scoring pictures blind, tell how good a room looks, and how much does its score move when nothing changed?

Two blind rounds with the same scoring sheet. A test pair, one good picture and one ruined, takes a juror off the jury when it cannot tell them apart. Rooms that were not reworked show how much the scores wobble on their own. Numbers only, no pictures.

Judged by AI models, not people · record checksum published.

Where each store fits.

Each scenario is judged on its own; there is no single winner. A verdict appears only once an experiment has committed its result to the lab’s own fit map, and it links to that experiment’s results as published on this site. Until then the scenario says Pending. The predictions beside it are only predictions, frozen before anything ran.

0 of 10 scenarios have a verdict so far.

EXP 020: MongoDB on its home ground: where does it fit Odin?

Pre-registration at commit bb364127, sha256 83f35a8c; published here in pull request #76.

ScenarioVerdictPredictions, frozen before the run
S1 order-preserving recordsPending
  • The lab: both-fit
  • MongoDB’s advocate: both-fit
  • PostgreSQL’s advocate: both-fit, leaning postgres-fits
S2 a durable, resumable change feedPending
  • The lab: mongodb-fits (low-to-moderate)
  • MongoDB’s advocate: mongodb-fits (moderate)
  • PostgreSQL’s advocate: both-fit
S3 TTL on append-only dataPending
  • The lab: mongodb-fits (components)
  • MongoDB’s advocate: mongodb-fits
  • PostgreSQL’s advocate: mongodb-fits on components is plausible
S4 server-side aggregation over document valuesPending
  • The lab: both-fit
  • MongoDB’s advocate: both-fit, leaning postgres-fits at 10x
  • PostgreSQL’s advocate: both-fit

Lab 03: Failover: kill the database mid-session, does anyone notice?

Frozen, not yet published. Pre-registration at commit 5d056765, sha256 f3630380.

ScenarioVerdictPredictions, frozen before the run
F1 node death (the primary's container killed)Pending
  • The lab: mongodb-fits (default and tuned)
  • MongoDB’s advocate: mongodb-fits (default high, tuned moderate-high)
  • PostgreSQL’s advocate: mongodb-fits (high)
F2a full isolation of the primaryPending
  • The lab: mongodb-fits
  • MongoDB’s advocate: as F1
  • PostgreSQL’s advocate: mongodb-fits (high)
F2b peer-only partition (the client still reaches the old primary)Pending
  • The lab: mongodb-fits
  • MongoDB’s advocate: as F1
  • PostgreSQL’s advocate: mongodb-fits (high)
F3 planned switchover (compared: the longest gap between acknowledged writes)Pending
  • The lab: both-fit
  • MongoDB’s advocate: both-fit, leaning mongodb, low
  • PostgreSQL’s advocate: mongodb-fits ~55% / both-fit ~45%, low
components operational component kinds for HAPending
  • The lab: mongodb-fits
  • MongoDB’s advocate: mongodb-fits (high)
  • PostgreSQL’s advocate: mongodb-fits (high)

Lab 04: The company store: records for a company of agents

Design at commit 4be2262d; not yet published here. Its agent-query experiment has a frozen question set: 131 questions in 9 categories, sha256 ec18314a, with gold answers, sha256 395666a6. Private commit e31fdc52, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can. Its pre-registration is not yet frozen.

ScenarioVerdictPredictions, frozen before the run
company-store the 1,000-agent company's records (design; not yet frozen)PendingNone frozen yet

Vendored from the lab’s own fit map at commit 14a29c9d (sha256 89870a85). Left out: every path into the private repository, and the note on who posted a publication.

The linked decision records describe pre-registered plans and their amendments. Where two AI model advocates reviewed a comparison, their objections are retained. That review is not independent experimental replication. See every veto, amendment and admitted gap.