Can a small model on your own machine do real agent tasks as well as the cloud models?
Hand-written rules, small local models and cloud models play the same tasks, set up the same way each time. Before each run we write down what we expect; afterwards we count, and give each result its margin of error and its limits.
Results published · pre-run plans private, checksums public.
When the database server a company writes to fails, which keeps the company's writes better: MongoDB or PostgreSQL?
We stop the main database server in the middle of a working session, on purpose, and check which writes it had confirmed are still there afterwards. Each side's advocate wrote down what they expect before anything ran.
Measured on one host; strict-ack and overlap limits remain.
What does an agent do when a planted ticket tells it to wipe the code repository, and what stops it?
A staged attack on a practice repository filled with made-up data. The same agent gets the same ticket twice: once with no controls, and once behind the engine's real guards, which refuse and record each command.
Staged attack, real refusals · record checksum published.
LATEST RUN · github-hosted · commit c169ad9b · record f2c48fc2a28a
Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.
This checks a committed record. A matching hash does not establish a new experiment or independent validation.
Which models catch the bugs in a code change, which raise false alarms on changes that were fine, and what does our own reviewer turn down?
Four models review the same changes, twice each, alongside a reviewer that runs as a tool, recounted from the records the experiments saved in the repository. We add the review verdicts on this site's own pull requests.
Measured · each count recounted from the repository records.
LATEST RUN · github-hosted · commit 88779efe · record 04fa0d5ba8de
Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.
This checks a committed record. A matching hash does not establish a new experiment or independent validation.
When a web address comes from data, can it run a script or open a lookalike site in the visitor's browser?
A real bug in our main code, fixed by one rule. Each booby-trapped address and each link-opening line from the rule's test goes through the rule, and the test is run.
A real fix, made-up examples · record checksum published.
Given failing builds it has never seen, does the CI doctor, the agent space's build fixer, make a fix a reviewer would call real?
A set of failing builds kept apart and locked before the doctor ran. Each answer is judged by two separate AI reviewers, and we recount their verdicts saved in the repository.
A negative result · record published with its checksum.
LATEST RUN · org-hosted · commit c169ad9b · record 9e1467d6b2c8
Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.
This checks a committed record. A matching hash does not establish a new experiment or independent validation.
When we show real software to a room, what stops a private name, figure or key from reaching a screen or a shipped file?
A made-up company on stage, a sealed build as the only way out, and a leak scan that fails the build. Its tests run, and a fresh scan of the sealed build.
Fiction on stage, a real scan · record cited by hash.
Asked about a change it cannot run, does a local model claim a failing test suite passes, or name files that do not exist, and does a harness change that?
A small made-up route planner and one change to it. Each local model answers the same question bare and with a harness that ran three tools first, and code scores every reply.
Measured on a laptop · record and checksum published.
Before a public link to the agent space is shared, how many anonymous guests can one space take before it falls behind?
The capacity sheet's estimate, made on another machine, beside a fresh run of the same load tool on a Mac, on an isolated instance. The two are boxed apart.
Estimate and measurement apart · record cited by hash.
When we fix our agents' instructions and tools, do their answers get better on work the fixes were not made on?
The same tasks, answered before and after the fixes. A separate AI reviewer judges each answer without being told which side wrote it. Two of the three sets of tasks were kept away from the fixing.
Judged by an AI, not a person · record checksum published.
LATEST RUN · org-hosted · commit c169ad9b · record f419d9215e2b
Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.
This checks a committed record. A matching hash does not establish a new experiment or independent validation.
Can a jury of AI models, scoring pictures blind, tell how good a room looks, and how much does its score move when nothing changed?
Two blind rounds with the same scoring sheet. A test pair, one good picture and one ruined, takes a juror off the jury when it cannot tell them apart. Rooms that were not reworked show how much the scores wobble on their own. Numbers only, no pictures.
Judged by AI models, not people · record checksum published.
Each scenario is judged on its own; there is no single winner. A verdict appears only once an experiment has committed its result to the lab’s own fit map, and it links to that experiment’s results as published on this site. Until then the scenario says Pending. The predictions beside it are only predictions, frozen before anything ran.
Lab 04: The company store: records for a company of agents
Design at commit 4be2262d; not yet published here. Its agent-query experiment has a frozen question set: 131 questions in 9 categories, sha256 ec18314a, with gold answers, sha256 395666a6. Private commit e31fdc52, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can. Its pre-registration is not yet frozen.
Scenario
Verdict
Predictions, frozen before the run
company-store the 1,000-agent company's records (design; not yet frozen)
Pending
None frozen yet
Vendored from the lab’s own fit map at commit 14a29c9d (sha256 89870a85). Left out: every path into the private repository, and the note on who posted a publication.
How we decide.
The linked decision records describe pre-registered plans and their amendments. Where two AI model advocates reviewed a comparison, their objections are retained. That review is not independent experimental replication. See every veto, amendment and admitted gap.