How we run
experiments.

A public number should have an inspectable method, date and source boundary. Some underlying corpora and operational receipts are private, so some results lack enough public material for outside reproduction. This page says how: what we write down before a run, how it is frozen, what each experiment may spend, what would prove us wrong, who checks a change before it is published, and how to rerun it yourself. Every claim below points to a file in the repository.

What a public claim must establish.

Publish a useful finding when its evidence supports the exact wording. A plan, merged change, passing build, source card or matching hash is not a verified outcome. Progress may be shared as progress, with what remains unknown stated beside it. Repeated runs and routine commits do not each need a new announcement.

Planned or pre-registered
A question and method are recorded. There is no result yet.
Prototype or fixture
A mechanism was exercised in the named demonstration or authored examples. This does not establish performance on unseen work or customer systems.
Measured result
Name the method, date, sample, environment and result record. State failures, uncertainty and what the measurement cannot establish.
Repeated by Odin
A rerun on another Odin machine can check repeatability. It is not independent replication by an outside team.
Outside replication
Use this description only with an identified outside reproducer, the method and result they checked, and a public report supporting that scope.

A hash identifies bytes; it does not validate the experiment or establish that a claim is true. An answer’s source card records attribution, not correctness. A dated recording is not a live service. Private evidence must be labelled author-reported where a reader cannot inspect or reproduce it.

The same boundary applies to page titles, social previews, feeds and Twitter/X or LinkedIn progress posts: the short version must retain any limitation that changes the meaning. Correct an overstatement where it appeared, preserve the earlier record and link the correction. This standard was clarified on .

1. The question comes first.

Before any counted run, an experiment's pre-registration is written down and published. It states the question, what is measured, how it is scored, the decision rule, what the experiment may spend and its known limits. Each one lives in its own folder, for example experiments/self-verify/preregistration.json (EXP 012) or experiments/jev-gate/preregistration.json (EXP 005).

Where the analysis is code, the code is frozen with the question: EXP 008 pins its analysis script experiments/latent-handoff/analyse.mjs by hash in a dated amendment (experiments/latent-handoff/amendment-02.json), and EXP 005 recomputes its published results from the committed runs with experiments/jev-gate/results.mjs.

Two studies on this site were published without a pre-registration: EXP 013 and EXP 014, the MongoDB studies. They are numbered so they can be cited, and the experiment list marks them “not pre-registered”: their method and data are published, but nothing about them was fixed before they ran.

2. Frozen by sha256.

Each pre-registration has a .sha256 beside it, such as experiments/self-verify/preregistration.sha256, written when the record is pushed. The push time is the freeze time. After that the record does not change.

A change after the freeze is a new, dated file that names its parent by hash: an amendment (experiments/jev-gate/amendment-01.json, experiments/self-verify/amendment-01/amendment.json) or an erratum (experiments/jev-gate/erratum-01.json). The site build refuses to publish a record whose bytes no longer match its hash: see scripts/jev-gate-prereg.mjs and scripts/ledger-publish.mjs. The published copy under site/data/ (served as data/ on this site) is the record, byte for byte.

3. A spend bound on every run.

Every pre-registration bounds its spend (a spend block; EXP 005, the first, calls it spendCap): the most the run may cost and what happens when that bound is reached. In EXP 012, for example, the run stops and the experiment is published as incomplete, with the finished count; nothing is extrapolated.

Experiments that pay for model calls keep a per-call spend ledger next to their results, for example experiments/jev-gate/spend-ledger.jsonl, experiments/nina-changes/spend-ledger.jsonl and experiments/blueprint-floor/spend-ledger.jsonl.

4. Rules that can prove us wrong.

The decision rule is part of the frozen record, so the result cannot choose its own bar. Results are published whatever they show, and a refuted claim stays refuted. So far that includes EXP 005 (the claim is refuted), EXP 007 (the premise is refuted), EXP 009 (its first prediction is refuted) and EXP 012 (the answer is no). Each status sentence on the experiment list comes from the experiment's own note, unedited (scripts/experiment-ledger.mjs).

Before a pre-registration is written we sometimes run a few calibration calls. When we do, the record says so, and states what we expect, so the result can contradict it.

5. Checked before it is published.

Changes to this site arrive as pull requests. On each one, the verify job in .github/workflows/publish.yml reruns the three recorded experiments with the released engine, runs every test, builds the site and checks it (links, anchors, transcript digests and the public export boundary, scripts/check.mjs). The same job runs again on every push to main, before the site is deployed.

Our review policy requires a separate review of the final change on its exact head before merge (AGENTS.md). A different agent or model can provide that code review; this is not external scientific peer review or independent reproduction of an experiment. A particular change’s review receipt must establish what was actually checked.

wording correction: earlier copy said every change had review by “a different agent and model”. The policy requires a separate exact-head review; the recorded receipt establishes the reviewer and scope for a particular change. That earlier sentence did not establish a different model for every review.

One gap is disclosed here. On 22 October 2026 a pull request was merged while its verify job was still running; that job then failed on a fence test whose base had moved, while the push run on main passed. Since 23 October the current version of our landing tool refuses to merge a pull request in this repository until verify has passed on its exact head. That is a check in our own tooling, not yet on GitHub's side: a rule there that requires the same check is waiting on the repository owner, and until it exists an older copy of the tool, or a person merging by hand, could still merge early.

6. Rerun it yourself.

You need git, Node 22, JDK 21 (with JAVA_HOME set to that JDK) and pnpm 10.11.1. Clone the repository: a ZIP download will not build, because the build reads git metadata. These are the same steps as Run locally in the README, one command per line, so they also paste into Windows PowerShell 5.1:

git -c core.autocrlf=false clone https://github.com/odin-labs-ai/odin-rnd
cd odin-rnd
npm install --global pnpm@10.11.1
pnpm install --frozen-lockfile --ignore-scripts
pnpm experiment
node scripts/run-roi-demos.mjs
pnpm test
pnpm build
pnpm check
pnpm serve

pnpm experiment reruns the three recorded experiments, pnpm test runs every check on the recorded evidence, pnpm build builds the site from the records and pnpm check checks links, digests and the export boundary. The clone line keeps the committed line endings, which the sha256 pins check byte for byte; on Linux and macOS that setting is already off. corepack enable also installs the pinned pnpm, but on Windows it needs an elevated shell (Run as administrator), so the steps use npm install --global pnpm@10.11.1, as CI does. pnpm test also needs python3 and bash on a POSIX system: on Windows, run it inside WSL or skip it.

The commands are in package.json and AGENTS.md; docs/VALIDATION.md records how the site itself was validated. For a single experiment, its folder holds the inputs and the frozen analysis; its field note says which parts you can rerun and which depend on a private corpus.

7. What this does not prove.

Most experiments here are small: one laptop, one model or a few dozen cases. Several use a private corpus, so you can check the hashes, the method and the counts but not rerun the cases. We wrote the experiments ourselves, about our own tools. A pre-registration stops us from moving the bar after the fact; it does not make a narrow question broad. Each field note states its own limits; read them before quoting a number.

Two pages hold the lab to this mechanically: provider economics and selected evidence and open questions. The build stops if a number on either is not read from an experiment record and linked to it (scripts/number-provenance.mjs).

wording clarification: the earlier Evidence link “what we have measured, and not” implied a complete inventory. The linked page presents selected published claims and open questions; this build check validates numbers shown there, not completeness.