The workbenches,
in full.

Two experiments you can rerun yourself, each with its recorded output, the command to run it and what the run does not show. Below them is the list of all our experiments and where each one stands. Opening this page runs nothing.

Can a written rule catch a change that breaks it?

We write down one rule about how the code is built, for example “this part may not depend on that part.” Then we run it on two copies of a project: one that follows the rule, and one we broke on purpose. The check should pass the first and flag the second.

SELECT EXPERIMENT
Watch the public runs
EXP / 001RECORDED RUN

Can a dependency cross the line?

A domain module imports the application layer. Does the same blueprint distinguish it from a conforming tree?

COPY A / FOLLOWS THE RULEGREEN / 100Should pass: it follows the rule.
COPY B / BROKEN ON PURPOSERED / 60Should be flagged: the break was planted.

Same rule, two copies, each with its conformance score (100 means no rule broken). The experiment counts as a success only when the check tells the copies apart.

Read the engine output
recipe module-layering [TypeScript/JavaScript · direct module graph] — Keep domain code below application code
GREEN conformant: score 100, exit 0
RED drift: score 60, would exit 1, violation domain-cannot-import-app
  observed forbidden direct import module:packages/domain/order.ts -> module:packages/app/checkout.ts is present
  evidence packages/domain/order.ts#L1
bce demo: module-layering discriminates GREEN from RED
REPRODUCE / NODE 22+ · FIRST RUN IS A LARGE DOWNLOAD: 0.3.1, THE VERSION THESE RUNS WERE RECORDED WITH, SHIPS ITS FULL DEV TOOLCHAIN
npm exec --yes --package=bce-engine@0.3.1 -- bce demo --recipe module-layering
RECORDED 2026-11-17T10:44:27.030ZFull result & provenance

Packaged fixtures, run with BCE 0.3.1. This tests the mechanism; it does not measure agent productivity or production reliability. Recording environment: GitHub Actions. Selectors inspect the recording; reproduction runs on your machine.

Can a model on your own laptop replace a hosted one?

The second bench compares two models instead of one rule on two copies: open weights running on a laptop against a hosted API, answering the same questions.

BENCH 02 / DECISIONS

EXP / 004Laya vs Jev: typed decisions without lock-in

Try it in your browser
EXP / 004ACCURACY / RED · 3 OF 4 CHECKS GREEN

Can an open model answer the same typed decisions on a laptop, with no vendor key?

WHAT IS A SYSTEM-1 DECISION MODEL?

It answers a typed question about a situation: pick one of these options, give a score, or say yes or no. It returns that answer with a confidence. It does not write text, so software can branch on the answer directly.

WHY NO VENDOR LOCK-IN MATTERS

A decision inside your control flow is a dependency. Jev is a hosted API: every answer needs its key, its network and its price. Laya publishes open Apache-2.0 weights that run on your own machine, so the hosted model can become a comparison instead of a requirement.

SPECIMEN A / LAYA · OPEN WEIGHTS, LOCAL39 / 48correct · median 9 ms per decision
SPECIMEN B / JEV · HOSTED API46 / 48correct · median 378 ms, including the network

Result: the local port gives the same decision as the model’s reference code on 48 / 48 rows, and its median call here took 9 ms on the laptop against 378 ms for the hosted call including its network round trip, but it is 14.6 points less accurate than Jev. That is beyond the 10-point limit set before the run, so the accuracy check is RED and stays that way.

  • REDWithin 10 points of Jev. 39 / 48 against 46 / 48: 14.6 points behind.
  • GREENConfident answers are right. 2 of 2 answers at confidence ≥ 0.9 were correct, but only 2 of 48 reached it.
  • GREENSame answers as the reference code. 48 / 48 same decision; largest probability difference 0.0021 (limit 0.01).
  • GREENFaster than the hosted call, network included. Median 9 ms on the laptop against 378 ms for the hosted call, including the network.
REPRODUCE / APPLE SILICON, MACOS 14 OR LATER + UV · FROM A CLONE OF THIS REPOSITORY
git clone --depth 1 https://github.com/odin-labs-ai/odin-rnd && cd odin-rnd
cd experiments/laya-vs-jev
uv venv -p 3.12 .venv
VIRTUAL_ENV=$PWD/.venv uv pip install -r requirements.txt
.venv/bin/python run.py            # add --out <file> to avoid overwriting today's recording
MEASURED 2026-09-24 13:34 UTC · APPLE M5 MAX · MLX 0.32.2Full result & provenance Read the field note

48 hand-written rows on one busy laptop (load average 36–47 during the run) is a mechanism check, not a benchmark. It does not predict accuracy on your decisions. Jev’s latency includes the public internet; Laya’s confident-answer figure rests on 2 answers. Opening this page runs nothing; the browser demo downloads and runs Laya only when you ask it to.

All experiments

The experiments on this site by number, the latest at the top, each with its question, where it stands and the page that records it.

  1. EXP 021 · RESULTSKill the database mid-session, does anyone notice?Single host: MongoDB fits four write-outage and two acknowledgement-gap scenarios; PostgreSQL two by feed gate. No acknowledged writes lost; no overall winner. Strict-ack and disturbance uncertainties remain.
  2. EXP 020 · PRE-REGISTEREDMongoDB on its home ground: where does it fit Odin?Pre-registered; S2 is complete, and amendment 05 records how its run went. S1, S3 and S4 have not run.
  3. EXP 019 · RESULTS · MIXEDRouting by our own rulebook, measured again: three fresh runs of the routed armPre-registered verdict: routing beats always-Opus, within a wide replicate allowance, and does not beat always-Luna.
  4. EXP 018 · RESULTSDoes routing by our own rulebook beat one fixed model on real tasks?Eight real coding tasks, three arms, three runs each, scored by the share of hidden test cases fixed (no task was fully solved). Per case fixed, a table copied from our routing rule paid 13.3 cents (61% of cases), always Opus 5.5 paid 24.7 (65%) and always GPT-6 Luna paid 4.3 (52%). The pre-registered verdict is incomplete: one routed cell hit its spend stop.
  5. EXP 017 · RESULTSWhat does a defect caught in review cost, model by model?Four models on one tool-free review seat caught between 26 and 31 of 60 defects, all within noise, at 0.06 to 8.3 cents a review. gpt-oss-120b, MiniMax M3.1 Flash and Claude Opus 5.5 are on the pre-registered frontier; Claude Sonnet 5.5 is not.
  6. EXP 016 · RESULTS · REFUTEDDoes the same model cost the same through every broker?Measured on 720 identical calls: two brokers billed exactly list price for Claude Sonnet 5.5 and GPT-4o mini, the third 10% above it on both, a spread of 1.10 against a pre-registered bar of 1.05. No broker served a different model.
  7. EXP 015 · PRE-REGISTEREDDoes Odin's cost estimate match what the brokers bill?Pre-registration published; no field note yet.
  8. EXP 014 · RESULTS · MIXEDWhere would MongoDB earn a place in Odin?99,000 requests and a separate 19,800-request repeat found two advantages over monolithic JSONB, but none over the stronger PostgreSQL alternatives, for these workloads on one machine. All seventeen outcomes and the durability amendment are retained.
  9. EXP 013 · RESULTS · REFUTEDDoes flexible workflow data need MongoDB?Both stores passed 20,000 operations, 42 correctness checks and ten evolution suites behind Temporal activities. For this workload, a MongoDB advantage remains unresolved and flexible JSON alone gave no demonstrated reason to switch.
  10. EXP 012 · RESULTS · REFUTEDCan a reviewer talk itself out of its false alarms?No. Asked to name a concrete failing input for every finding or drop it, MiniMax M3.1 Flash kept 249 of 269 of its own findings. False alarms stayed at 20 and 22 of 30 clean changes, against a pre-registered bar of 2.
  11. EXP 011 · PRE-REGISTEREDWhere does a model reviewer earn its keep?Pre-registered; not yet run.
  12. EXP 010 · PRE-REGISTEREDWould you notice it is not MongoDB?Pre-registration published in the repository, sha256 61f88f36e8ac…; no field note yet.
  13. EXP 009 · RESULTS · MIXEDUnload leaves no traceP1 refuted: 1 of 400 H1 trials diverged from R0 (95% Wilson upper bound 1.40%). P2 supported: 0 silent-inert and 0 false-inactive kernel configurations of 120. All 5 validity gates passed, so the record is informative.
  14. EXP 008 · PRE-REGISTEREDHand over the cache, not the textPre-registered; not yet run.
  15. EXP 007 · RESULTS · REFUTEDWhich of your gate's rules need a model at all?Pre-registered; measured 7 Oct 2026: the premise is refuted.
  16. EXP 006 · RESULTS · SUPPORTEDnina reviews the changePre-registered; measured 01 OCT 2026: bar met, nina in the spotlight. Amended 01 Oct 2026, before any counted run.
  17. EXP 005 · RESULTS · REFUTEDJev as a fast gatePre-registered; measured 29 Sep 2026: the claim is refuted.
  18. EXP 004 · MEASUREDSystem-1 decisions without lock-in: Jev vs open-weight LayaMeasured on one laptop, 48 hand-written rows.
  19. EXP 003 · RECORDED · SUPPORTEDWhat happens when configuration widens?Recorded 17 Nov 2026: the same rule passed the copy that follows it (GREEN, score 100) and flagged the copy we broke on purpose (RED, score 60).
  20. EXP 002 · RECORDED · SUPPORTEDDoes the boundary hold in Python?Recorded 17 Nov 2026: the same rule passed the copy that follows it (GREEN, score 100) and flagged the copy we broke on purpose (RED, score 60).
  21. EXP 001 · RECORDED · SUPPORTEDCan a dependency cross the line?Recorded 17 Nov 2026: the same rule passed the copy that follows it (GREEN, score 100) and flagged the copy we broke on purpose (RED, score 60).