LAB 13 · DOES AN AGENT COMPANY GET BETTER, MEASURED BLIND?

Does an agent company get better, measured blind?

The agents of our company on duty answered the same work items before and after we fixed their recipes and tools, and a separate AI reviewer judged every answer without being told which was which. On the nine items the fixes were made on, useful answers went from 2 of 18 to 10 of 18. On nine items held out from the fixing, from 3 of 18 to 5 of 18; on nine fresh ones, judged on the next version, from 4 of 18 to 12 of 18.

Each side of a set is 18 verdicts, not 9: nine items, each answered twice. The set the fixes were made on gained more than the first held-out set: 8 useful answers against 2.

What it is.

The lab in plain words, and how it runs.

An agent company is a team of AI agents, each with a job, such as reading a failed CI run and saying what went wrong. We fixed their recipes and tools. Did that make them better?

To find out without fooling ourselves, the agents answered the same work items before the fixes and after them, and a separate AI reviewer judged every answer without being told which version gave it. The blinding is partial, and the box below says how.

One set of items is the one we fixed on, so it says the least. Two other sets were held out from the fixing; the second was fresh, judged on the next version.

How it works: 27 work items, answered before and after the fixes, judged with partial blinding
  1. 27 work items in 3 setsThe tuning set, Held-out set one, Held-out set two
  2. The agents on dutyreal agent sessions on a small local model, qwen2.5:7b, with read-only tools

Before the fixes

  1. The old engine and recipesas they were on the main branch
  2. Each item answered twicethe same model, budget and agent per department in both arms

After the fixes

  1. The fixed engine and recipesv2 for the tuning set, v2 for held-out set one, v3 for held-out set two
  2. Each item answered twice againthe same items, the same reviewer

Judged by a separate AI reviewer, not a person

  1. The tuning setuseful answers: 2 of 18 before, 10 of 18 after
  2. Held-out set oneuseful answers: 3 of 18 before, 5 of 18 after
  3. Held-out set twouseful answers: 4 of 18 before, 12 of 18 after

Who judged.

An AI reviewer, not a person.

A SEPARATE AI REVIEWER, NOT A HUMAN

Every verdict on this page was given by a separate Claude session, which judged every answer against its own source. It is not a human's verdict. It gave each answer one of four labels: useful, partly, wrong or stuck; a session the system itself marked stuck counts as stuck, whatever the reviewer said of its partial answer.

The review sheet showed each item's four answers in a random order, with the key kept apart; the blinding is partial, because the shape of a tool's reply can show which arm wrote it.

How it was measured.

  • The agents. Real agent sessions on a small local model with read-only tools, the same model, budget and agent per department in both arms; the model is qwen2.5:7b.
  • Before. The engine and recipes on the main branch before the fixes.
  • After. The engine and recipes after the fixes: v2 for the tuning set and held-out set one, v3 for held-out set two.
  • Runs. Each item was answered twice in each arm.

The three sets

  • The tuning set: items T1 to T9, judged on v2 after the fixes.
  • Held-out set one: items A1 to A9, judged on v2 after the fixes.
  • Held-out set two: items B1 to B9, judged on v3 after the fixes.

The numbers.

Before and after, per set.

Each row is 18 verdicts: nine items, each answered twice. Useful is the share of those 18.

SetArmusefulpartlywrongstuckuseful share
The tuning setbefore2105111%
after (v2)1051256%
Held-out set onebefore3105017%
after (v2)594028%
Held-out set twobefore4122022%
after (v3)1242067%

Recounted at build from the per-item labels below; the build refuses the record if they do not give these counts, which are the committed summary's.

Every item, every run

Items by anonymous label: their keys name private repositories and CI runs, so only their kind is shown.

ItemKindBefore, runs one and twoAfter, runs one and two
T1a failed CI run to diagnosewrong, wrongpartly, wrong
T2a failed CI run to diagnoseuseful, usefuluseful, useful
T3a failed workflow run to diagnosewrong, wronguseful, useful
T4a pull request to reviewpartly, partlyuseful, partly
T5an issue to triagepartly, partlyuseful, useful
T6an issue to triagepartly, partlypartly, partly
T7an issue to triagepartly, partlyuseful, useful
T8a finance proposal to draftpartly, wrongpartly, useful
T9a finance proposal to draftstuck, partlystuck, stuck
A1a failed CI run to diagnosewrong, wrongpartly, partly
A2a failed CI run to diagnosewrong, wrongwrong, partly
A3a failed CI run to diagnoseuseful, partlypartly, wrong
A4a failed CI run to diagnosewrong, partlyuseful, useful
A5a pull request to reviewuseful, partlypartly, wrong
A6a pull request to reviewpartly, partlywrong, partly
A7an issue to triagepartly, partlypartly, partly
A8an issue to triagepartly, partlypartly, useful
A9an issue to triagepartly, usefuluseful, useful
B1a failed CI run to diagnosewrong, wrongwrong, wrong
B2a failed CI run to diagnoseuseful, partlyuseful, useful
B3a failed CI run to diagnosepartly, partlyuseful, partly
B4a failed CI run to diagnosepartly, usefuluseful, useful
B5a pull request to reviewpartly, partlyuseful, useful
B6a pull request to reviewpartly, partlyuseful, useful
B7an issue to triageuseful, partlyuseful, useful
B8an issue to triageuseful, partlyuseful, partly
B9an issue to triagepartly, partlypartly, partly

What they say.

  • The tuning set: useful from 11% to 56%, a gain of 8. The nine items of an earlier live run. The fixes were made on them: it is the set we studied while fixing, so it says least.
  • Held-out set one: useful from 17% to 28%, a gain of 2. Nine items picked by a fixed rule and frozen before any fix, so no fix had seen them. Its review found new failure shapes, which the next version, v3, fixed.
  • Held-out set two: useful from 22% to 67%, a gain of 8. Nine fresh items picked by the same rule after v3, leaving out both earlier sets, so no fix had seen them either. It judges v3, whose fixes came from held-out set one's review.
  • Tuning against held-out set one. The set the fixes were made on gained more than the first held-out set: 8 against 2.
  • A restart. The machine restarted during held-out set two, and its local model runtime updated itself; the after arm's second run of that set ran on the updated runtime. Run by run, useful of 9: before, 3 and then 1; after, 7 and then 5, the run on the updated runtime.

And on live work.

A summary only.

SUMMARY ONLY, NOT REPRODUCIBLE HERE

Then, on live work

After this measurement the agents went on duty on live work, a different set of items. The same reviewer judged 15 of their answers: 2 useful, 11 partly, 2 wrong. That is much worse than the second held-out set, and the write-up says so. Those answers were not committed, so this page cannot recount them: the figures are the write-up's.

Check it yourself.

Everything on this page comes from one record, published here and pinned by its sha256. It was read from the agent space's private repository: the blind before and after evaluation of the company's on-duty agents, its committed verdict labels, counts and summary, read at the main head.

Nothing here was unblinded for this page. The evaluation's own runner keeps the blind key and the raw answers local, by design: the reviewer's reasons quote private repositories. What was committed is the verdict labels, already unblinded, and this page recounts them.

The commits it stands on

  • Froze the tuning set and held-out set one, before any fix, . Private commit 7c446f09, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Froze held-out set two, to judge v3, . Private commit 79b0b315, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Added the verdict labels of all three sets, the counts and the summary, . Private commit c01782c4, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Added the write-up of the live run, . Private commit 4b1bf2df, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Last changed the write-up, . Private commit 8863df79, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
  • Last changed the runner, . Private commit cd29253e, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.

The files, by sha256

  • The evaluation's summary: the counts per set, before and after: 44c4b2f83b644564b754e2eddd1523918adcea0c5e50d7f90a2ba9c58c3fb94c
  • The tuning set's verdict labels, per item and run: 63fc8fb8198b7fe8d2faa968918f862c51c844db708a1bcdc00deb3db4484297
  • Held-out set one's verdict labels, per item and run: 04b949882961c4a6e8f02795ee15234dab66202376434e36d3c58ffeb2f5bb94
  • Held-out set two's verdict labels, per item and run: bfb90a5780718cbae5f1830224904218072eef632f652374810d0bddd45b7dc6
  • The tuning set's item list (not published: it names private repositories): c52a046df6c0e44275ff4e6d2367b4ea0abde2749505c8f273af4dbdd2473e80
  • Held-out set one's item list and the rule that picked it (not published): 32e3e7b62fa6e6a284f0b571785e5aefc8bb16d2c03d5f1fa25271888018a2da
  • Held-out set two's item list and the rule that picked it (not published): 431472e57e446257d44278259bda31c7c396ed6090178960baccac2cc835244f
  • The write-up of the measurement, and of the live run after it: edfbab57d7b9c204a50b0d5c2dfa5a2288539fab9c3ccea4d20bafbeb19c57c1
  • The evaluation's runner: pick, run, review sheet, unblind: 9df51663c02e884daf84e9df32d198d41e6305467257619fbbe478230c34115b

Main was checked by git merge-base --is-ancestor of each cited commit against the private main head, in a clone of the private repository, on .

What this does not show.

  • The judge is an AI, not a person, and the blinding is partial: the shape of a tool's reply can show which arm wrote it.
  • The sets are small: nine items each, 18 verdicts a side. A few answers either way move a share by ten points or more.
  • It shows that our own fixes (recipes, tool output, the facts handed over, routing) made these answers better. It does not show the model got better: the model was the same in both arms.
  • On live work afterwards the results were much worse; the write-up does not separate the two causes it names, other items and the updated runtime.

Its runs.

The runs of this lab's check that its record holds, each with its own record hash. The org machines' runs are a recount: each counts the committed verdict labels again, from the private repository's evidence at its main head, and compares them, set by set and with the live run, with the counts this page states. The evidence is frozen, so the recount is flat by construction: it gives the same result every day unless that evidence, or the counts this page states, change. It re-counts; it does not re-run the agents or re-judge their answers.

LATEST RUN · org-hosted · commit c169ad9b · record f419d9215e2b

Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.

This checks a committed record. A matching hash does not establish a new experiment or independent validation.

Each day on the org machines

One point per day the record holds org runs for: how many org machines reproduced the committed state hash. A value holds until the next day's runs; a day with no run is no point.

org machines that reproduced the hash, held until each change: 2026-11-06: 2; 2026-11-07: 2; 2026-11-08: 2; 2026-11-09: 2; 2026-11-10: 2; 2026-11-11: 2; 2026-11-12: 2; 2026-11-13: 22026-11-062026-11-1322222222
  1. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 9aaa4dcaaff0; ubuntu-latest, org-ci-heavy
  2. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 9aaa4dcaaff0; org-ci-heavy, ubuntu-latest
  3. : same hash on both org machines, 2 of 2 reproduced the committed hash; record 9aaa4dcaaff0; org-ci-heavy, ubuntu-latest
  4. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f6b457a9cb81; org-ci-heavy, ubuntu-latest
  5. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f6b457a9cb81; org-ci-heavy, ubuntu-latest
  6. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f419d9215e2b; org-ci-heavy, ubuntu-latest
  7. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f419d9215e2b; org-ci-heavy, ubuntu-latest
  8. : same hash on both org machines, 2 of 2 reproduced the committed hash; record f419d9215e2b; ubuntu-latest, org-ci-heavy