Can an agent fix a red build?
Not yet. We gave the agent space's CI doctor 9 fresh red builds from one of our own private repositories, frozen before it saw any of them. Two separate AI reviewers judged every answer, each blind to the other. Judged real fixes: 0 of 9. The diagnoses held up better than the fixes did, and the page says which set every number comes from.
What it is.
The lab in plain words, and how it runs.
When a build goes red, someone has to read the log, find the cause and fix it. This lab asks whether an agent can do that job.
The agent space's CI doctor got 9 fresh red builds from one of our own private repositories, frozen before it saw any of them. Two separate AI reviewers, each blind to the other, judged every answer.
The answer so far is no. Most of the 9 had nothing in the code to fix, and on the 3 that did, none of its answers was judged a real fix. Its diagnoses held up better than its fixes.
- 9 red buildsfrozen before the doctor ran: 5 closed by a rule, 3 code bugs, 1 not runnable here
- The CI doctorthe agent space's own agent: reads the failure, names a cause, may write a patch
Nothing in the code to fix
- 5 closed by a rulered by design, or a cause outside the repository; no model is asked
- No patchboth reviewers agreed that was right, every time
Code bugs
- 3 code bugsthe most this set could have had fixed
- 1 of 3 failed again herethe rest passed when run again, so there was nothing to check a fix against
- 0 fixes passedthe patcher found none that made the build green
Two AI reviewers, blind to each other
- 18 verdictseach of the 9 builds judged twice, one case at a time
- Cause and fix, judged apartthe diagnosis tallies are in the numbers below, each with the set it counts
- Judged real fixes: 0 of 9 buildsnone of these verdicts is a person's
The numbers, and the set each one counts.
Two sets, never added together.
A SEPARATE AI REVIEWER, NOT A HUMAN
Every verdict here is a separate AI reviewer's, given one case at a time with no sight of the other reviewer, the doctor's name or the run's outcome. None is a person's. The first two rows are what the doctor did; the rest are the reviewers' verdicts.
| What | Count | Of which set |
|---|---|---|
| Judged a real fix | 0 of 9 | the held-out nine, both reviewer rounds |
| Closed by a rule, with no model session | 5 of 9 | the held-out nine, both reviewer rounds |
| Diagnosis right | 11 of 18 verdicts | the held-out nine, both reviewer rounds |
| Diagnosis partly right | 3 of 18 verdicts | the held-out nine, both reviewer rounds |
| Diagnosis wrong | 4 of 18 verdicts | the held-out nine, both reviewer rounds |
| No patch, and rightly so | 14 of 18 verdicts | the held-out nine, both reviewer rounds |
| No patch, but a fix was missed | 4 of 18 verdicts | the held-out nine, both reviewer rounds |
| Diagnosis right | 17 of 27 judgments | the earlier set, three runs of the doctor |
| Diagnosis partly right | 10 of 27 judgments | the earlier set, three runs of the doctor |
| Diagnosis wrong | 0 of 27 judgments | the earlier set, three runs of the doctor |
| Patch judged a fix | 1 of 27 judgments | the earlier set, three runs of the doctor |
| Patch judged wrong | 6 of 27 judgments | the earlier set, three runs of the doctor |
Two sets, never added together. The held-out nine are the test: frozen before any doctor run, judged by two reviewer rounds. The earlier set is 3 runs of the doctor over 9 earlier red builds, judged once each; it is where the diagnosis figures in our own notes come from, and its one judged fix is there and not on the held-out set.
The nine held-out runs.
One row per red build, as the two reviewers judged it.
The nine newest red runs in a window of 7 hours, one per workflow and branch, all from one private repository, frozen on before the doctor ran on any of them. A run is named by a keyed hash, not its id: the repository is private, and so are its runs.
A SEPARATE AI REVIEWER, NOT A HUMAN
| Run | What started it | What the doctor did | The cause, as the reviewers saw it | The diagnosis | The patch | A real fix |
|---|---|---|---|---|---|---|
run-kxhhqpyt | pull request | no patch by rule | red by design, both | right, both | no patch, rightly, both | no |
run-nvyvnyrx | push | no patch by rule | red by design, both | right, both | no patch, rightly, both | no |
run-nwkhxkhw | pull request | no patch by rule | red by design, both | right, both | no patch, rightly, both | no |
run-pthgknth | pull request | the patcher found no fix | a code bug, both | right, both | no patch: a fix was missed, both | no |
run-qyysmwxv | merge group | did not reproduce (passes here) | a flake / a code bug | partly right / right | no patch, rightly / no patch: a fix was missed | no |
run-tggxjykp | push | did not reproduce (passes here) | a code bug / a flake | right / partly right | no patch: a fix was missed / no patch, rightly | no |
run-wmjyvhxm | schedule | not runnable here | red by design, both | wrong, both | no patch, rightly, both | no |
run-zqrtqjqn | workflow run | no patch by rule | red by design, both | wrong, both | no patch, rightly, both | no |
run-ztmkwskg | schedule | no patch by rule | config or environment / red by design | right / partly right | no patch, rightly, both | no |
Where the two reviewers differ, both are shown, the first round's first.
Why nothing was fixed.
Most of these builds had nothing in them to fix. By the reviewers' classes, 5 were red by design: a gate doing its job, such as one that refuses a change only a person may land. 1 had its cause outside the repository. A rule closes such runs before any model is asked; it closed 5, and on all 5 both reviewers said "no patch" was right.
That leaves 3 code bugs, the most this set could have had fixed. Only one of them failed again on our own machine, so only that one could be worked on and checked: the patcher found no fix. The other 2 passed when run again, so no patch could be shown to work either way. Every "a fix was missed" above is one of these 3.
So the honest reading is narrow: on this set the rule's 5 closures were all judged right, and the doctor fixed nothing.
Check it yourself.
Everything on this page comes from one record, published here and pinned by its sha256. It was read from the CI doctor's evidence in the agent space's private repository: the held-out set, both reviewer rounds, their score, the rubric and the earlier set's judgments, read at its main. Private commit aa35e6db, cited by hash, checked as part of the private main aa35e6db on ; this page cannot check it, anyone with access can.
SUMMARY ONLY, NOT REPRODUCIBLE HERE
A recount, not a re-score
The scorer's own inputs, the key that unblinds each case and the doctor's raw answers, stayed on the machine that ran it, by design, and are not in the repository. What is there is each reviewer's verdict and the score they gave. The record carries both, and this site's build counts the verdicts and refuses to publish unless they give the score, field by field. So the score is checked here; the doctor's runs are not re-run.
The commits it stands on
- The held-out set frozen: the nine newest red runs, before any doctor run on them, . Private commit
eb9314c0, cited by hash, checked as part of the private mainaa35e6dbon ; this page cannot check it, anyone with access can. - The held-out set run and judged by two blind reviewer rounds, and scored, . Private commit
6455c102, cited by hash, checked as part of the private mainaa35e6dbon ; this page cannot check it, anyone with access can. - The earlier set judged blind, and unblinded, . Private commit
65c3272b, cited by hash, checked as part of the private mainaa35e6dbon ; this page cannot check it, anyone with access can.
The files, by sha256
- The held-out set: nine red runs, frozen before the doctor saw any of them:
fe23e1e80ae2ef2a2491745dc85a0169a8e0ce69e296c5f846ffea4191012338 - The first blind reviewer round's verdicts on the held-out set:
16943c1528c5ea89465aac63877b129737a50187c996a0742e29356c4aae1999 - The second blind reviewer round's verdicts on the held-out set:
f6c9f8650090f9ef658ecaff6c9f3e514b70e58e8e92eb02f09a9bae98b26ac4 - The score the two rounds gave the held-out set:
1ac7a1ba6910ac9b3465666cc7b1200b5aca346c2b7f851a1d97e0b826cd03f2 - The earlier set's unblinded judgments, three doctor runs over nine red runs:
d18cb552d2c78eb7851ee02a7fb0f177c2da5c3cde599eca4677c8cee320d05d - The rubric each reviewer judged one case by:
46b56b831d1a5f17e47f2bc7735a6e40b341151fcac27bd8f6c227a8e0c36a7d
Main was checked by GitHub compare of each cited commit against the private main head; behind by 0 means the commit is part of main, on .
What this does not show.
- Nine builds from one repository, over one afternoon. It does not say how the doctor would do on another codebase, or on more code bugs.
- The reviewers are AI, not people. Each judged one case alone, but neither verdict was checked by a person here.
- The page does not name the builds, the repository or the checks, so a reader cannot look them up. Someone with access to the private repository can.
- The doctor's own runs are not re-run here, only counted.
Its runs.
The runs of this lab's check that its record holds, each with its own record hash. The org machines' runs are a recount: each counts the committed verdicts again, from the private repository's evidence at its main head, and compares them with the counts this page states. The evidence is frozen, so the recount is flat by construction: it gives the same result every day unless that evidence, or the counts this page states, change. It re-counts; it does not re-score or re-judge.
LATEST RUN · org-hosted · commit c169ad9b · record 9e1467d6b2c8
Matching record hash on two org machine classes. The same record hash on org-ci-heavy, ubuntu-latest.
This checks a committed record. A matching hash does not establish a new experiment or independent validation.
Each day on the org machines
One point per day the record holds org runs for: how many org machines reproduced the committed state hash. A value holds until the next day's runs; a day with no run is no point.
- : same hash on both org machines, 2 of 2 reproduced the committed hash; record
6a7cf1bd96ba; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
6a7cf1bd96ba; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
6a7cf1bd96ba; ubuntu-latest, org-ci-heavy - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
6a7cf1bd96ba; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
6a7cf1bd96ba; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
a964b2f3fc74; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
a964b2f3fc74; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
9e1467d6b2c8; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
9e1467d6b2c8; org-ci-heavy, ubuntu-latest - : same hash on both org machines, 2 of 2 reproduced the committed hash; record
9e1467d6b2c8; ubuntu-latest, org-ci-heavy