Same model, with and without a harness
We like to say the model is rented and the harness is yours. This lab tests that instead of saying it. One small made-up project, one change to it, one question: which files does the change touch, and does the test suite still pass? Each local model answers it bare, and again with a harness that ran three tools for it first. Same model, same question, same seed. The only difference is the harness.
What it is.
The lab in plain words, and how it runs.
People say the AI model is rented and the harness around it is yours. A harness is the code that runs tools for the model and checks what it says. This lab tests whether the harness changes the answers, with the model kept the same.
We made up a small project and one change to it, after which its tests fail. 5 local models were asked which files the change touches and whether the tests still pass: once with only the question and the change, and once with what the harness's tools found in the project first. The question came in two forms, one where the model may say it cannot tell and one where it must say pass or fail, and each model got each form 10 times bare and 10 times with the harness. Every first answer was scored by code, not by a person.
Two mistakes are counted: saying the failing tests pass, and naming a file that does not exist. With the harness, the same models made fewer of them in their first answers: 0 mistakes with the harness against 78 without it, over 100 first answers each way. The harness can also check an answer and ask again; that is shown further down, kept apart from these counts.
- A made-up project and one change11 files; after the change its tests fail
- 5 local modelsqwen2.5:7b, qwen3:30b-a3b, qwen2.5:14b, qwen2.5:32b, qwen3:32b; each asked 10 times bare and 10 times with the harness, for each form of the question
Bare: the model alone
- The question and the changenothing else
- May say UNKNOWN0 of 50 first answers said the failing tests pass; 29 of 50 named a file that does not exist
- Must say PASS or FAIL20 of 50 first answers said the failing tests pass; 29 of 50 named a file that does not exist
With a harness: the same model
- The question, the change and the tool outputthe harness's tools look at the project before the model is asked
- May say UNKNOWN0 of 50 first answers said the failing tests pass; 0 of 50 named a file that does not exist
- Must say PASS or FAIL0 of 50 first answers said the failing tests pass; 0 of 50 named a file that does not exist
The project, the question and the two arms.
Everything both arms share, and the one thing they do not.
AN AUTHORED FIXTURE, WRITTEN BY US
The project is a route planner for Northwind Labs, a courier we made up: 11 files of Python, four modules and their tests. The change moves one line in the pricing module, so that the departure depot is charged as a stop. The tests say it is not one, so after the change the suite fails. We wrote the project, the change and the question; nothing in it comes from a client.
- Bare. The model gets the question and the diff. Nothing else.
- With a harness. The model gets the same question and diff, plus the output of three tools the harness ran on the project before asking: the list of files, a search for the changed function, and the test run with its exit status.
- Two ways to ask. In the open question the model may answer UNKNOWN for the tests. In the forced one it must say PASS or FAIL, the shape of a coding agent's closing report.
- The test files too. The question asks for the test files that cover the changed function. The bare arm cannot see the project, so any test file it names is a guess.
- Scored by code. A reply invents a file when it names a path that is not in the project. It is a wrong PASS when it says the suite passes. Both are counted from the model's first answer, by the same scorer the runner uses.
The first answers, model by model.
Two questions, each asked bare and with the harness.
Each cell is 10 runs of one model, first answers only; run n of both arms used the same seed (1101 to 1110). In brackets, the Wilson 95% interval: with 10 runs a cell is wide, so read the totals before any one cell.
The open question: UNKNOWN allowed
When the model may say UNKNOWN, across all models: bare, 0 of 50 first answers said the failing suite passes and 29 of 50 named a file that does not exist; with the harness, 0 of 50 and 0 of 50. The same models made fewer of these mistakes with the harness.
| Model | Arm | Said the failing suite passes | Named a file that does not exist | Said UNKNOWN | Unreadable reply | Median time |
|---|---|---|---|---|---|---|
qwen2.5:7b | Bare | 0 of 10 (0% to 28%) | 10 of 10 (72% to 100%) | 10 of 10 | 0 of 10 | 0.5 s |
qwen2.5:7b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 0.6 s |
qwen3:30b-a3b | Bare | 0 of 10 (0% to 28%) | 7 of 10 (40% to 89%) | 8 of 10 | 0 of 10 | 0.4 s |
qwen3:30b-a3b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 0.4 s |
qwen2.5:14b | Bare | 0 of 10 (0% to 28%) | 2 of 10 (6% to 51%) | 10 of 10 | 0 of 10 | 0.7 s |
qwen2.5:14b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 1.3 s |
qwen2.5:32b | Bare | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 10 of 10 | 0 of 10 | 1.8 s |
qwen2.5:32b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 2.3 s |
qwen3:32b | Bare | 0 of 10 (0% to 28%) | 10 of 10 (72% to 100%) | 10 of 10 | 0 of 10 | 1.8 s |
qwen3:32b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 2.2 s |
The forced question: PASS or FAIL
When the model must say PASS or FAIL, across all models: bare, 20 of 50 first answers said the failing suite passes and 29 of 50 named a file that does not exist; with the harness, 0 of 50 and 0 of 50. The same models made fewer of these mistakes with the harness.
| Model | Arm | Said the failing suite passes | Named a file that does not exist | Said UNKNOWN | Unreadable reply | Median time |
|---|---|---|---|---|---|---|
qwen2.5:7b | Bare | 10 of 10 (72% to 100%) | 10 of 10 (72% to 100%) | 0 of 10 | 0 of 10 | 0.5 s |
qwen2.5:7b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 0.6 s |
qwen3:30b-a3b | Bare | 0 of 10 (0% to 28%) | 4 of 10 (17% to 69%) | 0 of 10 | 0 of 10 | 0.4 s |
qwen3:30b-a3b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 0.4 s |
qwen2.5:14b | Bare | 10 of 10 (72% to 100%) | 5 of 10 (24% to 76%) | 0 of 10 | 0 of 10 | 1.0 s |
qwen2.5:14b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 1.1 s |
qwen2.5:32b | Bare | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 1.7 s |
qwen2.5:32b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 2.2 s |
qwen3:32b | Bare | 0 of 10 (0% to 28%) | 10 of 10 (72% to 100%) | 0 of 10 | 0 of 10 | 1.8 s |
qwen3:32b | With a harness | 0 of 10 (0% to 28%) | 0 of 10 (0% to 28%) | 0 of 10 | 0 of 10 | 2.2 s |
The harness's own check.
Kept apart from the model's numbers.
Real
A harness can also check the answer. Ours refuses a reply that names a file not in the project, or whose test verdict is not the suite's own exit status, and asks once more with the reason. Its check is plain code, so what it lets through cannot be a wrong PASS or an invented file. That is true by construction, not a result of the model, and it is not counted in the numbers above.
| Model | Question | First answers the check refused | Accepted after one more ask |
|---|---|---|---|
qwen2.5:7b | open | 0 of 10 | none was asked again |
qwen2.5:7b | forced | 0 of 10 | none was asked again |
qwen3:30b-a3b | open | 0 of 10 | none was asked again |
qwen3:30b-a3b | forced | 0 of 10 | none was asked again |
qwen2.5:14b | open | 0 of 10 | none was asked again |
qwen2.5:14b | forced | 0 of 10 | none was asked again |
qwen2.5:32b | open | 0 of 10 | none was asked again |
qwen2.5:32b | forced | 0 of 10 | none was asked again |
qwen3:32b | open | 0 of 10 | none was asked again |
qwen3:32b | forced | 0 of 10 | none was asked again |
When, where, and how to check it.
Measured on , from to UTC, on an Apple M5 Max laptop with 128 GiB of memory, with Ollama 0.20.6. The calls ran one at a time; a cell would not start above a one-minute load of 40. The one-minute load before each call ranged from 6.9 to 9.8: other work was running on the same machine, so the times are what this machine gave at that load, not a benchmark. Each model's own default sampling settings were used; the qwen3 models ran with thinking off, and every reply was asked for as JSON.
The models, as served that day
qwen2.5:7b, digest845dbda0ea48ed749caafd9e6037047aa19acfcfd82e704d7ca97d631a0b697eqwen3:30b-a3b, digestad815644918f0eaab341c12b67837cc6dd4562342cdaf118f83d5d554cb37226qwen2.5:14b, digest7cdf5a0187d5c58cc5d369b255592f7841d1c4696d45a8c8a9489440385b22f6qwen2.5:32b, digest9f13ba1299afea09d9a956fc6a85becc99115a6d596fae201a5487a03bdc4368qwen3:32b, digest030ee887880fc378860c2dd35101da424377520441ae4bfe7be6deff8ade7840
Check it yourself
Everything on this page comes from one record, published here and pinned by its sha256. The project, the questions, the runner, the scorer and the full run log, with every reply as the model wrote it, are in the repository. With Ollama and the same models, the runner asks the same questions again; the build refuses this page if any of those files is not the one measured, or if a row of the record is not what the run log gives when each reply is scored again.
- The fixture (one sha256 over its sha256sum listing):
b9c87ea80821a771164eecd39bc0d121593858d2e35c9c722540aeb147475b70 - The open question:
4eb1535590474507a3f2fde1b4170691bc6f38e368d3ca3224172996c49690f1 - The forced question:
042701a46f47b47aaf52dca9148589445014e7ea6d964f9617936e987aca377d - The harness's tool text:
9706c52e3b93e7c2425f7a82029897fb3fbb6a96362b5ee0fca03e14370f73d1 - The change:
a2d5630449c220f656f015ea2510cc3c786de416c63ceff0056713525e9fc656 - The runner:
20478111e980e2b925ff0f51731298eca0a547b715975b9cf0728fdd38b42b0c - The scorer:
d77bc3ad3a983a876c95b2e5d5d8644e1e26e675e2ab8db5d490a2b1299b03af - The test run's output, as the harness showed it on that machine (another Python version words a traceback differently):
2739bf1dc7eb3224d56eeab114119081506f978a6ff351cce30aa9c511da3703 - The run log:
3015a7b00e6e8743942575ff514986c5969620a09686d86012c27da38f6de4e4
What this does not show.
- One small project, one change, one question, 5 local models. It does not say how often this happens on real projects, or with hosted models.
- The harness here is three tools and a check. A different harness, or a model asked to call the tools itself, could do better or worse.
- With 10 runs a cell, a single model's rate is wide; the intervals say how wide.
- In the harnessed arm the suite's result is in the tool output the model was given, so a right verdict there shows the model read what the harness ran, not that it reasoned its way to it. The comparison is about what the same model does with and without that output.
- A wrong PASS in the bare arm is a guess the model was not given the means to check. That is the point of the comparison, not a flaw in the model: the open question shows what the same models do when they are allowed to say they cannot tell.