LAB 02 · EVERY BRAIN ON THE SAME TASKS

Every brain on the same tasks.

Can a 1 GB model on your own machine do real agent tasks as well as the cloud models? Small local models, cloud models and a set of hand-written rules, all on the same seeded tasks. We write down what we expect before each run, then count.

What it is.

An agent is a program that keeps deciding what to do next: walk somewhere, buy something, help someone who asks. Its brain is whatever makes that decision. Here a brain is one of three things: a set of hand-written rules, a small language model running on our own desktop PC, or a cloud model reached over the internet.

Every brain plays the same small test worlds on the same seeds, so each one meets exactly the same situations. A task counts as done only when the world itself says it is done.

Before each run we write down what we expect and commit it. Afterwards we score those predictions by a rule fixed in advance, and we publish the ones that fail too.

How it works: 10 brains on the same seeded tasks, 4,120 episodes in 4 experiments, a task done only when the world says so
  1. 10 brainshand-written rules, 4 on our own machine, 5 in the cloud
  2. Predictions first4 experiments, each written down and committed before its run

EXP-W01 and EXP-W02: the whole task set, local and cloud

  1. 7 task families in 5 worldsseeds 40–47, the same for every brain, harness h4
  2. 9 brains, 280 episodes eachthe rules, 3 local, 5 cloud
  3. Best cloud 274 / 280, best local 266 / 280Cloud: Mistral Small 3.2. Local: Our tuned 1 GB model. The rules: 218 / 280. The 95 % intervals overlap: no detectable difference.

EXP-W03: Are the cloud models really better at following a person's instruction?

  1. One task in 5 worldsseeds 100–139, the same for every brain, harness h4
  2. 4 brains, 200 episodes eachthe rules, 1 local, 2 cloud
  3. Best cloud 190 / 200, best local 153 / 200Cloud: Mistral Small 3.2. Local: Our tuned 1 GB model. The rules: 55 / 200. The 95 % intervals do not overlap: the gap is real.

EXP-W04: After fixing the bench, is the gap still there?

  1. One task in 5 worldsseeds 140–179, the same for every brain, harness h5
  2. 4 brains, 200 episodes eachthe rules, 2 local, 1 cloud
  3. Best cloud 188 / 200, best local 174 / 200Cloud: Mistral Small 3.2. Local: Our tuned 1 GB model, brain v3.2. The rules: 62 / 200. The registered 95 % intervals overlap, but that is not claimed. 25 of Mistral Small 3.2's 200 episodes (World D, seeds 140–164) ran during a service outage. With them re-run, which is not the registered result, Mistral Small 3.2 finished 199 / 200. Counted with the re-run, the best cloud and best local 95 % intervals do not overlap.

Small test worlds, not your workload. A brain that does well here has done well on these tasks, on these seeds, on this bench. It does not promise the same on yours.

What we found.

Every number here is counted from one of the experiments below, which says where it came from.

  1. Across the whole task set, the small local model keeps up.

    Our tuned 1 GB model finished 266 / 280 tasks (95.0 %) on our own machine, with no per-task charge (running it still takes the machine and its power). The best cloud model, Mistral Small 3.2, finished 274 / 280. Their 95 % intervals overlap, so by our rule there is no detectable difference.

  2. Following a person's instruction, the cloud models are clearly better.

    When a person tells the agent what to do, the cloud models win, and with enough episodes the gap is real: Mistral Small 3.2 finished 190 / 200 against 153 / 200 for our model, and the intervals (91.0–97.3 % against 70.2–81.8 %) do not overlap. Fixing a bench bug narrowed the gap. It did not close it.

  3. Every model beats the rules.

    In every experiment, every model brain finished more tasks than the hand-written rules, with intervals that do not overlap. On instructions alone, the rules managed 55 / 200.

  4. What we got wrong, and say so.

    The bench showed a shop's price for one item while the seller charged for the whole stack, so some buys an agent thought it could afford were refused. The first three experiments ran with that bug; the fourth ran after the fix, and the two are never compared as one measurement. In the fourth, our boldest prediction held as registered, but only because the cloud model's service failed for part of the run. With those 25 episodes re-run it finished 199 / 200, so we do not claim that prediction.

The whole task set: local and cloud on the same seeds.

EXP-W01 measured the local brains and EXP-W02 the cloud ones, on the same seeds, worlds, tasks, harness and brain version, so they share one chart.

Mistral Small 3.2 EXP-W02Mistral Small 3.2 (EXP-W02): 274 / 280 finished, 97.9 %, 95 % interval 95.4–99.0 %
GPT-4o mini EXP-W02GPT-4o mini (EXP-W02): 273 / 280 finished, 97.5 %, 95 % interval 94.9–98.8 %
Llama 3.3 70B EXP-W02Llama 3.3 70B (EXP-W02): 273 / 280 finished, 97.5 %, 95 % interval 94.9–98.8 %
Claude Haiku 4.5 EXP-W02Claude Haiku 4.5 (EXP-W02): 272 / 280 finished, 97.1 %, 95 % interval 94.5–98.5 %
Our tuned 1 GB model EXP-W01Our tuned 1 GB model (EXP-W01): 266 / 280 finished, 95.0 %, 95 % interval 91.8–97.0 %
Llama 3.2 3B EXP-W01Llama 3.2 3B (EXP-W01): 260 / 280 finished, 92.9 %, 95 % interval 89.2–95.3 %
Gemini 2.5 Flash EXP-W02Gemini 2.5 Flash (EXP-W02): 259 / 280 finished, 92.5 %, 95 % interval 88.8–95.0 %
Qwen2.5 1.5B, untuned EXP-W01Qwen2.5 1.5B, untuned (EXP-W01): 253 / 280 finished, 90.4 %, 95 % interval 86.3–93.3 %
Hand-written rules EXP-W01Hand-written rules (EXP-W01): 218 / 280 finished, 77.9 %, 95 % interval 72.6–82.3 %
Share of tasks finished, per brain, EXP-W01 and EXP-W02. The dot is the share of tasks finished; the line is its 95 % interval. Two brains differ only where their lines do not overlap. The axis starts at 70 %. The table below holds the same numbers.
ExperimentBrainWhereFinishedRate95 % intervalFallbacksModel time per task
EXP-W02Mistral Small 3.2Mistral Small 3.2, 24B, instruct.Cloud274 / 28097.9 %95.4–99.0 %51.2 s
EXP-W02GPT-4o miniGPT-4o mini.Cloud273 / 28097.5 %94.9–98.8 %80.8 s
EXP-W02Llama 3.3 70BLlama 3.3 70B, instruct.Cloud273 / 28097.5 %94.9–98.8 %11.4 s
EXP-W02Claude Haiku 4.5Claude Haiku 4.5.Cloud272 / 28097.1 %94.5–98.5 %01.1 s
EXP-W01Our tuned 1 GB modelQwen2.5 1.5B, tuned on our own decisions.Local266 / 28095.0 %91.8–97.0 %45.1 s
EXP-W01Llama 3.2 3BA general model from another family, about 2 GB.Local260 / 28092.9 %89.2–95.3 %34.2 s
EXP-W02Gemini 2.5 FlashGemini 2.5 Flash.Cloud259 / 28092.5 %88.8–95.0 %1580.3 s
EXP-W01Qwen2.5 1.5B, untunedThe base our model was tuned from.Local253 / 28090.4 %86.3–93.3 %903.7 s
EXP-W01Hand-written rulesThe control: no model at all.Rules218 / 28077.9 %72.6–82.3 %0—
Finished per task family, out of the episodes each family got.
BrainTask family A: buildTask family B: a group challengeTask family C: earn moneyTask family D: help a newcomerTask family E: a chain of goalsTask family F: survive a sudden eventFollow a person's instruction
Mistral Small 3.2 EXP-W0240 / 4040 / 4040 / 4040 / 4040 / 4036 / 4038 / 40
GPT-4o mini EXP-W0240 / 4040 / 4040 / 4040 / 4040 / 4036 / 4037 / 40
Llama 3.3 70B EXP-W0240 / 4040 / 4040 / 4040 / 4040 / 4036 / 4037 / 40
Claude Haiku 4.5 EXP-W0240 / 4040 / 4040 / 4040 / 4040 / 4036 / 4036 / 40
Our tuned 1 GB model EXP-W0140 / 4040 / 4040 / 4040 / 4040 / 4036 / 4030 / 40
Llama 3.2 3B EXP-W0140 / 4040 / 4040 / 4040 / 4040 / 4036 / 4024 / 40
Gemini 2.5 Flash EXP-W0240 / 4040 / 4040 / 4039 / 4040 / 4036 / 4024 / 40
Qwen2.5 1.5B, untuned EXP-W0140 / 4040 / 4040 / 4039 / 4040 / 4036 / 4018 / 40
Hand-written rules EXP-W0140 / 4040 / 4040 / 4039 / 4040 / 408 / 4011 / 40
Finished per kind of instruction.
BrainBuy something for themDeal with a threatHand something overGo and meet someone
Mistral Small 3.2 EXP-W028 / 1010 / 1010 / 1010 / 10
GPT-4o mini EXP-W027 / 1010 / 1010 / 1010 / 10
Llama 3.3 70B EXP-W027 / 1010 / 1010 / 1010 / 10
Claude Haiku 4.5 EXP-W028 / 1010 / 1010 / 108 / 10
Our tuned 1 GB model EXP-W015 / 1010 / 106 / 109 / 10
Llama 3.2 3B EXP-W015 / 102 / 1010 / 107 / 10
Gemini 2.5 Flash EXP-W027 / 106 / 105 / 106 / 10
Qwen2.5 1.5B, untuned EXP-W013 / 1010 / 101 / 104 / 10
Hand-written rules EXP-W014 / 102 / 100 / 105 / 10

EXP-W01 · 24 October 2026Which small local model makes the best agent brain?

Four brains on our own machine, on the whole task set: does a general model from another family play as well as our tuned 1 GB model, and how much does the tuning add?

7 task families · 5 worlds · seeds 40–47 · 280 episodes per brain · harness h4

What we predicted, and what happened

  1. holds on the point estimate, no detectable difference

    P1. Our tuned model finishes at least as many tasks as Llama 3.2 3B.

    266 / 280 against 260 / 280, and the intervals overlap (91.8–97.0 % against 89.2–95.3 %). A general model of another family plays about as well, at twice the memory.

  2. fails

    P2. Both model brains beat the rules on following an instruction by at least 15 of 40.

    Our tuned model clears it (30 / 40 against the rules' 11 / 40). Llama 3.2 3B does not (24 / 40).

  3. not supported

    P3. The untuned Qwen2.5 1.5B scores below our tuned model, with intervals that do not overlap.

    253 / 280 against 266 / 280; the intervals overlap (86.3–93.3 % against 91.8–97.0 %).

Not predicted, but clear

  • Every model brain beats the rules overall, with intervals that do not overlap: the rules at 72.6–82.3 %, the lowest model brain at 86.3–93.3 %.
  • The tuning shows in the fallbacks, not the headline: the untuned base fell back to the rules 90 times, our tuned model 4 times.

Caveats

  • Ran on harness h4, which showed a shop's price for one item while the seller charged for the whole stack, so some buys an agent believed it could afford were refused. The bug was found in EXP-W03's traces and fixed in h5, before EXP-W04. These rows are never compared with h5 rows as the same measurement.
  • The local model server stopped serving the models several times mid-run. The bench recorded those episodes as skipped and the run was resumed with the identical command, so only the missing episodes ran. Every brain has its full count of episodes; the skipped rows hold no result and are not counted.
  • The run paused several times when the PC was short of free memory, because other work was running on it. These were pauses, not aborts.
  • The untuned model's fallbacks are spread over all five worlds and none sits near an outage, so they are the model's own.
  • On following an instruction, our tuned model scored 30 / 40 and the untuned base 18 / 40: the largest gap in the table, but with this few episodes the intervals still overlap. EXP-W03 was run to settle the question.

PRE-REGISTERED AT 3d7137a3 · PRE-REGISTRATION SHA-256 7191ed201db39f7a… · COUNTED FROM 1,120 RESULT ROWS AT 158b75d116e3

EXP-W02 · 24 October 2026Do cloud models make better agent brains than our tuned local model?

Five cloud models on the same seeds, worlds and tasks as EXP-W01: does any of them beat our tuned 1 GB model?

7 task families · 5 worlds · seeds 40–47 · 280 episodes per brain · harness h4

What we predicted, and what happened

  1. holds

    P1. No cloud model beats our tuned local model overall with intervals that do not overlap.

    The best, Mistral Small 3.2 at 95.4–99.0 %, still overlaps our tuned model at 91.8–97.0 %.

  2. holds

    P2. At least one cloud model beats our tuned model on following an instruction by 5 of 40 or more.

    Mistral Small 3.2 38 / 40, Llama 3.3 70B 37 / 40, GPT-4o mini 37 / 40 and Claude Haiku 4.5 36 / 40, against our model's 30 / 40. None of these is a difference by the interval rule yet.

  3. holds

    P3. Every cloud model stays under the cost per successful task set before the run.

    By a wide margin. Cost was tracked, but this site does not publish money figures.

  4. holds

    P4. All five beat the rules with intervals that do not overlap.

    The lowest, Gemini 2.5 Flash at 88.8–95.0 %, is above the rules at 72.6–82.3 %.

Not predicted, but clear

  • Gemini 2.5 Flash fell back to the rules 158 times, far more than any other model. Its replies did not fit the bench's strict reply format. That is a fit problem between this model and our format, not a verdict on the model in general, and it is why Gemini trails on following an instruction.

Caveats

  • Ran on harness h4, with the shop-price bug described under EXP-W01. These rows are never compared with h5 rows as the same measurement.
  • A second bench bug: the bench's own cost column read zero for every cloud model in this run, so its leaderboard labelled them free and local. They were not free. The bug was fixed after the run; the result files are unchanged.
  • The cloud models are compared with EXP-W01's local rows: the same seeds, worlds, tasks, harness and brain version, measured earlier the same day and not re-run.
  • Every cloud model ran all its episodes, with no service errors, skips or spending-cap hits.

PRE-REGISTERED AT c3fa1656 · PRE-REGISTRATION SHA-256 09d4adcd58c5216b… · COUNTED FROM 1,400 RESULT ROWS AT 158b75d116e3

Following a person's instruction, with more episodes.

Only the instruction task, on five times as many seeds, on the bench as it was (harness h4).

Mistral Small 3.2Mistral Small 3.2: 190 / 200 finished, 95.0 %, 95 % interval 91.0–97.3 %
GPT-4o miniGPT-4o mini: 177 / 200 finished, 88.5 %, 95 % interval 83.3–92.2 %
Our tuned 1 GB modelOur tuned 1 GB model: 153 / 200 finished, 76.5 %, 95 % interval 70.2–81.8 %
Hand-written rulesHand-written rules: 55 / 200 finished, 27.5 %, 95 % interval 21.8–34.1 %
Share of tasks finished, per brain, EXP-W03. The dot is the share of tasks finished; the line is its 95 % interval. Two brains differ only where their lines do not overlap. The axis starts at 20 %. The table below holds the same numbers.
BrainWhereFinishedRate95 % intervalFallbacksModel time per task
Mistral Small 3.2Mistral Small 3.2, 24B, instruct.Cloud190 / 20095.0 %91.0–97.3 %82.4 s
GPT-4o miniGPT-4o mini.Cloud177 / 20088.5 %83.3–92.2 %243.0 s
Our tuned 1 GB modelQwen2.5 1.5B, tuned on our own decisions.Local153 / 20076.5 %70.2–81.8 %24.4 s
Hand-written rulesThe control: no model at all.Rules55 / 20027.5 %21.8–34.1 %0—
Finished per world, out of the episodes each world got.
BrainWorld AWorld BWorld CWorld DWorld E
Mistral Small 3.240 / 4030 / 4040 / 4040 / 4040 / 40
GPT-4o mini38 / 4028 / 4040 / 4037 / 4034 / 40
Our tuned 1 GB model19 / 4030 / 4036 / 4036 / 4032 / 40
Hand-written rules17 / 406 / 4012 / 4010 / 4010 / 40
Finished per kind of instruction.
BrainBuy something for themDeal with a threatHand something overGo and meet someone
Mistral Small 3.240 / 5050 / 5050 / 5050 / 50
GPT-4o mini34 / 5050 / 5050 / 5043 / 50
Our tuned 1 GB model26 / 5049 / 5042 / 5036 / 50
Hand-written rules8 / 5018 / 5015 / 5014 / 50

EXP-W03 · 24 October 2026Are the cloud models really better at following a person's instruction?

EXP-W02's largest gap, powered: only the instruction task, on five times as many seeds, for our tuned model, the two best-value cloud models and the rules.

One task · 5 worlds · seeds 100–139 · 200 episodes per brain · harness h4

What we predicted, and what happened

  1. holds

    P1. Mistral Small 3.2 beats our tuned local model on following an instruction, with intervals that do not overlap.

    91.0–97.3 % against 70.2–81.8 %.

  2. holds

    P2. GPT-4o mini does too.

    83.3–92.2 % against 70.2–81.8 %.

  3. holds

    P3. Every model brain beats the rules, with intervals that do not overlap.

    The lowest model, ours at 70.2–81.8 %, against the rules at 21.8–34.1 %.

  4. holds

    P4. The whole run stays under the cost ceiling set before it.

    By the real spend on the key, which matches the bench's own estimate. Cost was tracked, but this site does not publish money figures.

Not predicted, but clear

  • Mistral Small 3.2 against GPT-4o mini shows no detectable difference: 91.0–97.3 % against 83.3–92.2 %.
  • Where our model loses is mostly one world: World A, 19 / 40 against Mistral Small 3.2's 40 / 40. World B is hard for every brain.

Caveats

  • Ran on harness h4, with the shop-price bug described under EXP-W01. Reading this experiment's failed episodes is how the bug was found. EXP-W04 asks the same question after the fix, on new seeds; the two are never compared as one measurement.
  • No skips and no service errors, and the bench's cost column now prices the cloud models.

PRE-REGISTERED AT 1f75de87 · PRE-REGISTRATION SHA-256 2357a516ac3b9487… · COUNTED FROM 800 RESULT ROWS AT 158b75d116e3

The same question after the bench fix.

The instruction task on new seeds, after the shop-price fix (harness h5). These rows are never compared with the ones above as one measurement, so they get their own chart.

Mistral Small 3.2Mistral Small 3.2: 188 / 200 finished, 94.0 %, 95 % interval 89.8–96.5 %
Our tuned 1 GB model, brain v3.2Our tuned 1 GB model, brain v3.2: 174 / 200 finished, 87.0 %, 95 % interval 81.6–91.0 %
Our tuned 1 GB modelOur tuned 1 GB model: 172 / 200 finished, 86.0 %, 95 % interval 80.5–90.1 %
Hand-written rulesHand-written rules: 62 / 200 finished, 31.0 %, 95 % interval 25.0–37.7 %
Share of tasks finished, per brain, EXP-W04. The dot is the share of tasks finished; the line is its 95 % interval. Two brains differ only where their lines do not overlap. The axis starts at 20 %. The table below holds the same numbers.
BrainWhereFinishedRate95 % intervalFallbacksModel time per task
Mistral Small 3.2Mistral Small 3.2, 24B, instruct.Cloud188 / 20094.0 %89.8–96.5 %1131.9 s
Our tuned 1 GB model, brain v3.2The same model, with named arguments and the agent's bag and money stated.Local174 / 20087.0 %81.6–91.0 %03.3 s
Our tuned 1 GB modelQwen2.5 1.5B, tuned on our own decisions.Local172 / 20086.0 %80.5–90.1 %04.1 s
Hand-written rulesThe control: no model at all.Rules62 / 20031.0 %25.0–37.7 %0—
Finished per world, out of the episodes each world got.
BrainWorld AWorld BWorld CWorld DWorld E
Mistral Small 3.240 / 4040 / 4039 / 4029 / 4040 / 40
Our tuned 1 GB model, brain v3.236 / 4032 / 4035 / 4035 / 4036 / 40
Our tuned 1 GB model36 / 4036 / 4035 / 4036 / 4029 / 40
Hand-written rules17 / 409 / 4013 / 4010 / 4013 / 40
Finished per kind of instruction.
BrainBuy something for themDeal with a threatHand something overGo and meet someone
Mistral Small 3.245 / 5046 / 5049 / 5048 / 50
Our tuned 1 GB model, brain v3.243 / 5049 / 5043 / 5039 / 50
Our tuned 1 GB model35 / 5050 / 5049 / 5038 / 50
Hand-written rules15 / 5022 / 5010 / 5015 / 50

EXP-W04 · 25 October 2026After fixing the bench, is the gap still there?

The instruction task again, on new seeds, under harness h5 (the shop-price fix) and with a second version of our model's brain that names its arguments.

One task · 5 worlds · seeds 140–179 · 200 episodes per brain · harness h5

What we predicted, and what happened

  1. holds

    P1. Under h5, our tuned model with brain v3.1 finishes at least 165 of 200.

    It finished 172 / 200.

  2. holds

    P2. Brain v3.2 against v3.1 on our tuned model: no detectable difference.

    174 / 200 against 172 / 200; the intervals overlap.

  3. holds as registered, not claimed

    P3. Mistral Small 3.2 against our tuned model on brain v3.2: no detectable difference. The bold one.

    The intervals overlap, narrowly (89.8–96.5 % against 81.6–91.0 %). It is not robust: part of the cloud model's run hit service errors. With those 25 episodes re-run, it finishes 199 / 200 (97.2–99.9 %), clearly better than ours. We do not claim this prediction.

  4. holds

    P4. Every model brain beats the rules, with intervals that do not overlap.

    The rules at 25.0–37.7 %.

  5. holds

    P5. The run stays under the cost ceiling set before it.

    By the bench's own estimate for the cloud model's episodes. Cost was tracked, but this site does not publish money figures.

Not predicted, but clear

  • Brain v3.2 does not measurably help on top of the fix. It does better at buying something for the person (43 / 50 against 35 / 50) and gives back on handing something over (43 / 50 against 49 / 50).
  • Where our model still loses is going to meet someone and buying for them: the indirect phrasings, such as being asked to meet by someone rather than to go somewhere.

Caveats

  • Ran on harness h5, which shows the cost of one purchase, and is never compared with EXP-W03's h4 rows as if they were the same measurement.
  • The first launch skipped the cloud model: an old, dead key in the PC's environment was refused. The arm was resumed with the identical command and the live key, so only the missing episodes ran.
  • In World D, seeds 140–164, the cloud model's calls returned errors: a call gave no reply, and the rules decided instead. From the next seed on, World D is normal. This biases the result against the cloud model, and the run log does not record the error status, so an outage looks like a weak model. The bench should record a service error apart from an unusable reply.
  • Sensitivity check, disclosed and not part of the registered result: those 25 episodes were re-run with the same command and seeds, and 25 of them finished, with only ordinary fallbacks and no service errors.
  • The real spend on the key cannot be used for this run, because other work spent on the same key at the same time; P5 rests on the bench's own estimate.
  • The PC was short of memory at times. Episodes waited; none were dropped.

PRE-REGISTERED AT 5a01a950 · PRE-REGISTRATION SHA-256 cd25acbc7ff10e7d… · COUNTED FROM 800 RESULT ROWS AT 158b75d116e3

How it was run.

  • Pre-registered. The question, the brains, the worlds, the seeds, the predictions and the decision rule are written down and committed before the first episode. Each experiment below names that commit.
  • Seeded. Every brain plays the same seeds, so it meets the same situations. No experiment reuses another's seeds.
  • Counted, with intervals. A result is tasks finished out of episodes, with a Wilson 95 % interval. Two brains differ only when their intervals do not overlap; otherwise we say there is no detectable difference.
  • How a model brain works. The rules handle routine moves, and the model decides when a person speaks to the agent or a choice matters. When a model's reply cannot be used, the rules decide instead. We count those moments as fallbacks.
  • Where it ran. The local models ran on one desktop PC, with 12 logical CPUs and 32 GB of memory. The cloud models were reached through a commercial model broker, at temperature 0, on a key with a hard spending cap.
  • Cost. We tracked what each run cost. This site does not publish money figures, so cost appears only as which predictions about it held.
  • What it does not measure. How fast a brain answers on your hardware, how it does outside these five small worlds, or with real people giving the instructions: here the instructions come from a fixed set of phrasings. A cloud model's time includes the network; a local model's time was measured on one shared, sometimes busy PC.
  • Names. The test worlds come from a private test game, so this page calls them World A to World E and names each task family by what it asks, not by the game's own words.

Counted from the result files of a private repository at the commit below, and checked row by row against the telemetry pack recomputed from the same files. Each figure names its experiment, its file, the rows or field it came from, and its pack row. Commit 158b75d116e3a4bb4dd6574a221d468f0b419390. Read the full record, with the rows each figure was counted from.