{
 "format": "odin-brain-bench",
 "version": 1,
 "source": {
  "repository": "private",
  "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
  "about": "Counted from the result files of a private repository at the commit below, and checked row by row against the telemetry pack recomputed from the same files. Each figure names its experiment, its file, the rows or field it came from, and its pack row.",
  "pack": {
   "format": "odin-telemetry",
   "version": 1,
   "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
   "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
   "builtAtCommit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
   "rule": "every figure carries its source; nothing without one is published"
  }
 },
 "labels": {
  "where": {
   "rules": "Rules",
   "local": "Local",
   "cloud": "Cloud"
  },
  "worlds": {
   "a": "World A",
   "b": "World B",
   "c": "World C",
   "d": "World D",
   "e": "World E"
  },
  "tasks": {
   "a": "Task family A: build",
   "b": "Task family B: a group challenge",
   "c": "Task family C: earn money",
   "d": "Task family D: help a newcomer",
   "e": "Task family E: a chain of goals",
   "f": "Task family F: survive a sudden event",
   "follow": "Follow a person's instruction"
  },
  "kinds": {
   "deal": "Deal with a threat",
   "hand": "Hand something over",
   "meet": "Go and meet someone",
   "buy": "Buy something for them"
  }
 },
 "experiments": [
  {
   "id": "EXP-W01",
   "title": "Which small local model makes the best agent brain?",
   "question": "Four brains on our own machine, on the whole task set: does a general model from another family play as well as our tuned 1 GB model, and how much does the tuning add?",
   "date": "2026-10-24",
   "design": {
    "tasks": 7,
    "worlds": 5,
    "seeds": {
     "from": 40,
     "to": 47,
     "count": 8
    },
    "brains": 4,
    "episodesPerBrain": 280,
    "episodes": 1120,
    "skippedRows": 30,
    "harness": [
     "h4"
    ],
    "source": {
     "experiment": "EXP-W01",
     "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
     "file": "results/results.jsonl",
     "lines": [
      [
       1,
       1150
      ]
     ]
    }
   },
   "prereg": {
    "commit": "3d7137a3",
    "sha256": "7191ed201db39f7a1e842fb80c11d7d5646cb949eb999a231335b4a350fbea83",
    "file": "PREREG.md",
    "experiment": "EXP-W01"
   },
   "comparators": [],
   "predictions": [
    {
     "id": "P1",
     "text": "Our tuned model finishes at least as many tasks as Llama 3.2 3B.",
     "verdict": "holds on the point estimate, no detectable difference",
     "detail": "{EXP-W01:tuned-1gb:of} against {EXP-W01:llama3.2-3b:of}, and the intervals overlap ({EXP-W01:tuned-1gb:ci} against {EXP-W01:llama3.2-3b:ci}). A general model of another family plays about as well, at twice the memory."
    },
    {
     "id": "P2",
     "text": "Both model brains beat the rules on following an instruction by at least 15 of 40.",
     "verdict": "fails",
     "detail": "Our tuned model clears it ({EXP-W01:tuned-1gb:task.follow} against the rules' {EXP-W01:rules:task.follow}). Llama 3.2 3B does not ({EXP-W01:llama3.2-3b:task.follow})."
    },
    {
     "id": "P3",
     "text": "The untuned Qwen2.5 1.5B scores below our tuned model, with intervals that do not overlap.",
     "verdict": "not supported",
     "detail": "{EXP-W01:qwen2.5-1.5b:of} against {EXP-W01:tuned-1gb:of}; the intervals overlap ({EXP-W01:qwen2.5-1.5b:ci} against {EXP-W01:tuned-1gb:ci})."
    }
   ],
   "unpredicted": [
    "Every model brain beats the rules overall, with intervals that do not overlap: the rules at {EXP-W01:rules:ci}, the lowest model brain at {EXP-W01:qwen2.5-1.5b:ci}.",
    "The tuning shows in the fallbacks, not the headline: the untuned base fell back to the rules {EXP-W01:qwen2.5-1.5b:fallbacks} times, our tuned model {EXP-W01:tuned-1gb:fallbacks} times."
   ],
   "caveats": [
    "Ran on harness h4, which showed a shop's price for one item while the seller charged for the whole stack, so some buys an agent believed it could afford were refused. The bug was found in EXP-W03's traces and fixed in h5, before EXP-W04. These rows are never compared with h5 rows as the same measurement.",
    "The local model server stopped serving the models several times mid-run. The bench recorded those episodes as skipped and the run was resumed with the identical command, so only the missing episodes ran. Every brain has its full count of episodes; the skipped rows hold no result and are not counted.",
    "The run paused several times when the PC was short of free memory, because other work was running on it. These were pauses, not aborts.",
    "The untuned model's fallbacks are spread over all five worlds and none sits near an outage, so they are the model's own.",
    "On following an instruction, our tuned model scored {EXP-W01:tuned-1gb:task.follow} and the untuned base {EXP-W01:qwen2.5-1.5b:task.follow}: the largest gap in the table, but with this few episodes the intervals still overlap. EXP-W03 was run to settle the question."
   ],
   "brains": [
    {
     "id": "tuned-1gb",
     "label": "Our tuned 1 GB model",
     "where": "local",
     "model": "Qwen2.5 1.5B, tuned on our own decisions.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 266,
     "rate": 0.95,
     "ci": [
      0.918,
      0.97
     ],
     "source": {
      "experiment": "EXP-W01",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        540,
        696
       ],
       [
        698,
        701
       ],
       [
        703,
        705
       ],
       [
        707,
        709
       ],
       [
        712,
        799
       ],
       [
        956,
        980
       ]
      ],
      "sha256": "025bc42e59bc119d45d83e7fa7167507e7813e57408daeea3c29b64720629c40",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/0"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/1",
       "results": {
        "file": ".ai/experiments/EXP-W01-small-local-brains/results/results.jsonl",
        "commit": "7357cf82",
        "sha256": "a63e18189cd699b91c9edd02a1487bfbe8348a3e7e7905c50c2d0477c34cb007",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 4,
     "tokensPerTask": 793.2,
     "modelMsPerTask": 5139.3,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           572,
           579
          ],
          [
           628,
           635
          ],
          [
           684,
           691
          ],
          [
           720,
           727
          ],
          [
           776,
           783
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           548,
           555
          ],
          [
           604,
           611
          ],
          [
           660,
           667
          ],
          [
           707,
           709
          ],
          [
           752,
           759
          ],
          [
           968,
           972
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           556,
           563
          ],
          [
           612,
           619
          ],
          [
           668,
           675
          ],
          [
           760,
           767
          ],
          [
           973,
           980
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           580,
           587
          ],
          [
           636,
           643
          ],
          [
           692,
           696
          ],
          [
           728,
           735
          ],
          [
           784,
           791
          ],
          [
           956,
           958
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           540,
           547
          ],
          [
           596,
           603
          ],
          [
           652,
           659
          ],
          [
           703,
           705
          ],
          [
           744,
           751
          ],
          [
           963,
           967
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           564,
           571
          ],
          [
           620,
           627
          ],
          [
           676,
           683
          ],
          [
           712,
           719
          ],
          [
           768,
           775
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 30,
        "rate": 0.75,
        "ci": [
         0.598,
         0.858
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           588,
           595
          ],
          [
           644,
           651
          ],
          [
           698,
           701
          ],
          [
           736,
           743
          ],
          [
           792,
           799
          ],
          [
           959,
           962
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 5,
        "rate": 0.5,
        "ci": [
         0.237,
         0.763
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           591,
           591
          ],
          [
           595,
           595
          ],
          [
           647,
           647
          ],
          [
           651,
           651
          ],
          [
           701,
           701
          ],
          [
           739,
           739
          ],
          [
           743,
           743
          ],
          [
           795,
           795
          ],
          [
           799,
           799
          ],
          [
           962,
           962
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           588,
           588
          ],
          [
           592,
           592
          ],
          [
           644,
           644
          ],
          [
           648,
           648
          ],
          [
           698,
           698
          ],
          [
           736,
           736
          ],
          [
           740,
           740
          ],
          [
           792,
           792
          ],
          [
           796,
           796
          ],
          [
           959,
           959
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 6,
        "rate": 0.6,
        "ci": [
         0.313,
         0.832
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           589,
           589
          ],
          [
           593,
           593
          ],
          [
           645,
           645
          ],
          [
           649,
           649
          ],
          [
           699,
           699
          ],
          [
           737,
           737
          ],
          [
           741,
           741
          ],
          [
           793,
           793
          ],
          [
           797,
           797
          ],
          [
           960,
           960
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 9,
        "rate": 0.9,
        "ci": [
         0.596,
         0.982
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           590,
           590
          ],
          [
           594,
           594
          ],
          [
           646,
           646
          ],
          [
           650,
           650
          ],
          [
           700,
           700
          ],
          [
           738,
           738
          ],
          [
           742,
           742
          ],
          [
           794,
           794
          ],
          [
           798,
           798
          ],
          [
           961,
           961
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on bench harness h4, whose shop price bug (unit, not stack) hit every brain; see EXP-W04"
    },
    {
     "id": "llama3.2-3b",
     "label": "Llama 3.2 3B",
     "where": "local",
     "model": "A general model from another family, about 2 GB.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 260,
     "rate": 0.9286,
     "ci": [
      0.892,
      0.953
     ],
     "source": {
      "experiment": "EXP-W01",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        800,
        909
       ],
       [
        981,
        1150
       ]
      ],
      "sha256": "025bc42e59bc119d45d83e7fa7167507e7813e57408daeea3c29b64720629c40",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/1"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/0",
       "results": {
        "file": ".ai/experiments/EXP-W01-small-local-brains/results/results.jsonl",
        "commit": "7357cf82",
        "sha256": "a63e18189cd699b91c9edd02a1487bfbe8348a3e7e7905c50c2d0477c34cb007",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 3,
     "tokensPerTask": 824.7,
     "modelMsPerTask": 4224.4,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           832,
           839
          ],
          [
           888,
           895
          ],
          [
           1015,
           1022
          ],
          [
           1071,
           1078
          ],
          [
           1127,
           1134
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           808,
           815
          ],
          [
           864,
           871
          ],
          [
           991,
           998
          ],
          [
           1047,
           1054
          ],
          [
           1103,
           1110
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           816,
           823
          ],
          [
           872,
           879
          ],
          [
           999,
           1006
          ],
          [
           1055,
           1062
          ],
          [
           1111,
           1118
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           840,
           847
          ],
          [
           896,
           903
          ],
          [
           1023,
           1030
          ],
          [
           1079,
           1086
          ],
          [
           1135,
           1142
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           800,
           807
          ],
          [
           856,
           863
          ],
          [
           983,
           990
          ],
          [
           1039,
           1046
          ],
          [
           1095,
           1102
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           824,
           831
          ],
          [
           880,
           887
          ],
          [
           1007,
           1014
          ],
          [
           1063,
           1070
          ],
          [
           1119,
           1126
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 24,
        "rate": 0.6,
        "ci": [
         0.446,
         0.737
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           848,
           855
          ],
          [
           904,
           909
          ],
          [
           981,
           982
          ],
          [
           1031,
           1038
          ],
          [
           1087,
           1094
          ],
          [
           1143,
           1150
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 5,
        "rate": 0.5,
        "ci": [
         0.237,
         0.763
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           851,
           851
          ],
          [
           855,
           855
          ],
          [
           907,
           907
          ],
          [
           982,
           982
          ],
          [
           1034,
           1034
          ],
          [
           1038,
           1038
          ],
          [
           1090,
           1090
          ],
          [
           1094,
           1094
          ],
          [
           1146,
           1146
          ],
          [
           1150,
           1150
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 2,
        "rate": 0.2,
        "ci": [
         0.057,
         0.51
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           848,
           848
          ],
          [
           852,
           852
          ],
          [
           904,
           904
          ],
          [
           908,
           908
          ],
          [
           1031,
           1031
          ],
          [
           1035,
           1035
          ],
          [
           1087,
           1087
          ],
          [
           1091,
           1091
          ],
          [
           1143,
           1143
          ],
          [
           1147,
           1147
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           849,
           849
          ],
          [
           853,
           853
          ],
          [
           905,
           905
          ],
          [
           909,
           909
          ],
          [
           1032,
           1032
          ],
          [
           1036,
           1036
          ],
          [
           1088,
           1088
          ],
          [
           1092,
           1092
          ],
          [
           1144,
           1144
          ],
          [
           1148,
           1148
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 7,
        "rate": 0.7,
        "ci": [
         0.397,
         0.892
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           850,
           850
          ],
          [
           854,
           854
          ],
          [
           906,
           906
          ],
          [
           981,
           981
          ],
          [
           1033,
           1033
          ],
          [
           1037,
           1037
          ],
          [
           1089,
           1089
          ],
          [
           1093,
           1093
          ],
          [
           1145,
           1145
          ],
          [
           1149,
           1149
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on bench harness h4, whose shop price bug (unit, not stack) hit every brain; see EXP-W04"
    },
    {
     "id": "qwen2.5-1.5b",
     "label": "Qwen2.5 1.5B, untuned",
     "where": "local",
     "model": "The base our model was tuned from.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 253,
     "rate": 0.9036,
     "ci": [
      0.863,
      0.933
     ],
     "source": {
      "experiment": "EXP-W01",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        281,
        536
       ],
       [
        932,
        955
       ]
      ],
      "sha256": "025bc42e59bc119d45d83e7fa7167507e7813e57408daeea3c29b64720629c40",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/2"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/2",
       "results": {
        "file": ".ai/experiments/EXP-W01-small-local-brains/results/results.jsonl",
        "commit": "7357cf82",
        "sha256": "a63e18189cd699b91c9edd02a1487bfbe8348a3e7e7905c50c2d0477c34cb007",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 90,
     "tokensPerTask": 979.7,
     "modelMsPerTask": 3729.2,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           313,
           320
          ],
          [
           369,
           376
          ],
          [
           425,
           432
          ],
          [
           481,
           488
          ],
          [
           932,
           939
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           289,
           296
          ],
          [
           345,
           352
          ],
          [
           401,
           408
          ],
          [
           457,
           464
          ],
          [
           513,
           520
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           297,
           304
          ],
          [
           353,
           360
          ],
          [
           409,
           416
          ],
          [
           465,
           472
          ],
          [
           521,
           528
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 39,
        "rate": 0.975,
        "ci": [
         0.871,
         0.996
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           321,
           328
          ],
          [
           377,
           384
          ],
          [
           433,
           440
          ],
          [
           489,
           496
          ],
          [
           940,
           947
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           281,
           288
          ],
          [
           337,
           344
          ],
          [
           393,
           400
          ],
          [
           449,
           456
          ],
          [
           505,
           512
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           305,
           312
          ],
          [
           361,
           368
          ],
          [
           417,
           424
          ],
          [
           473,
           480
          ],
          [
           529,
           536
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 18,
        "rate": 0.45,
        "ci": [
         0.307,
         0.602
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           329,
           336
          ],
          [
           385,
           392
          ],
          [
           441,
           448
          ],
          [
           497,
           504
          ],
          [
           948,
           955
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 3,
        "rate": 0.3,
        "ci": [
         0.108,
         0.603
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           332,
           332
          ],
          [
           336,
           336
          ],
          [
           388,
           388
          ],
          [
           392,
           392
          ],
          [
           444,
           444
          ],
          [
           448,
           448
          ],
          [
           500,
           500
          ],
          [
           504,
           504
          ],
          [
           951,
           951
          ],
          [
           955,
           955
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           329,
           329
          ],
          [
           333,
           333
          ],
          [
           385,
           385
          ],
          [
           389,
           389
          ],
          [
           441,
           441
          ],
          [
           445,
           445
          ],
          [
           497,
           497
          ],
          [
           501,
           501
          ],
          [
           948,
           948
          ],
          [
           952,
           952
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 1,
        "rate": 0.1,
        "ci": [
         0.018,
         0.404
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           330,
           330
          ],
          [
           334,
           334
          ],
          [
           386,
           386
          ],
          [
           390,
           390
          ],
          [
           442,
           442
          ],
          [
           446,
           446
          ],
          [
           498,
           498
          ],
          [
           502,
           502
          ],
          [
           949,
           949
          ],
          [
           953,
           953
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 4,
        "rate": 0.4,
        "ci": [
         0.168,
         0.687
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           331,
           331
          ],
          [
           335,
           335
          ],
          [
           387,
           387
          ],
          [
           391,
           391
          ],
          [
           443,
           443
          ],
          [
           447,
           447
          ],
          [
           499,
           499
          ],
          [
           503,
           503
          ],
          [
           950,
           950
          ],
          [
           954,
           954
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on bench harness h4, whose shop price bug (unit, not stack) hit every brain; see EXP-W04"
    },
    {
     "id": "rules",
     "label": "Hand-written rules",
     "where": "rules",
     "model": "The control: no model at all.",
     "brainVersion": null,
     "harness": "h4",
     "n": 280,
     "successes": 218,
     "rate": 0.7786,
     "ci": [
      0.726,
      0.823
     ],
     "source": {
      "experiment": "EXP-W01",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        1,
        280
       ]
      ],
      "sha256": "025bc42e59bc119d45d83e7fa7167507e7813e57408daeea3c29b64720629c40",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/3"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/3",
       "results": {
        "file": ".ai/experiments/EXP-W01-small-local-brains/results/results.jsonl",
        "commit": "7357cf82",
        "sha256": "a63e18189cd699b91c9edd02a1487bfbe8348a3e7e7905c50c2d0477c34cb007",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 0,
     "modelMsPerTask": 0,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           33,
           40
          ],
          [
           89,
           96
          ],
          [
           145,
           152
          ],
          [
           201,
           208
          ],
          [
           257,
           264
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           9,
           16
          ],
          [
           65,
           72
          ],
          [
           121,
           128
          ],
          [
           177,
           184
          ],
          [
           233,
           240
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           17,
           24
          ],
          [
           73,
           80
          ],
          [
           129,
           136
          ],
          [
           185,
           192
          ],
          [
           241,
           248
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 39,
        "rate": 0.975,
        "ci": [
         0.871,
         0.996
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           41,
           48
          ],
          [
           97,
           104
          ],
          [
           153,
           160
          ],
          [
           209,
           216
          ],
          [
           265,
           272
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           8
          ],
          [
           57,
           64
          ],
          [
           113,
           120
          ],
          [
           169,
           176
          ],
          [
           225,
           232
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 8,
        "rate": 0.2,
        "ci": [
         0.105,
         0.348
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           25,
           32
          ],
          [
           81,
           88
          ],
          [
           137,
           144
          ],
          [
           193,
           200
          ],
          [
           249,
           256
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 11,
        "rate": 0.275,
        "ci": [
         0.161,
         0.428
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           49,
           56
          ],
          [
           105,
           112
          ],
          [
           161,
           168
          ],
          [
           217,
           224
          ],
          [
           273,
           280
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 4,
        "rate": 0.4,
        "ci": [
         0.168,
         0.687
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           52,
           52
          ],
          [
           56,
           56
          ],
          [
           108,
           108
          ],
          [
           112,
           112
          ],
          [
           164,
           164
          ],
          [
           168,
           168
          ],
          [
           220,
           220
          ],
          [
           224,
           224
          ],
          [
           276,
           276
          ],
          [
           280,
           280
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 2,
        "rate": 0.2,
        "ci": [
         0.057,
         0.51
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           49,
           49
          ],
          [
           53,
           53
          ],
          [
           105,
           105
          ],
          [
           109,
           109
          ],
          [
           161,
           161
          ],
          [
           165,
           165
          ],
          [
           217,
           217
          ],
          [
           221,
           221
          ],
          [
           273,
           273
          ],
          [
           277,
           277
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 0,
        "rate": 0,
        "ci": [
         0,
         0.278
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           50,
           50
          ],
          [
           54,
           54
          ],
          [
           106,
           106
          ],
          [
           110,
           110
          ],
          [
           162,
           162
          ],
          [
           166,
           166
          ],
          [
           218,
           218
          ],
          [
           222,
           222
          ],
          [
           274,
           274
          ],
          [
           278,
           278
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 5,
        "rate": 0.5,
        "ci": [
         0.237,
         0.763
        ],
        "source": {
         "experiment": "EXP-W01",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           51,
           51
          ],
          [
           55,
           55
          ],
          [
           107,
           107
          ],
          [
           111,
           111
          ],
          [
           163,
           163
          ],
          [
           167,
           167
          ],
          [
           219,
           219
          ],
          [
           223,
           223
          ],
          [
           275,
           275
          ],
          [
           279,
           279
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on bench harness h4, whose shop price bug (unit, not stack) hit every brain; see EXP-W04"
    }
   ]
  },
  {
   "id": "EXP-W02",
   "title": "Do cloud models make better agent brains than our tuned local model?",
   "question": "Five cloud models on the same seeds, worlds and tasks as EXP-W01: does any of them beat our tuned 1 GB model?",
   "date": "2026-10-24",
   "design": {
    "tasks": 7,
    "worlds": 5,
    "seeds": {
     "from": 40,
     "to": 47,
     "count": 8
    },
    "brains": 5,
    "episodesPerBrain": 280,
    "episodes": 1400,
    "skippedRows": 0,
    "harness": [
     "h4"
    ],
    "source": {
     "experiment": "EXP-W02",
     "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
     "file": "results/results.jsonl",
     "lines": [
      [
       1,
       1400
      ]
     ]
    }
   },
   "prereg": {
    "commit": "c3fa1656",
    "sha256": "09d4adcd58c5216bcf8063fc78766e557c1d3877340c901d38d3ec5d6cb672e4",
    "file": "PREREG.md",
    "experiment": "EXP-W02"
   },
   "comparators": [
    "EXP-W01"
   ],
   "predictions": [
    {
     "id": "P1",
     "text": "No cloud model beats our tuned local model overall with intervals that do not overlap.",
     "verdict": "holds",
     "detail": "The best, Mistral Small 3.2 at {EXP-W02:mistral-small-3.2:ci}, still overlaps our tuned model at {EXP-W01:tuned-1gb:ci}."
    },
    {
     "id": "P2",
     "text": "At least one cloud model beats our tuned model on following an instruction by 5 of 40 or more.",
     "verdict": "holds",
     "detail": "Mistral Small 3.2 {EXP-W02:mistral-small-3.2:task.follow}, Llama 3.3 70B {EXP-W02:llama-3.3-70b:task.follow}, GPT-4o mini {EXP-W02:gpt-4o-mini:task.follow} and Claude Haiku 4.5 {EXP-W02:claude-haiku-4.5:task.follow}, against our model's {EXP-W01:tuned-1gb:task.follow}. None of these is a difference by the interval rule yet."
    },
    {
     "id": "P3",
     "text": "Every cloud model stays under the cost per successful task set before the run.",
     "verdict": "holds",
     "detail": "By a wide margin. Cost was tracked, but this site does not publish money figures."
    },
    {
     "id": "P4",
     "text": "All five beat the rules with intervals that do not overlap.",
     "verdict": "holds",
     "detail": "The lowest, Gemini 2.5 Flash at {EXP-W02:gemini-2.5-flash:ci}, is above the rules at {EXP-W01:rules:ci}."
    }
   ],
   "unpredicted": [
    "Gemini 2.5 Flash fell back to the rules {EXP-W02:gemini-2.5-flash:fallbacks} times, far more than any other model. Its replies did not fit the bench's strict reply format. That is a fit problem between this model and our format, not a verdict on the model in general, and it is why Gemini trails on following an instruction."
   ],
   "caveats": [
    "Ran on harness h4, with the shop-price bug described under EXP-W01. These rows are never compared with h5 rows as the same measurement.",
    "A second bench bug: the bench's own cost column read zero for every cloud model in this run, so its leaderboard labelled them free and local. They were not free. The bug was fixed after the run; the result files are unchanged.",
    "The cloud models are compared with EXP-W01's local rows: the same seeds, worlds, tasks, harness and brain version, measured earlier the same day and not re-run.",
    "Every cloud model ran all its episodes, with no service errors, skips or spending-cap hits."
   ],
   "brains": [
    {
     "id": "mistral-small-3.2",
     "label": "Mistral Small 3.2",
     "where": "cloud",
     "model": "Mistral Small 3.2, 24B, instruct.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 274,
     "rate": 0.9786,
     "ci": [
      0.954,
      0.99
     ],
     "source": {
      "experiment": "EXP-W02",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        1121,
        1400
       ]
      ],
      "sha256": "9f93568f92829ae7d618d7ba3047f78f5a44fc40db1d11e66726d6a9be276d82",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/0"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/7",
       "results": {
        "file": ".ai/experiments/EXP-W02-vendor-brains/results/results.jsonl",
        "commit": "63bc371f",
        "sha256": "1d24611817d02f97ee3a0aa8c84110cc2c3008d10faf498a940d61a1776cd9ac",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 5,
     "tokensPerTask": 586.2,
     "modelMsPerTask": 1244.9,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1153,
           1160
          ],
          [
           1209,
           1216
          ],
          [
           1265,
           1272
          ],
          [
           1321,
           1328
          ],
          [
           1377,
           1384
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1129,
           1136
          ],
          [
           1185,
           1192
          ],
          [
           1241,
           1248
          ],
          [
           1297,
           1304
          ],
          [
           1353,
           1360
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1137,
           1144
          ],
          [
           1193,
           1200
          ],
          [
           1249,
           1256
          ],
          [
           1305,
           1312
          ],
          [
           1361,
           1368
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1161,
           1168
          ],
          [
           1217,
           1224
          ],
          [
           1273,
           1280
          ],
          [
           1329,
           1336
          ],
          [
           1385,
           1392
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1121,
           1128
          ],
          [
           1177,
           1184
          ],
          [
           1233,
           1240
          ],
          [
           1289,
           1296
          ],
          [
           1345,
           1352
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1145,
           1152
          ],
          [
           1201,
           1208
          ],
          [
           1257,
           1264
          ],
          [
           1313,
           1320
          ],
          [
           1369,
           1376
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 38,
        "rate": 0.95,
        "ci": [
         0.835,
         0.986
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1169,
           1176
          ],
          [
           1225,
           1232
          ],
          [
           1281,
           1288
          ],
          [
           1337,
           1344
          ],
          [
           1393,
           1400
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 8,
        "rate": 0.8,
        "ci": [
         0.49,
         0.943
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1172,
           1172
          ],
          [
           1176,
           1176
          ],
          [
           1228,
           1228
          ],
          [
           1232,
           1232
          ],
          [
           1284,
           1284
          ],
          [
           1288,
           1288
          ],
          [
           1340,
           1340
          ],
          [
           1344,
           1344
          ],
          [
           1396,
           1396
          ],
          [
           1400,
           1400
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1169,
           1169
          ],
          [
           1173,
           1173
          ],
          [
           1225,
           1225
          ],
          [
           1229,
           1229
          ],
          [
           1281,
           1281
          ],
          [
           1285,
           1285
          ],
          [
           1337,
           1337
          ],
          [
           1341,
           1341
          ],
          [
           1393,
           1393
          ],
          [
           1397,
           1397
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1170,
           1170
          ],
          [
           1174,
           1174
          ],
          [
           1226,
           1226
          ],
          [
           1230,
           1230
          ],
          [
           1282,
           1282
          ],
          [
           1286,
           1286
          ],
          [
           1338,
           1338
          ],
          [
           1342,
           1342
          ],
          [
           1394,
           1394
          ],
          [
           1398,
           1398
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1171,
           1171
          ],
          [
           1175,
           1175
          ],
          [
           1227,
           1227
          ],
          [
           1231,
           1231
          ],
          [
           1283,
           1283
          ],
          [
           1287,
           1287
          ],
          [
           1339,
           1339
          ],
          [
           1343,
           1343
          ],
          [
           1395,
           1395
          ],
          [
           1399,
           1399
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "gpt-4o-mini",
     "label": "GPT-4o mini",
     "where": "cloud",
     "model": "GPT-4o mini.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 273,
     "rate": 0.975,
     "ci": [
      0.949,
      0.988
     ],
     "source": {
      "experiment": "EXP-W02",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        1,
        280
       ]
      ],
      "sha256": "9f93568f92829ae7d618d7ba3047f78f5a44fc40db1d11e66726d6a9be276d82",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/2"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/8",
       "results": {
        "file": ".ai/experiments/EXP-W02-vendor-brains/results/results.jsonl",
        "commit": "63bc371f",
        "sha256": "1d24611817d02f97ee3a0aa8c84110cc2c3008d10faf498a940d61a1776cd9ac",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 8,
     "tokensPerTask": 566.5,
     "modelMsPerTask": 814.2,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           33,
           40
          ],
          [
           89,
           96
          ],
          [
           145,
           152
          ],
          [
           201,
           208
          ],
          [
           257,
           264
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           9,
           16
          ],
          [
           65,
           72
          ],
          [
           121,
           128
          ],
          [
           177,
           184
          ],
          [
           233,
           240
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           17,
           24
          ],
          [
           73,
           80
          ],
          [
           129,
           136
          ],
          [
           185,
           192
          ],
          [
           241,
           248
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           41,
           48
          ],
          [
           97,
           104
          ],
          [
           153,
           160
          ],
          [
           209,
           216
          ],
          [
           265,
           272
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           8
          ],
          [
           57,
           64
          ],
          [
           113,
           120
          ],
          [
           169,
           176
          ],
          [
           225,
           232
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           25,
           32
          ],
          [
           81,
           88
          ],
          [
           137,
           144
          ],
          [
           193,
           200
          ],
          [
           249,
           256
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 37,
        "rate": 0.925,
        "ci": [
         0.801,
         0.974
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           49,
           56
          ],
          [
           105,
           112
          ],
          [
           161,
           168
          ],
          [
           217,
           224
          ],
          [
           273,
           280
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 7,
        "rate": 0.7,
        "ci": [
         0.397,
         0.892
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           52,
           52
          ],
          [
           56,
           56
          ],
          [
           108,
           108
          ],
          [
           112,
           112
          ],
          [
           164,
           164
          ],
          [
           168,
           168
          ],
          [
           220,
           220
          ],
          [
           224,
           224
          ],
          [
           276,
           276
          ],
          [
           280,
           280
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           49,
           49
          ],
          [
           53,
           53
          ],
          [
           105,
           105
          ],
          [
           109,
           109
          ],
          [
           161,
           161
          ],
          [
           165,
           165
          ],
          [
           217,
           217
          ],
          [
           221,
           221
          ],
          [
           273,
           273
          ],
          [
           277,
           277
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           50,
           50
          ],
          [
           54,
           54
          ],
          [
           106,
           106
          ],
          [
           110,
           110
          ],
          [
           162,
           162
          ],
          [
           166,
           166
          ],
          [
           218,
           218
          ],
          [
           222,
           222
          ],
          [
           274,
           274
          ],
          [
           278,
           278
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           51,
           51
          ],
          [
           55,
           55
          ],
          [
           107,
           107
          ],
          [
           111,
           111
          ],
          [
           163,
           163
          ],
          [
           167,
           167
          ],
          [
           219,
           219
          ],
          [
           223,
           223
          ],
          [
           275,
           275
          ],
          [
           279,
           279
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "llama-3.3-70b",
     "label": "Llama 3.3 70B",
     "where": "cloud",
     "model": "Llama 3.3 70B, instruct.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 273,
     "rate": 0.975,
     "ci": [
      0.949,
      0.988
     ],
     "source": {
      "experiment": "EXP-W02",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        841,
        1120
       ]
      ],
      "sha256": "9f93568f92829ae7d618d7ba3047f78f5a44fc40db1d11e66726d6a9be276d82",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/1"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/6",
       "results": {
        "file": ".ai/experiments/EXP-W02-vendor-brains/results/results.jsonl",
        "commit": "63bc371f",
        "sha256": "1d24611817d02f97ee3a0aa8c84110cc2c3008d10faf498a940d61a1776cd9ac",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 1,
     "tokensPerTask": 554.9,
     "modelMsPerTask": 1404.3,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           873,
           880
          ],
          [
           929,
           936
          ],
          [
           985,
           992
          ],
          [
           1041,
           1048
          ],
          [
           1097,
           1104
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           849,
           856
          ],
          [
           905,
           912
          ],
          [
           961,
           968
          ],
          [
           1017,
           1024
          ],
          [
           1073,
           1080
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           857,
           864
          ],
          [
           913,
           920
          ],
          [
           969,
           976
          ],
          [
           1025,
           1032
          ],
          [
           1081,
           1088
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           881,
           888
          ],
          [
           937,
           944
          ],
          [
           993,
           1000
          ],
          [
           1049,
           1056
          ],
          [
           1105,
           1112
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           841,
           848
          ],
          [
           897,
           904
          ],
          [
           953,
           960
          ],
          [
           1009,
           1016
          ],
          [
           1065,
           1072
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           865,
           872
          ],
          [
           921,
           928
          ],
          [
           977,
           984
          ],
          [
           1033,
           1040
          ],
          [
           1089,
           1096
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 37,
        "rate": 0.925,
        "ci": [
         0.801,
         0.974
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           889,
           896
          ],
          [
           945,
           952
          ],
          [
           1001,
           1008
          ],
          [
           1057,
           1064
          ],
          [
           1113,
           1120
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 7,
        "rate": 0.7,
        "ci": [
         0.397,
         0.892
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           892,
           892
          ],
          [
           896,
           896
          ],
          [
           948,
           948
          ],
          [
           952,
           952
          ],
          [
           1004,
           1004
          ],
          [
           1008,
           1008
          ],
          [
           1060,
           1060
          ],
          [
           1064,
           1064
          ],
          [
           1116,
           1116
          ],
          [
           1120,
           1120
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           889,
           889
          ],
          [
           893,
           893
          ],
          [
           945,
           945
          ],
          [
           949,
           949
          ],
          [
           1001,
           1001
          ],
          [
           1005,
           1005
          ],
          [
           1057,
           1057
          ],
          [
           1061,
           1061
          ],
          [
           1113,
           1113
          ],
          [
           1117,
           1117
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           890,
           890
          ],
          [
           894,
           894
          ],
          [
           946,
           946
          ],
          [
           950,
           950
          ],
          [
           1002,
           1002
          ],
          [
           1006,
           1006
          ],
          [
           1058,
           1058
          ],
          [
           1062,
           1062
          ],
          [
           1114,
           1114
          ],
          [
           1118,
           1118
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           891,
           891
          ],
          [
           895,
           895
          ],
          [
           947,
           947
          ],
          [
           951,
           951
          ],
          [
           1003,
           1003
          ],
          [
           1007,
           1007
          ],
          [
           1059,
           1059
          ],
          [
           1063,
           1063
          ],
          [
           1115,
           1115
          ],
          [
           1119,
           1119
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "claude-haiku-4.5",
     "label": "Claude Haiku 4.5",
     "where": "cloud",
     "model": "Claude Haiku 4.5.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 272,
     "rate": 0.9714,
     "ci": [
      0.945,
      0.985
     ],
     "source": {
      "experiment": "EXP-W02",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        281,
        560
       ]
      ],
      "sha256": "9f93568f92829ae7d618d7ba3047f78f5a44fc40db1d11e66726d6a9be276d82",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/3"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/4",
       "results": {
        "file": ".ai/experiments/EXP-W02-vendor-brains/results/results.jsonl",
        "commit": "63bc371f",
        "sha256": "1d24611817d02f97ee3a0aa8c84110cc2c3008d10faf498a940d61a1776cd9ac",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 675.1,
     "modelMsPerTask": 1052.4,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           313,
           320
          ],
          [
           369,
           376
          ],
          [
           425,
           432
          ],
          [
           481,
           488
          ],
          [
           537,
           544
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           289,
           296
          ],
          [
           345,
           352
          ],
          [
           401,
           408
          ],
          [
           457,
           464
          ],
          [
           513,
           520
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           297,
           304
          ],
          [
           353,
           360
          ],
          [
           409,
           416
          ],
          [
           465,
           472
          ],
          [
           521,
           528
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           321,
           328
          ],
          [
           377,
           384
          ],
          [
           433,
           440
          ],
          [
           489,
           496
          ],
          [
           545,
           552
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           281,
           288
          ],
          [
           337,
           344
          ],
          [
           393,
           400
          ],
          [
           449,
           456
          ],
          [
           505,
           512
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           305,
           312
          ],
          [
           361,
           368
          ],
          [
           417,
           424
          ],
          [
           473,
           480
          ],
          [
           529,
           536
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           329,
           336
          ],
          [
           385,
           392
          ],
          [
           441,
           448
          ],
          [
           497,
           504
          ],
          [
           553,
           560
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 8,
        "rate": 0.8,
        "ci": [
         0.49,
         0.943
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           332,
           332
          ],
          [
           336,
           336
          ],
          [
           388,
           388
          ],
          [
           392,
           392
          ],
          [
           444,
           444
          ],
          [
           448,
           448
          ],
          [
           500,
           500
          ],
          [
           504,
           504
          ],
          [
           556,
           556
          ],
          [
           560,
           560
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           329,
           329
          ],
          [
           333,
           333
          ],
          [
           385,
           385
          ],
          [
           389,
           389
          ],
          [
           441,
           441
          ],
          [
           445,
           445
          ],
          [
           497,
           497
          ],
          [
           501,
           501
          ],
          [
           553,
           553
          ],
          [
           557,
           557
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 10,
        "rate": 1,
        "ci": [
         0.722,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           330,
           330
          ],
          [
           334,
           334
          ],
          [
           386,
           386
          ],
          [
           390,
           390
          ],
          [
           442,
           442
          ],
          [
           446,
           446
          ],
          [
           498,
           498
          ],
          [
           502,
           502
          ],
          [
           554,
           554
          ],
          [
           558,
           558
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 8,
        "rate": 0.8,
        "ci": [
         0.49,
         0.943
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           331,
           331
          ],
          [
           335,
           335
          ],
          [
           387,
           387
          ],
          [
           391,
           391
          ],
          [
           443,
           443
          ],
          [
           447,
           447
          ],
          [
           499,
           499
          ],
          [
           503,
           503
          ],
          [
           555,
           555
          ],
          [
           559,
           559
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "gemini-2.5-flash",
     "label": "Gemini 2.5 Flash",
     "where": "cloud",
     "model": "Gemini 2.5 Flash.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 280,
     "successes": 259,
     "rate": 0.925,
     "ci": [
      0.888,
      0.95
     ],
     "source": {
      "experiment": "EXP-W02",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        561,
        840
       ]
      ],
      "sha256": "9f93568f92829ae7d618d7ba3047f78f5a44fc40db1d11e66726d6a9be276d82",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/4"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/5",
       "results": {
        "file": ".ai/experiments/EXP-W02-vendor-brains/results/results.jsonl",
        "commit": "63bc371f",
        "sha256": "1d24611817d02f97ee3a0aa8c84110cc2c3008d10faf498a940d61a1776cd9ac",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 158,
     "tokensPerTask": 341.5,
     "modelMsPerTask": 281,
     "breakdown": {
      "task": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           593,
           600
          ],
          [
           649,
           656
          ],
          [
           705,
           712
          ],
          [
           761,
           768
          ],
          [
           817,
           824
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           569,
           576
          ],
          [
           625,
           632
          ],
          [
           681,
           688
          ],
          [
           737,
           744
          ],
          [
           793,
           800
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           577,
           584
          ],
          [
           633,
           640
          ],
          [
           689,
           696
          ],
          [
           745,
           752
          ],
          [
           801,
           808
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 39,
        "rate": 0.975,
        "ci": [
         0.871,
         0.996
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           601,
           608
          ],
          [
           657,
           664
          ],
          [
           713,
           720
          ],
          [
           769,
           776
          ],
          [
           825,
           832
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           561,
           568
          ],
          [
           617,
           624
          ],
          [
           673,
           680
          ],
          [
           729,
           736
          ],
          [
           785,
           792
          ]
         ]
        }
       },
       "f": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           585,
           592
          ],
          [
           641,
           648
          ],
          [
           697,
           704
          ],
          [
           753,
           760
          ],
          [
           809,
           816
          ]
         ]
        }
       },
       "follow": {
        "n": 40,
        "successes": 24,
        "rate": 0.6,
        "ci": [
         0.446,
         0.737
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           609,
           616
          ],
          [
           665,
           672
          ],
          [
           721,
           728
          ],
          [
           777,
           784
          ],
          [
           833,
           840
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 10,
        "successes": 7,
        "rate": 0.7,
        "ci": [
         0.397,
         0.892
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           612,
           612
          ],
          [
           616,
           616
          ],
          [
           668,
           668
          ],
          [
           672,
           672
          ],
          [
           724,
           724
          ],
          [
           728,
           728
          ],
          [
           780,
           780
          ],
          [
           784,
           784
          ],
          [
           836,
           836
          ],
          [
           840,
           840
          ]
         ]
        }
       },
       "deal": {
        "n": 10,
        "successes": 6,
        "rate": 0.6,
        "ci": [
         0.313,
         0.832
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           609,
           609
          ],
          [
           613,
           613
          ],
          [
           665,
           665
          ],
          [
           669,
           669
          ],
          [
           721,
           721
          ],
          [
           725,
           725
          ],
          [
           777,
           777
          ],
          [
           781,
           781
          ],
          [
           833,
           833
          ],
          [
           837,
           837
          ]
         ]
        }
       },
       "hand": {
        "n": 10,
        "successes": 5,
        "rate": 0.5,
        "ci": [
         0.237,
         0.763
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           610,
           610
          ],
          [
           614,
           614
          ],
          [
           666,
           666
          ],
          [
           670,
           670
          ],
          [
           722,
           722
          ],
          [
           726,
           726
          ],
          [
           778,
           778
          ],
          [
           782,
           782
          ],
          [
           834,
           834
          ],
          [
           838,
           838
          ]
         ]
        }
       },
       "meet": {
        "n": 10,
        "successes": 6,
        "rate": 0.6,
        "ci": [
         0.313,
         0.832
        ],
        "source": {
         "experiment": "EXP-W02",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           611,
           611
          ],
          [
           615,
           615
          ],
          [
           667,
           667
          ],
          [
           671,
           671
          ],
          [
           723,
           723
          ],
          [
           727,
           727
          ],
          [
           779,
           779
          ],
          [
           783,
           783
          ],
          [
           835,
           835
          ],
          [
           839,
           839
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    }
   ]
  },
  {
   "id": "EXP-W03",
   "title": "Are the cloud models really better at following a person's instruction?",
   "question": "EXP-W02's largest gap, powered: only the instruction task, on five times as many seeds, for our tuned model, the two best-value cloud models and the rules.",
   "date": "2026-10-24",
   "design": {
    "tasks": 1,
    "worlds": 5,
    "seeds": {
     "from": 100,
     "to": 139,
     "count": 40
    },
    "brains": 4,
    "episodesPerBrain": 200,
    "episodes": 800,
    "skippedRows": 0,
    "harness": [
     "h4"
    ],
    "source": {
     "experiment": "EXP-W03",
     "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
     "file": "results/results.jsonl",
     "lines": [
      [
       1,
       800
      ]
     ]
    }
   },
   "prereg": {
    "commit": "1f75de87",
    "sha256": "2357a516ac3b9487e0a565bc43e4b0b79dd19652e886334eaeb2b881b7480a4c",
    "file": "PREREG.md",
    "experiment": "EXP-W03"
   },
   "comparators": [],
   "predictions": [
    {
     "id": "P1",
     "text": "Mistral Small 3.2 beats our tuned local model on following an instruction, with intervals that do not overlap.",
     "verdict": "holds",
     "detail": "{EXP-W03:mistral-small-3.2:ci} against {EXP-W03:tuned-1gb:ci}."
    },
    {
     "id": "P2",
     "text": "GPT-4o mini does too.",
     "verdict": "holds",
     "detail": "{EXP-W03:gpt-4o-mini:ci} against {EXP-W03:tuned-1gb:ci}."
    },
    {
     "id": "P3",
     "text": "Every model brain beats the rules, with intervals that do not overlap.",
     "verdict": "holds",
     "detail": "The lowest model, ours at {EXP-W03:tuned-1gb:ci}, against the rules at {EXP-W03:rules:ci}."
    },
    {
     "id": "P4",
     "text": "The whole run stays under the cost ceiling set before it.",
     "verdict": "holds",
     "detail": "By the real spend on the key, which matches the bench's own estimate. Cost was tracked, but this site does not publish money figures."
    }
   ],
   "unpredicted": [
    "Mistral Small 3.2 against GPT-4o mini shows no detectable difference: {EXP-W03:mistral-small-3.2:ci} against {EXP-W03:gpt-4o-mini:ci}.",
    "Where our model loses is mostly one world: World A, {EXP-W03:tuned-1gb:world.a} against Mistral Small 3.2's {EXP-W03:mistral-small-3.2:world.a}. World B is hard for every brain."
   ],
   "caveats": [
    "Ran on harness h4, with the shop-price bug described under EXP-W01. Reading this experiment's failed episodes is how the bug was found. EXP-W04 asks the same question after the fix, on new seeds; the two are never compared as one measurement.",
    "No skips and no service errors, and the bench's cost column now prices the cloud models."
   ],
   "brains": [
    {
     "id": "mistral-small-3.2",
     "label": "Mistral Small 3.2",
     "where": "cloud",
     "model": "Mistral Small 3.2, 24B, instruct.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 200,
     "successes": 190,
     "rate": 0.95,
     "ci": [
      0.91,
      0.973
     ],
     "source": {
      "experiment": "EXP-W03",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        401,
        600
       ]
      ],
      "sha256": "198ddb47486ad4743f0fc85bacfa6a1696a086b816d6a6cb2c6983cad63a4648",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/0"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/10",
       "results": {
        "file": ".ai/experiments/EXP-W03-follow-instruction/results/results.jsonl",
        "commit": "f2e5759b",
        "sha256": "c082a5e358f2efb2233c8cbd2b4b4a480689f5ab824a087d8c80262d2349e383",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 8,
     "tokensPerTask": 2025.6,
     "modelMsPerTask": 2424.8,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           401,
           440
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 30,
        "rate": 0.75,
        "ci": [
         0.598,
         0.858
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           441,
           480
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           481,
           520
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           521,
           560
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           561,
           600
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 40,
        "rate": 0.8,
        "ci": [
         0.67,
         0.888
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           404,
           404
          ],
          [
           408,
           408
          ],
          [
           412,
           412
          ],
          [
           416,
           416
          ],
          [
           420,
           420
          ],
          [
           424,
           424
          ],
          [
           428,
           428
          ],
          [
           432,
           432
          ],
          [
           436,
           436
          ],
          [
           440,
           440
          ],
          [
           444,
           444
          ],
          [
           448,
           448
          ],
          [
           452,
           452
          ],
          [
           456,
           456
          ],
          [
           460,
           460
          ],
          [
           464,
           464
          ],
          [
           468,
           468
          ],
          [
           472,
           472
          ],
          [
           476,
           476
          ],
          [
           480,
           480
          ],
          [
           484,
           484
          ],
          [
           488,
           488
          ],
          [
           492,
           492
          ],
          [
           496,
           496
          ],
          [
           500,
           500
          ],
          [
           504,
           504
          ],
          [
           508,
           508
          ],
          [
           512,
           512
          ],
          [
           516,
           516
          ],
          [
           520,
           520
          ],
          [
           524,
           524
          ],
          [
           528,
           528
          ],
          [
           532,
           532
          ],
          [
           536,
           536
          ],
          [
           540,
           540
          ],
          [
           544,
           544
          ],
          [
           548,
           548
          ],
          [
           552,
           552
          ],
          [
           556,
           556
          ],
          [
           560,
           560
          ],
          [
           564,
           564
          ],
          [
           568,
           568
          ],
          [
           572,
           572
          ],
          [
           576,
           576
          ],
          [
           580,
           580
          ],
          [
           584,
           584
          ],
          [
           588,
           588
          ],
          [
           592,
           592
          ],
          [
           596,
           596
          ],
          [
           600,
           600
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           401,
           401
          ],
          [
           405,
           405
          ],
          [
           409,
           409
          ],
          [
           413,
           413
          ],
          [
           417,
           417
          ],
          [
           421,
           421
          ],
          [
           425,
           425
          ],
          [
           429,
           429
          ],
          [
           433,
           433
          ],
          [
           437,
           437
          ],
          [
           441,
           441
          ],
          [
           445,
           445
          ],
          [
           449,
           449
          ],
          [
           453,
           453
          ],
          [
           457,
           457
          ],
          [
           461,
           461
          ],
          [
           465,
           465
          ],
          [
           469,
           469
          ],
          [
           473,
           473
          ],
          [
           477,
           477
          ],
          [
           481,
           481
          ],
          [
           485,
           485
          ],
          [
           489,
           489
          ],
          [
           493,
           493
          ],
          [
           497,
           497
          ],
          [
           501,
           501
          ],
          [
           505,
           505
          ],
          [
           509,
           509
          ],
          [
           513,
           513
          ],
          [
           517,
           517
          ],
          [
           521,
           521
          ],
          [
           525,
           525
          ],
          [
           529,
           529
          ],
          [
           533,
           533
          ],
          [
           537,
           537
          ],
          [
           541,
           541
          ],
          [
           545,
           545
          ],
          [
           549,
           549
          ],
          [
           553,
           553
          ],
          [
           557,
           557
          ],
          [
           561,
           561
          ],
          [
           565,
           565
          ],
          [
           569,
           569
          ],
          [
           573,
           573
          ],
          [
           577,
           577
          ],
          [
           581,
           581
          ],
          [
           585,
           585
          ],
          [
           589,
           589
          ],
          [
           593,
           593
          ],
          [
           597,
           597
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           402,
           402
          ],
          [
           406,
           406
          ],
          [
           410,
           410
          ],
          [
           414,
           414
          ],
          [
           418,
           418
          ],
          [
           422,
           422
          ],
          [
           426,
           426
          ],
          [
           430,
           430
          ],
          [
           434,
           434
          ],
          [
           438,
           438
          ],
          [
           442,
           442
          ],
          [
           446,
           446
          ],
          [
           450,
           450
          ],
          [
           454,
           454
          ],
          [
           458,
           458
          ],
          [
           462,
           462
          ],
          [
           466,
           466
          ],
          [
           470,
           470
          ],
          [
           474,
           474
          ],
          [
           478,
           478
          ],
          [
           482,
           482
          ],
          [
           486,
           486
          ],
          [
           490,
           490
          ],
          [
           494,
           494
          ],
          [
           498,
           498
          ],
          [
           502,
           502
          ],
          [
           506,
           506
          ],
          [
           510,
           510
          ],
          [
           514,
           514
          ],
          [
           518,
           518
          ],
          [
           522,
           522
          ],
          [
           526,
           526
          ],
          [
           530,
           530
          ],
          [
           534,
           534
          ],
          [
           538,
           538
          ],
          [
           542,
           542
          ],
          [
           546,
           546
          ],
          [
           550,
           550
          ],
          [
           554,
           554
          ],
          [
           558,
           558
          ],
          [
           562,
           562
          ],
          [
           566,
           566
          ],
          [
           570,
           570
          ],
          [
           574,
           574
          ],
          [
           578,
           578
          ],
          [
           582,
           582
          ],
          [
           586,
           586
          ],
          [
           590,
           590
          ],
          [
           594,
           594
          ],
          [
           598,
           598
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           403,
           403
          ],
          [
           407,
           407
          ],
          [
           411,
           411
          ],
          [
           415,
           415
          ],
          [
           419,
           419
          ],
          [
           423,
           423
          ],
          [
           427,
           427
          ],
          [
           431,
           431
          ],
          [
           435,
           435
          ],
          [
           439,
           439
          ],
          [
           443,
           443
          ],
          [
           447,
           447
          ],
          [
           451,
           451
          ],
          [
           455,
           455
          ],
          [
           459,
           459
          ],
          [
           463,
           463
          ],
          [
           467,
           467
          ],
          [
           471,
           471
          ],
          [
           475,
           475
          ],
          [
           479,
           479
          ],
          [
           483,
           483
          ],
          [
           487,
           487
          ],
          [
           491,
           491
          ],
          [
           495,
           495
          ],
          [
           499,
           499
          ],
          [
           503,
           503
          ],
          [
           507,
           507
          ],
          [
           511,
           511
          ],
          [
           515,
           515
          ],
          [
           519,
           519
          ],
          [
           523,
           523
          ],
          [
           527,
           527
          ],
          [
           531,
           531
          ],
          [
           535,
           535
          ],
          [
           539,
           539
          ],
          [
           543,
           543
          ],
          [
           547,
           547
          ],
          [
           551,
           551
          ],
          [
           555,
           555
          ],
          [
           559,
           559
          ],
          [
           563,
           563
          ],
          [
           567,
           567
          ],
          [
           571,
           571
          ],
          [
           575,
           575
          ],
          [
           579,
           579
          ],
          [
           583,
           583
          ],
          [
           587,
           587
          ],
          [
           591,
           591
          ],
          [
           595,
           595
          ],
          [
           599,
           599
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "gpt-4o-mini",
     "label": "GPT-4o mini",
     "where": "cloud",
     "model": "GPT-4o mini.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 200,
     "successes": 177,
     "rate": 0.885,
     "ci": [
      0.833,
      0.922
     ],
     "source": {
      "experiment": "EXP-W03",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        601,
        800
       ]
      ],
      "sha256": "198ddb47486ad4743f0fc85bacfa6a1696a086b816d6a6cb2c6983cad63a4648",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/1"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/11",
       "results": {
        "file": ".ai/experiments/EXP-W03-follow-instruction/results/results.jsonl",
        "commit": "f2e5759b",
        "sha256": "c082a5e358f2efb2233c8cbd2b4b4a480689f5ab824a087d8c80262d2349e383",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 24,
     "tokensPerTask": 2080.2,
     "modelMsPerTask": 3039,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 38,
        "rate": 0.95,
        "ci": [
         0.835,
         0.986
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           601,
           640
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 28,
        "rate": 0.7,
        "ci": [
         0.546,
         0.819
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           641,
           680
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           681,
           720
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 37,
        "rate": 0.925,
        "ci": [
         0.801,
         0.974
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           721,
           760
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 34,
        "rate": 0.85,
        "ci": [
         0.709,
         0.929
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           761,
           800
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 34,
        "rate": 0.68,
        "ci": [
         0.542,
         0.792
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           604,
           604
          ],
          [
           608,
           608
          ],
          [
           612,
           612
          ],
          [
           616,
           616
          ],
          [
           620,
           620
          ],
          [
           624,
           624
          ],
          [
           628,
           628
          ],
          [
           632,
           632
          ],
          [
           636,
           636
          ],
          [
           640,
           640
          ],
          [
           644,
           644
          ],
          [
           648,
           648
          ],
          [
           652,
           652
          ],
          [
           656,
           656
          ],
          [
           660,
           660
          ],
          [
           664,
           664
          ],
          [
           668,
           668
          ],
          [
           672,
           672
          ],
          [
           676,
           676
          ],
          [
           680,
           680
          ],
          [
           684,
           684
          ],
          [
           688,
           688
          ],
          [
           692,
           692
          ],
          [
           696,
           696
          ],
          [
           700,
           700
          ],
          [
           704,
           704
          ],
          [
           708,
           708
          ],
          [
           712,
           712
          ],
          [
           716,
           716
          ],
          [
           720,
           720
          ],
          [
           724,
           724
          ],
          [
           728,
           728
          ],
          [
           732,
           732
          ],
          [
           736,
           736
          ],
          [
           740,
           740
          ],
          [
           744,
           744
          ],
          [
           748,
           748
          ],
          [
           752,
           752
          ],
          [
           756,
           756
          ],
          [
           760,
           760
          ],
          [
           764,
           764
          ],
          [
           768,
           768
          ],
          [
           772,
           772
          ],
          [
           776,
           776
          ],
          [
           780,
           780
          ],
          [
           784,
           784
          ],
          [
           788,
           788
          ],
          [
           792,
           792
          ],
          [
           796,
           796
          ],
          [
           800,
           800
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           601,
           601
          ],
          [
           605,
           605
          ],
          [
           609,
           609
          ],
          [
           613,
           613
          ],
          [
           617,
           617
          ],
          [
           621,
           621
          ],
          [
           625,
           625
          ],
          [
           629,
           629
          ],
          [
           633,
           633
          ],
          [
           637,
           637
          ],
          [
           641,
           641
          ],
          [
           645,
           645
          ],
          [
           649,
           649
          ],
          [
           653,
           653
          ],
          [
           657,
           657
          ],
          [
           661,
           661
          ],
          [
           665,
           665
          ],
          [
           669,
           669
          ],
          [
           673,
           673
          ],
          [
           677,
           677
          ],
          [
           681,
           681
          ],
          [
           685,
           685
          ],
          [
           689,
           689
          ],
          [
           693,
           693
          ],
          [
           697,
           697
          ],
          [
           701,
           701
          ],
          [
           705,
           705
          ],
          [
           709,
           709
          ],
          [
           713,
           713
          ],
          [
           717,
           717
          ],
          [
           721,
           721
          ],
          [
           725,
           725
          ],
          [
           729,
           729
          ],
          [
           733,
           733
          ],
          [
           737,
           737
          ],
          [
           741,
           741
          ],
          [
           745,
           745
          ],
          [
           749,
           749
          ],
          [
           753,
           753
          ],
          [
           757,
           757
          ],
          [
           761,
           761
          ],
          [
           765,
           765
          ],
          [
           769,
           769
          ],
          [
           773,
           773
          ],
          [
           777,
           777
          ],
          [
           781,
           781
          ],
          [
           785,
           785
          ],
          [
           789,
           789
          ],
          [
           793,
           793
          ],
          [
           797,
           797
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           602,
           602
          ],
          [
           606,
           606
          ],
          [
           610,
           610
          ],
          [
           614,
           614
          ],
          [
           618,
           618
          ],
          [
           622,
           622
          ],
          [
           626,
           626
          ],
          [
           630,
           630
          ],
          [
           634,
           634
          ],
          [
           638,
           638
          ],
          [
           642,
           642
          ],
          [
           646,
           646
          ],
          [
           650,
           650
          ],
          [
           654,
           654
          ],
          [
           658,
           658
          ],
          [
           662,
           662
          ],
          [
           666,
           666
          ],
          [
           670,
           670
          ],
          [
           674,
           674
          ],
          [
           678,
           678
          ],
          [
           682,
           682
          ],
          [
           686,
           686
          ],
          [
           690,
           690
          ],
          [
           694,
           694
          ],
          [
           698,
           698
          ],
          [
           702,
           702
          ],
          [
           706,
           706
          ],
          [
           710,
           710
          ],
          [
           714,
           714
          ],
          [
           718,
           718
          ],
          [
           722,
           722
          ],
          [
           726,
           726
          ],
          [
           730,
           730
          ],
          [
           734,
           734
          ],
          [
           738,
           738
          ],
          [
           742,
           742
          ],
          [
           746,
           746
          ],
          [
           750,
           750
          ],
          [
           754,
           754
          ],
          [
           758,
           758
          ],
          [
           762,
           762
          ],
          [
           766,
           766
          ],
          [
           770,
           770
          ],
          [
           774,
           774
          ],
          [
           778,
           778
          ],
          [
           782,
           782
          ],
          [
           786,
           786
          ],
          [
           790,
           790
          ],
          [
           794,
           794
          ],
          [
           798,
           798
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 43,
        "rate": 0.86,
        "ci": [
         0.738,
         0.93
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           603,
           603
          ],
          [
           607,
           607
          ],
          [
           611,
           611
          ],
          [
           615,
           615
          ],
          [
           619,
           619
          ],
          [
           623,
           623
          ],
          [
           627,
           627
          ],
          [
           631,
           631
          ],
          [
           635,
           635
          ],
          [
           639,
           639
          ],
          [
           643,
           643
          ],
          [
           647,
           647
          ],
          [
           651,
           651
          ],
          [
           655,
           655
          ],
          [
           659,
           659
          ],
          [
           663,
           663
          ],
          [
           667,
           667
          ],
          [
           671,
           671
          ],
          [
           675,
           675
          ],
          [
           679,
           679
          ],
          [
           683,
           683
          ],
          [
           687,
           687
          ],
          [
           691,
           691
          ],
          [
           695,
           695
          ],
          [
           699,
           699
          ],
          [
           703,
           703
          ],
          [
           707,
           707
          ],
          [
           711,
           711
          ],
          [
           715,
           715
          ],
          [
           719,
           719
          ],
          [
           723,
           723
          ],
          [
           727,
           727
          ],
          [
           731,
           731
          ],
          [
           735,
           735
          ],
          [
           739,
           739
          ],
          [
           743,
           743
          ],
          [
           747,
           747
          ],
          [
           751,
           751
          ],
          [
           755,
           755
          ],
          [
           759,
           759
          ],
          [
           763,
           763
          ],
          [
           767,
           767
          ],
          [
           771,
           771
          ],
          [
           775,
           775
          ],
          [
           779,
           779
          ],
          [
           783,
           783
          ],
          [
           787,
           787
          ],
          [
           791,
           791
          ],
          [
           795,
           795
          ],
          [
           799,
           799
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "tuned-1gb",
     "label": "Our tuned 1 GB model",
     "where": "local",
     "model": "Qwen2.5 1.5B, tuned on our own decisions.",
     "brainVersion": "v3.1",
     "harness": "h4",
     "n": 200,
     "successes": 153,
     "rate": 0.765,
     "ci": [
      0.702,
      0.818
     ],
     "source": {
      "experiment": "EXP-W03",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        201,
        400
       ]
      ],
      "sha256": "198ddb47486ad4743f0fc85bacfa6a1696a086b816d6a6cb2c6983cad63a4648",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/2"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/9",
       "results": {
        "file": ".ai/experiments/EXP-W03-follow-instruction/results/results.jsonl",
        "commit": "f2e5759b",
        "sha256": "c082a5e358f2efb2233c8cbd2b4b4a480689f5ab824a087d8c80262d2349e383",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 2,
     "tokensPerTask": 3165.7,
     "modelMsPerTask": 4448.3,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 19,
        "rate": 0.475,
        "ci": [
         0.329,
         0.625
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           201,
           240
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 30,
        "rate": 0.75,
        "ci": [
         0.598,
         0.858
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           241,
           280
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           281,
           320
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           321,
           360
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 32,
        "rate": 0.8,
        "ci": [
         0.652,
         0.895
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           361,
           400
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 26,
        "rate": 0.52,
        "ci": [
         0.385,
         0.652
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           204,
           204
          ],
          [
           208,
           208
          ],
          [
           212,
           212
          ],
          [
           216,
           216
          ],
          [
           220,
           220
          ],
          [
           224,
           224
          ],
          [
           228,
           228
          ],
          [
           232,
           232
          ],
          [
           236,
           236
          ],
          [
           240,
           240
          ],
          [
           244,
           244
          ],
          [
           248,
           248
          ],
          [
           252,
           252
          ],
          [
           256,
           256
          ],
          [
           260,
           260
          ],
          [
           264,
           264
          ],
          [
           268,
           268
          ],
          [
           272,
           272
          ],
          [
           276,
           276
          ],
          [
           280,
           280
          ],
          [
           284,
           284
          ],
          [
           288,
           288
          ],
          [
           292,
           292
          ],
          [
           296,
           296
          ],
          [
           300,
           300
          ],
          [
           304,
           304
          ],
          [
           308,
           308
          ],
          [
           312,
           312
          ],
          [
           316,
           316
          ],
          [
           320,
           320
          ],
          [
           324,
           324
          ],
          [
           328,
           328
          ],
          [
           332,
           332
          ],
          [
           336,
           336
          ],
          [
           340,
           340
          ],
          [
           344,
           344
          ],
          [
           348,
           348
          ],
          [
           352,
           352
          ],
          [
           356,
           356
          ],
          [
           360,
           360
          ],
          [
           364,
           364
          ],
          [
           368,
           368
          ],
          [
           372,
           372
          ],
          [
           376,
           376
          ],
          [
           380,
           380
          ],
          [
           384,
           384
          ],
          [
           388,
           388
          ],
          [
           392,
           392
          ],
          [
           396,
           396
          ],
          [
           400,
           400
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 49,
        "rate": 0.98,
        "ci": [
         0.895,
         0.996
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           201,
           201
          ],
          [
           205,
           205
          ],
          [
           209,
           209
          ],
          [
           213,
           213
          ],
          [
           217,
           217
          ],
          [
           221,
           221
          ],
          [
           225,
           225
          ],
          [
           229,
           229
          ],
          [
           233,
           233
          ],
          [
           237,
           237
          ],
          [
           241,
           241
          ],
          [
           245,
           245
          ],
          [
           249,
           249
          ],
          [
           253,
           253
          ],
          [
           257,
           257
          ],
          [
           261,
           261
          ],
          [
           265,
           265
          ],
          [
           269,
           269
          ],
          [
           273,
           273
          ],
          [
           277,
           277
          ],
          [
           281,
           281
          ],
          [
           285,
           285
          ],
          [
           289,
           289
          ],
          [
           293,
           293
          ],
          [
           297,
           297
          ],
          [
           301,
           301
          ],
          [
           305,
           305
          ],
          [
           309,
           309
          ],
          [
           313,
           313
          ],
          [
           317,
           317
          ],
          [
           321,
           321
          ],
          [
           325,
           325
          ],
          [
           329,
           329
          ],
          [
           333,
           333
          ],
          [
           337,
           337
          ],
          [
           341,
           341
          ],
          [
           345,
           345
          ],
          [
           349,
           349
          ],
          [
           353,
           353
          ],
          [
           357,
           357
          ],
          [
           361,
           361
          ],
          [
           365,
           365
          ],
          [
           369,
           369
          ],
          [
           373,
           373
          ],
          [
           377,
           377
          ],
          [
           381,
           381
          ],
          [
           385,
           385
          ],
          [
           389,
           389
          ],
          [
           393,
           393
          ],
          [
           397,
           397
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 42,
        "rate": 0.84,
        "ci": [
         0.715,
         0.917
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           202,
           202
          ],
          [
           206,
           206
          ],
          [
           210,
           210
          ],
          [
           214,
           214
          ],
          [
           218,
           218
          ],
          [
           222,
           222
          ],
          [
           226,
           226
          ],
          [
           230,
           230
          ],
          [
           234,
           234
          ],
          [
           238,
           238
          ],
          [
           242,
           242
          ],
          [
           246,
           246
          ],
          [
           250,
           250
          ],
          [
           254,
           254
          ],
          [
           258,
           258
          ],
          [
           262,
           262
          ],
          [
           266,
           266
          ],
          [
           270,
           270
          ],
          [
           274,
           274
          ],
          [
           278,
           278
          ],
          [
           282,
           282
          ],
          [
           286,
           286
          ],
          [
           290,
           290
          ],
          [
           294,
           294
          ],
          [
           298,
           298
          ],
          [
           302,
           302
          ],
          [
           306,
           306
          ],
          [
           310,
           310
          ],
          [
           314,
           314
          ],
          [
           318,
           318
          ],
          [
           322,
           322
          ],
          [
           326,
           326
          ],
          [
           330,
           330
          ],
          [
           334,
           334
          ],
          [
           338,
           338
          ],
          [
           342,
           342
          ],
          [
           346,
           346
          ],
          [
           350,
           350
          ],
          [
           354,
           354
          ],
          [
           358,
           358
          ],
          [
           362,
           362
          ],
          [
           366,
           366
          ],
          [
           370,
           370
          ],
          [
           374,
           374
          ],
          [
           378,
           378
          ],
          [
           382,
           382
          ],
          [
           386,
           386
          ],
          [
           390,
           390
          ],
          [
           394,
           394
          ],
          [
           398,
           398
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 36,
        "rate": 0.72,
        "ci": [
         0.583,
         0.825
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           203,
           203
          ],
          [
           207,
           207
          ],
          [
           211,
           211
          ],
          [
           215,
           215
          ],
          [
           219,
           219
          ],
          [
           223,
           223
          ],
          [
           227,
           227
          ],
          [
           231,
           231
          ],
          [
           235,
           235
          ],
          [
           239,
           239
          ],
          [
           243,
           243
          ],
          [
           247,
           247
          ],
          [
           251,
           251
          ],
          [
           255,
           255
          ],
          [
           259,
           259
          ],
          [
           263,
           263
          ],
          [
           267,
           267
          ],
          [
           271,
           271
          ],
          [
           275,
           275
          ],
          [
           279,
           279
          ],
          [
           283,
           283
          ],
          [
           287,
           287
          ],
          [
           291,
           291
          ],
          [
           295,
           295
          ],
          [
           299,
           299
          ],
          [
           303,
           303
          ],
          [
           307,
           307
          ],
          [
           311,
           311
          ],
          [
           315,
           315
          ],
          [
           319,
           319
          ],
          [
           323,
           323
          ],
          [
           327,
           327
          ],
          [
           331,
           331
          ],
          [
           335,
           335
          ],
          [
           339,
           339
          ],
          [
           343,
           343
          ],
          [
           347,
           347
          ],
          [
           351,
           351
          ],
          [
           355,
           355
          ],
          [
           359,
           359
          ],
          [
           363,
           363
          ],
          [
           367,
           367
          ],
          [
           371,
           371
          ],
          [
           375,
           375
          ],
          [
           379,
           379
          ],
          [
           383,
           383
          ],
          [
           387,
           387
          ],
          [
           391,
           391
          ],
          [
           395,
           395
          ],
          [
           399,
           399
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    },
    {
     "id": "rules",
     "label": "Hand-written rules",
     "where": "rules",
     "model": "The control: no model at all.",
     "brainVersion": null,
     "harness": "h4",
     "n": 200,
     "successes": 55,
     "rate": 0.275,
     "ci": [
      0.218,
      0.341
     ],
     "source": {
      "experiment": "EXP-W03",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        1,
        200
       ]
      ],
      "sha256": "198ddb47486ad4743f0fc85bacfa6a1696a086b816d6a6cb2c6983cad63a4648",
      "checkedAgainst": {
       "file": "results/leaderboard.json",
       "pointer": "/brains/3"
      },
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/12",
       "results": {
        "file": ".ai/experiments/EXP-W03-follow-instruction/results/results.jsonl",
        "commit": "f2e5759b",
        "sha256": "c082a5e358f2efb2233c8cbd2b4b4a480689f5ab824a087d8c80262d2349e383",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 0,
     "modelMsPerTask": 0,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 17,
        "rate": 0.425,
        "ci": [
         0.285,
         0.578
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           40
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 6,
        "rate": 0.15,
        "ci": [
         0.071,
         0.291
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           41,
           80
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 12,
        "rate": 0.3,
        "ci": [
         0.181,
         0.454
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           81,
           120
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 10,
        "rate": 0.25,
        "ci": [
         0.142,
         0.402
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           121,
           160
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 10,
        "rate": 0.25,
        "ci": [
         0.142,
         0.402
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           161,
           200
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 8,
        "rate": 0.16,
        "ci": [
         0.083,
         0.285
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           4,
           4
          ],
          [
           8,
           8
          ],
          [
           12,
           12
          ],
          [
           16,
           16
          ],
          [
           20,
           20
          ],
          [
           24,
           24
          ],
          [
           28,
           28
          ],
          [
           32,
           32
          ],
          [
           36,
           36
          ],
          [
           40,
           40
          ],
          [
           44,
           44
          ],
          [
           48,
           48
          ],
          [
           52,
           52
          ],
          [
           56,
           56
          ],
          [
           60,
           60
          ],
          [
           64,
           64
          ],
          [
           68,
           68
          ],
          [
           72,
           72
          ],
          [
           76,
           76
          ],
          [
           80,
           80
          ],
          [
           84,
           84
          ],
          [
           88,
           88
          ],
          [
           92,
           92
          ],
          [
           96,
           96
          ],
          [
           100,
           100
          ],
          [
           104,
           104
          ],
          [
           108,
           108
          ],
          [
           112,
           112
          ],
          [
           116,
           116
          ],
          [
           120,
           120
          ],
          [
           124,
           124
          ],
          [
           128,
           128
          ],
          [
           132,
           132
          ],
          [
           136,
           136
          ],
          [
           140,
           140
          ],
          [
           144,
           144
          ],
          [
           148,
           148
          ],
          [
           152,
           152
          ],
          [
           156,
           156
          ],
          [
           160,
           160
          ],
          [
           164,
           164
          ],
          [
           168,
           168
          ],
          [
           172,
           172
          ],
          [
           176,
           176
          ],
          [
           180,
           180
          ],
          [
           184,
           184
          ],
          [
           188,
           188
          ],
          [
           192,
           192
          ],
          [
           196,
           196
          ],
          [
           200,
           200
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 18,
        "rate": 0.36,
        "ci": [
         0.241,
         0.499
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           1
          ],
          [
           5,
           5
          ],
          [
           9,
           9
          ],
          [
           13,
           13
          ],
          [
           17,
           17
          ],
          [
           21,
           21
          ],
          [
           25,
           25
          ],
          [
           29,
           29
          ],
          [
           33,
           33
          ],
          [
           37,
           37
          ],
          [
           41,
           41
          ],
          [
           45,
           45
          ],
          [
           49,
           49
          ],
          [
           53,
           53
          ],
          [
           57,
           57
          ],
          [
           61,
           61
          ],
          [
           65,
           65
          ],
          [
           69,
           69
          ],
          [
           73,
           73
          ],
          [
           77,
           77
          ],
          [
           81,
           81
          ],
          [
           85,
           85
          ],
          [
           89,
           89
          ],
          [
           93,
           93
          ],
          [
           97,
           97
          ],
          [
           101,
           101
          ],
          [
           105,
           105
          ],
          [
           109,
           109
          ],
          [
           113,
           113
          ],
          [
           117,
           117
          ],
          [
           121,
           121
          ],
          [
           125,
           125
          ],
          [
           129,
           129
          ],
          [
           133,
           133
          ],
          [
           137,
           137
          ],
          [
           141,
           141
          ],
          [
           145,
           145
          ],
          [
           149,
           149
          ],
          [
           153,
           153
          ],
          [
           157,
           157
          ],
          [
           161,
           161
          ],
          [
           165,
           165
          ],
          [
           169,
           169
          ],
          [
           173,
           173
          ],
          [
           177,
           177
          ],
          [
           181,
           181
          ],
          [
           185,
           185
          ],
          [
           189,
           189
          ],
          [
           193,
           193
          ],
          [
           197,
           197
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 15,
        "rate": 0.3,
        "ci": [
         0.191,
         0.438
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           2,
           2
          ],
          [
           6,
           6
          ],
          [
           10,
           10
          ],
          [
           14,
           14
          ],
          [
           18,
           18
          ],
          [
           22,
           22
          ],
          [
           26,
           26
          ],
          [
           30,
           30
          ],
          [
           34,
           34
          ],
          [
           38,
           38
          ],
          [
           42,
           42
          ],
          [
           46,
           46
          ],
          [
           50,
           50
          ],
          [
           54,
           54
          ],
          [
           58,
           58
          ],
          [
           62,
           62
          ],
          [
           66,
           66
          ],
          [
           70,
           70
          ],
          [
           74,
           74
          ],
          [
           78,
           78
          ],
          [
           82,
           82
          ],
          [
           86,
           86
          ],
          [
           90,
           90
          ],
          [
           94,
           94
          ],
          [
           98,
           98
          ],
          [
           102,
           102
          ],
          [
           106,
           106
          ],
          [
           110,
           110
          ],
          [
           114,
           114
          ],
          [
           118,
           118
          ],
          [
           122,
           122
          ],
          [
           126,
           126
          ],
          [
           130,
           130
          ],
          [
           134,
           134
          ],
          [
           138,
           138
          ],
          [
           142,
           142
          ],
          [
           146,
           146
          ],
          [
           150,
           150
          ],
          [
           154,
           154
          ],
          [
           158,
           158
          ],
          [
           162,
           162
          ],
          [
           166,
           166
          ],
          [
           170,
           170
          ],
          [
           174,
           174
          ],
          [
           178,
           178
          ],
          [
           182,
           182
          ],
          [
           186,
           186
          ],
          [
           190,
           190
          ],
          [
           194,
           194
          ],
          [
           198,
           198
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 14,
        "rate": 0.28,
        "ci": [
         0.175,
         0.417
        ],
        "source": {
         "experiment": "EXP-W03",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           3,
           3
          ],
          [
           7,
           7
          ],
          [
           11,
           11
          ],
          [
           15,
           15
          ],
          [
           19,
           19
          ],
          [
           23,
           23
          ],
          [
           27,
           27
          ],
          [
           31,
           31
          ],
          [
           35,
           35
          ],
          [
           39,
           39
          ],
          [
           43,
           43
          ],
          [
           47,
           47
          ],
          [
           51,
           51
          ],
          [
           55,
           55
          ],
          [
           59,
           59
          ],
          [
           63,
           63
          ],
          [
           67,
           67
          ],
          [
           71,
           71
          ],
          [
           75,
           75
          ],
          [
           79,
           79
          ],
          [
           83,
           83
          ],
          [
           87,
           87
          ],
          [
           91,
           91
          ],
          [
           95,
           95
          ],
          [
           99,
           99
          ],
          [
           103,
           103
          ],
          [
           107,
           107
          ],
          [
           111,
           111
          ],
          [
           115,
           115
          ],
          [
           119,
           119
          ],
          [
           123,
           123
          ],
          [
           127,
           127
          ],
          [
           131,
           131
          ],
          [
           135,
           135
          ],
          [
           139,
           139
          ],
          [
           143,
           143
          ],
          [
           147,
           147
          ],
          [
           151,
           151
          ],
          [
           155,
           155
          ],
          [
           159,
           159
          ],
          [
           163,
           163
          ],
          [
           167,
           167
          ],
          [
           171,
           171
          ],
          [
           175,
           175
          ],
          [
           179,
           179
          ],
          [
           183,
           183
          ],
          [
           187,
           187
          ],
          [
           191,
           191
          ],
          [
           195,
           195
          ],
          [
           199,
           199
          ]
         ]
        }
       }
      }
     },
     "packNote": "ran on harness h4 (the shop price bug; see EXP-W04)"
    }
   ]
  },
  {
   "id": "EXP-W04",
   "title": "After fixing the bench, is the gap still there?",
   "question": "The instruction task again, on new seeds, under harness h5 (the shop-price fix) and with a second version of our model's brain that names its arguments.",
   "date": "2026-10-25",
   "design": {
    "tasks": 1,
    "worlds": 5,
    "seeds": {
     "from": 140,
     "to": 179,
     "count": 40
    },
    "brains": 4,
    "episodesPerBrain": 200,
    "episodes": 800,
    "skippedRows": 5,
    "harness": [
     "h5"
    ],
    "source": {
     "experiment": "EXP-W04",
     "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
     "file": "results/results.jsonl",
     "lines": [
      [
       1,
       805
      ]
     ]
    }
   },
   "prereg": {
    "commit": "5a01a950",
    "sha256": "cd25acbc7ff10e7d3f5f7275c72add458c402d20485bd5ccfd8f6eadaa6c1ebb",
    "file": "PREREG.md",
    "experiment": "EXP-W04"
   },
   "comparators": [],
   "predictions": [
    {
     "id": "P1",
     "text": "Under h5, our tuned model with brain v3.1 finishes at least 165 of 200.",
     "verdict": "holds",
     "detail": "It finished {EXP-W04:tuned-1gb:of}."
    },
    {
     "id": "P2",
     "text": "Brain v3.2 against v3.1 on our tuned model: no detectable difference.",
     "verdict": "holds",
     "detail": "{EXP-W04:tuned-1gb-v3.2:of} against {EXP-W04:tuned-1gb:of}; the intervals overlap."
    },
    {
     "id": "P3",
     "text": "Mistral Small 3.2 against our tuned model on brain v3.2: no detectable difference. The bold one.",
     "verdict": "holds as registered, not claimed",
     "detail": "The intervals overlap, narrowly ({EXP-W04:mistral-small-3.2:ci} against {EXP-W04:tuned-1gb-v3.2:ci}). It is not robust: part of the cloud model's run hit service errors. With those {EXP-W04:mistral-small-3.2:rerun} episodes re-run, it finishes {EXP-W04:mistral-small-3.2:sensitivity} ({EXP-W04:mistral-small-3.2:sensitivityCi}), clearly better than ours. We do not claim this prediction."
    },
    {
     "id": "P4",
     "text": "Every model brain beats the rules, with intervals that do not overlap.",
     "verdict": "holds",
     "detail": "The rules at {EXP-W04:rules:ci}."
    },
    {
     "id": "P5",
     "text": "The run stays under the cost ceiling set before it.",
     "verdict": "holds",
     "detail": "By the bench's own estimate for the cloud model's episodes. Cost was tracked, but this site does not publish money figures."
    }
   ],
   "unpredicted": [
    "Brain v3.2 does not measurably help on top of the fix. It does better at buying something for the person ({EXP-W04:tuned-1gb-v3.2:kind.buy} against {EXP-W04:tuned-1gb:kind.buy}) and gives back on handing something over ({EXP-W04:tuned-1gb-v3.2:kind.hand} against {EXP-W04:tuned-1gb:kind.hand}).",
    "Where our model still loses is going to meet someone and buying for them: the indirect phrasings, such as being asked to meet by someone rather than to go somewhere."
   ],
   "caveats": [
    "Ran on harness h5, which shows the cost of one purchase, and is never compared with EXP-W03's h4 rows as if they were the same measurement.",
    "The first launch skipped the cloud model: an old, dead key in the PC's environment was refused. The arm was resumed with the identical command and the live key, so only the missing episodes ran.",
    "In World D, seeds {EXP-W04:mistral-small-3.2:errorSeeds}, the cloud model's calls returned errors: a call gave no reply, and the rules decided instead. From the next seed on, World D is normal. This biases the result against the cloud model, and the run log does not record the error status, so an outage looks like a weak model. The bench should record a service error apart from an unusable reply.",
    "Sensitivity check, disclosed and not part of the registered result: those {EXP-W04:mistral-small-3.2:rerun} episodes were re-run with the same command and seeds, and {EXP-W04:mistral-small-3.2:rerunWon} of them finished, with only ordinary fallbacks and no service errors.",
    "The real spend on the key cannot be used for this run, because other work spent on the same key at the same time; P5 rests on the bench's own estimate.",
    "The PC was short of memory at times. Episodes waited; none were dropped."
   ],
   "brains": [
    {
     "id": "mistral-small-3.2",
     "label": "Mistral Small 3.2",
     "where": "cloud",
     "model": "Mistral Small 3.2, 24B, instruct.",
     "brainVersion": "v3.1",
     "harness": "h5",
     "n": 200,
     "successes": 188,
     "rate": 0.94,
     "ci": [
      0.898,
      0.965
     ],
     "source": {
      "experiment": "EXP-W04",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        606,
        805
       ]
      ],
      "sha256": "dff4707dd2d8f7d6cff4f29001cbe015248d5b26c46f23767be9ee8b937a7665",
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/14",
       "results": {
        "file": ".ai/experiments/EXP-W04-h5-named-args/results/results.jsonl",
        "commit": "96d6f7a3",
        "sha256": "4c7ef3e30415cc7db7ce755cfb24ef6c60e5f9321fbb979b1a19bfdc4cb5ec7e",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 113,
     "tokensPerTask": 1623,
     "modelMsPerTask": 1920.5,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           606,
           645
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           646,
           685
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 39,
        "rate": 0.975,
        "ci": [
         0.871,
         0.996
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           686,
           725
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 29,
        "rate": 0.725,
        "ci": [
         0.572,
         0.839
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           726,
           765
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 40,
        "rate": 1,
        "ci": [
         0.912,
         1
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           766,
           805
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 45,
        "rate": 0.9,
        "ci": [
         0.786,
         0.957
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           609,
           609
          ],
          [
           613,
           613
          ],
          [
           617,
           617
          ],
          [
           621,
           621
          ],
          [
           625,
           625
          ],
          [
           629,
           629
          ],
          [
           633,
           633
          ],
          [
           637,
           637
          ],
          [
           641,
           641
          ],
          [
           645,
           645
          ],
          [
           649,
           649
          ],
          [
           653,
           653
          ],
          [
           657,
           657
          ],
          [
           661,
           661
          ],
          [
           665,
           665
          ],
          [
           669,
           669
          ],
          [
           673,
           673
          ],
          [
           677,
           677
          ],
          [
           681,
           681
          ],
          [
           685,
           685
          ],
          [
           689,
           689
          ],
          [
           693,
           693
          ],
          [
           697,
           697
          ],
          [
           701,
           701
          ],
          [
           705,
           705
          ],
          [
           709,
           709
          ],
          [
           713,
           713
          ],
          [
           717,
           717
          ],
          [
           721,
           721
          ],
          [
           725,
           725
          ],
          [
           729,
           729
          ],
          [
           733,
           733
          ],
          [
           737,
           737
          ],
          [
           741,
           741
          ],
          [
           745,
           745
          ],
          [
           749,
           749
          ],
          [
           753,
           753
          ],
          [
           757,
           757
          ],
          [
           761,
           761
          ],
          [
           765,
           765
          ],
          [
           769,
           769
          ],
          [
           773,
           773
          ],
          [
           777,
           777
          ],
          [
           781,
           781
          ],
          [
           785,
           785
          ],
          [
           789,
           789
          ],
          [
           793,
           793
          ],
          [
           797,
           797
          ],
          [
           801,
           801
          ],
          [
           805,
           805
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 46,
        "rate": 0.92,
        "ci": [
         0.812,
         0.968
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           606,
           606
          ],
          [
           610,
           610
          ],
          [
           614,
           614
          ],
          [
           618,
           618
          ],
          [
           622,
           622
          ],
          [
           626,
           626
          ],
          [
           630,
           630
          ],
          [
           634,
           634
          ],
          [
           638,
           638
          ],
          [
           642,
           642
          ],
          [
           646,
           646
          ],
          [
           650,
           650
          ],
          [
           654,
           654
          ],
          [
           658,
           658
          ],
          [
           662,
           662
          ],
          [
           666,
           666
          ],
          [
           670,
           670
          ],
          [
           674,
           674
          ],
          [
           678,
           678
          ],
          [
           682,
           682
          ],
          [
           686,
           686
          ],
          [
           690,
           690
          ],
          [
           694,
           694
          ],
          [
           698,
           698
          ],
          [
           702,
           702
          ],
          [
           706,
           706
          ],
          [
           710,
           710
          ],
          [
           714,
           714
          ],
          [
           718,
           718
          ],
          [
           722,
           722
          ],
          [
           726,
           726
          ],
          [
           730,
           730
          ],
          [
           734,
           734
          ],
          [
           738,
           738
          ],
          [
           742,
           742
          ],
          [
           746,
           746
          ],
          [
           750,
           750
          ],
          [
           754,
           754
          ],
          [
           758,
           758
          ],
          [
           762,
           762
          ],
          [
           766,
           766
          ],
          [
           770,
           770
          ],
          [
           774,
           774
          ],
          [
           778,
           778
          ],
          [
           782,
           782
          ],
          [
           786,
           786
          ],
          [
           790,
           790
          ],
          [
           794,
           794
          ],
          [
           798,
           798
          ],
          [
           802,
           802
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 49,
        "rate": 0.98,
        "ci": [
         0.895,
         0.996
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           607,
           607
          ],
          [
           611,
           611
          ],
          [
           615,
           615
          ],
          [
           619,
           619
          ],
          [
           623,
           623
          ],
          [
           627,
           627
          ],
          [
           631,
           631
          ],
          [
           635,
           635
          ],
          [
           639,
           639
          ],
          [
           643,
           643
          ],
          [
           647,
           647
          ],
          [
           651,
           651
          ],
          [
           655,
           655
          ],
          [
           659,
           659
          ],
          [
           663,
           663
          ],
          [
           667,
           667
          ],
          [
           671,
           671
          ],
          [
           675,
           675
          ],
          [
           679,
           679
          ],
          [
           683,
           683
          ],
          [
           687,
           687
          ],
          [
           691,
           691
          ],
          [
           695,
           695
          ],
          [
           699,
           699
          ],
          [
           703,
           703
          ],
          [
           707,
           707
          ],
          [
           711,
           711
          ],
          [
           715,
           715
          ],
          [
           719,
           719
          ],
          [
           723,
           723
          ],
          [
           727,
           727
          ],
          [
           731,
           731
          ],
          [
           735,
           735
          ],
          [
           739,
           739
          ],
          [
           743,
           743
          ],
          [
           747,
           747
          ],
          [
           751,
           751
          ],
          [
           755,
           755
          ],
          [
           759,
           759
          ],
          [
           763,
           763
          ],
          [
           767,
           767
          ],
          [
           771,
           771
          ],
          [
           775,
           775
          ],
          [
           779,
           779
          ],
          [
           783,
           783
          ],
          [
           787,
           787
          ],
          [
           791,
           791
          ],
          [
           795,
           795
          ],
          [
           799,
           799
          ],
          [
           803,
           803
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 48,
        "rate": 0.96,
        "ci": [
         0.865,
         0.989
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           608,
           608
          ],
          [
           612,
           612
          ],
          [
           616,
           616
          ],
          [
           620,
           620
          ],
          [
           624,
           624
          ],
          [
           628,
           628
          ],
          [
           632,
           632
          ],
          [
           636,
           636
          ],
          [
           640,
           640
          ],
          [
           644,
           644
          ],
          [
           648,
           648
          ],
          [
           652,
           652
          ],
          [
           656,
           656
          ],
          [
           660,
           660
          ],
          [
           664,
           664
          ],
          [
           668,
           668
          ],
          [
           672,
           672
          ],
          [
           676,
           676
          ],
          [
           680,
           680
          ],
          [
           684,
           684
          ],
          [
           688,
           688
          ],
          [
           692,
           692
          ],
          [
           696,
           696
          ],
          [
           700,
           700
          ],
          [
           704,
           704
          ],
          [
           708,
           708
          ],
          [
           712,
           712
          ],
          [
           716,
           716
          ],
          [
           720,
           720
          ],
          [
           724,
           724
          ],
          [
           728,
           728
          ],
          [
           732,
           732
          ],
          [
           736,
           736
          ],
          [
           740,
           740
          ],
          [
           744,
           744
          ],
          [
           748,
           748
          ],
          [
           752,
           752
          ],
          [
           756,
           756
          ],
          [
           760,
           760
          ],
          [
           764,
           764
          ],
          [
           768,
           768
          ],
          [
           772,
           772
          ],
          [
           776,
           776
          ],
          [
           780,
           780
          ],
          [
           784,
           784
          ],
          [
           788,
           788
          ],
          [
           792,
           792
          ],
          [
           796,
           796
          ],
          [
           800,
           800
          ],
          [
           804,
           804
          ]
         ]
        }
       }
      }
     },
     "sensitivity": {
      "world": "d",
      "seeds": {
       "from": 140,
       "to": 164
      },
      "rerun": 25,
      "rerunWon": 25,
      "rerunFallbacks": 3,
      "n": 200,
      "successes": 199,
      "ci": [
       0.972,
       0.999
      ],
      "emptyReplyEpisodes": 15,
      "source": {
       "experiment": "EXP-W04",
       "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
       "file": "sensitivity/",
       "fileIs": "the only file in this folder, identified by its sha256",
       "sha256": "b7e3be262334d503025394942b995735b7cb668f95a0f3a7bc7ad86afe554022",
       "lines": [
        [
         1,
         25
        ]
       ],
       "replaces": {
        "file": "results/results.jsonl",
        "sha256": "dff4707dd2d8f7d6cff4f29001cbe015248d5b26c46f23767be9ee8b937a7665",
        "lines": [
         [
          726,
          750
         ]
        ]
       },
       "pack": {
        "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
        "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
        "pointer": "/benchmark/17",
        "results": {
         "file": ".ai/experiments/EXP-W04-h5-named-args/sensitivity/",
         "commit": "96d6f7a3",
         "sha256": "7bdf8474d70400fa20d4bf5cd92265c332b965d3ef832d2a8a71923a81cfec6e",
         "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
        }
       }
      },
      "packNote": "not the registered result: the 25 episodes hit by an API outage, re-run",
      "registered": false
     },
     "packNote": "harness h5 (the bug fixed). P3 ('no detectable difference') held only because of an API outage and is declined: with the 25 outage episodes re-run, the cloud model is clearly better"
    },
    {
     "id": "tuned-1gb-v3.2",
     "label": "Our tuned 1 GB model, brain v3.2",
     "where": "local",
     "model": "The same model, with named arguments and the agent's bag and money stated.",
     "brainVersion": "v3.2",
     "harness": "h5",
     "n": 200,
     "successes": 174,
     "rate": 0.87,
     "ci": [
      0.816,
      0.91
     ],
     "source": {
      "experiment": "EXP-W04",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        401,
        600
       ]
      ],
      "sha256": "dff4707dd2d8f7d6cff4f29001cbe015248d5b26c46f23767be9ee8b937a7665",
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/15",
       "results": {
        "file": ".ai/experiments/EXP-W04-h5-named-args/results/results.jsonl",
        "commit": "96d6f7a3",
        "sha256": "4c7ef3e30415cc7db7ce755cfb24ef6c60e5f9321fbb979b1a19bfdc4cb5ec7e",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 3390.8,
     "modelMsPerTask": 3333.5,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           401,
           440
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 32,
        "rate": 0.8,
        "ci": [
         0.652,
         0.895
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           441,
           480
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 35,
        "rate": 0.875,
        "ci": [
         0.739,
         0.945
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           481,
           520
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 35,
        "rate": 0.875,
        "ci": [
         0.739,
         0.945
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           521,
           560
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           561,
           600
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 43,
        "rate": 0.86,
        "ci": [
         0.738,
         0.93
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           404,
           404
          ],
          [
           408,
           408
          ],
          [
           412,
           412
          ],
          [
           416,
           416
          ],
          [
           420,
           420
          ],
          [
           424,
           424
          ],
          [
           428,
           428
          ],
          [
           432,
           432
          ],
          [
           436,
           436
          ],
          [
           440,
           440
          ],
          [
           444,
           444
          ],
          [
           448,
           448
          ],
          [
           452,
           452
          ],
          [
           456,
           456
          ],
          [
           460,
           460
          ],
          [
           464,
           464
          ],
          [
           468,
           468
          ],
          [
           472,
           472
          ],
          [
           476,
           476
          ],
          [
           480,
           480
          ],
          [
           484,
           484
          ],
          [
           488,
           488
          ],
          [
           492,
           492
          ],
          [
           496,
           496
          ],
          [
           500,
           500
          ],
          [
           504,
           504
          ],
          [
           508,
           508
          ],
          [
           512,
           512
          ],
          [
           516,
           516
          ],
          [
           520,
           520
          ],
          [
           524,
           524
          ],
          [
           528,
           528
          ],
          [
           532,
           532
          ],
          [
           536,
           536
          ],
          [
           540,
           540
          ],
          [
           544,
           544
          ],
          [
           548,
           548
          ],
          [
           552,
           552
          ],
          [
           556,
           556
          ],
          [
           560,
           560
          ],
          [
           564,
           564
          ],
          [
           568,
           568
          ],
          [
           572,
           572
          ],
          [
           576,
           576
          ],
          [
           580,
           580
          ],
          [
           584,
           584
          ],
          [
           588,
           588
          ],
          [
           592,
           592
          ],
          [
           596,
           596
          ],
          [
           600,
           600
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 49,
        "rate": 0.98,
        "ci": [
         0.895,
         0.996
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           401,
           401
          ],
          [
           405,
           405
          ],
          [
           409,
           409
          ],
          [
           413,
           413
          ],
          [
           417,
           417
          ],
          [
           421,
           421
          ],
          [
           425,
           425
          ],
          [
           429,
           429
          ],
          [
           433,
           433
          ],
          [
           437,
           437
          ],
          [
           441,
           441
          ],
          [
           445,
           445
          ],
          [
           449,
           449
          ],
          [
           453,
           453
          ],
          [
           457,
           457
          ],
          [
           461,
           461
          ],
          [
           465,
           465
          ],
          [
           469,
           469
          ],
          [
           473,
           473
          ],
          [
           477,
           477
          ],
          [
           481,
           481
          ],
          [
           485,
           485
          ],
          [
           489,
           489
          ],
          [
           493,
           493
          ],
          [
           497,
           497
          ],
          [
           501,
           501
          ],
          [
           505,
           505
          ],
          [
           509,
           509
          ],
          [
           513,
           513
          ],
          [
           517,
           517
          ],
          [
           521,
           521
          ],
          [
           525,
           525
          ],
          [
           529,
           529
          ],
          [
           533,
           533
          ],
          [
           537,
           537
          ],
          [
           541,
           541
          ],
          [
           545,
           545
          ],
          [
           549,
           549
          ],
          [
           553,
           553
          ],
          [
           557,
           557
          ],
          [
           561,
           561
          ],
          [
           565,
           565
          ],
          [
           569,
           569
          ],
          [
           573,
           573
          ],
          [
           577,
           577
          ],
          [
           581,
           581
          ],
          [
           585,
           585
          ],
          [
           589,
           589
          ],
          [
           593,
           593
          ],
          [
           597,
           597
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 43,
        "rate": 0.86,
        "ci": [
         0.738,
         0.93
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           402,
           402
          ],
          [
           406,
           406
          ],
          [
           410,
           410
          ],
          [
           414,
           414
          ],
          [
           418,
           418
          ],
          [
           422,
           422
          ],
          [
           426,
           426
          ],
          [
           430,
           430
          ],
          [
           434,
           434
          ],
          [
           438,
           438
          ],
          [
           442,
           442
          ],
          [
           446,
           446
          ],
          [
           450,
           450
          ],
          [
           454,
           454
          ],
          [
           458,
           458
          ],
          [
           462,
           462
          ],
          [
           466,
           466
          ],
          [
           470,
           470
          ],
          [
           474,
           474
          ],
          [
           478,
           478
          ],
          [
           482,
           482
          ],
          [
           486,
           486
          ],
          [
           490,
           490
          ],
          [
           494,
           494
          ],
          [
           498,
           498
          ],
          [
           502,
           502
          ],
          [
           506,
           506
          ],
          [
           510,
           510
          ],
          [
           514,
           514
          ],
          [
           518,
           518
          ],
          [
           522,
           522
          ],
          [
           526,
           526
          ],
          [
           530,
           530
          ],
          [
           534,
           534
          ],
          [
           538,
           538
          ],
          [
           542,
           542
          ],
          [
           546,
           546
          ],
          [
           550,
           550
          ],
          [
           554,
           554
          ],
          [
           558,
           558
          ],
          [
           562,
           562
          ],
          [
           566,
           566
          ],
          [
           570,
           570
          ],
          [
           574,
           574
          ],
          [
           578,
           578
          ],
          [
           582,
           582
          ],
          [
           586,
           586
          ],
          [
           590,
           590
          ],
          [
           594,
           594
          ],
          [
           598,
           598
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 39,
        "rate": 0.78,
        "ci": [
         0.648,
         0.872
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           403,
           403
          ],
          [
           407,
           407
          ],
          [
           411,
           411
          ],
          [
           415,
           415
          ],
          [
           419,
           419
          ],
          [
           423,
           423
          ],
          [
           427,
           427
          ],
          [
           431,
           431
          ],
          [
           435,
           435
          ],
          [
           439,
           439
          ],
          [
           443,
           443
          ],
          [
           447,
           447
          ],
          [
           451,
           451
          ],
          [
           455,
           455
          ],
          [
           459,
           459
          ],
          [
           463,
           463
          ],
          [
           467,
           467
          ],
          [
           471,
           471
          ],
          [
           475,
           475
          ],
          [
           479,
           479
          ],
          [
           483,
           483
          ],
          [
           487,
           487
          ],
          [
           491,
           491
          ],
          [
           495,
           495
          ],
          [
           499,
           499
          ],
          [
           503,
           503
          ],
          [
           507,
           507
          ],
          [
           511,
           511
          ],
          [
           515,
           515
          ],
          [
           519,
           519
          ],
          [
           523,
           523
          ],
          [
           527,
           527
          ],
          [
           531,
           531
          ],
          [
           535,
           535
          ],
          [
           539,
           539
          ],
          [
           543,
           543
          ],
          [
           547,
           547
          ],
          [
           551,
           551
          ],
          [
           555,
           555
          ],
          [
           559,
           559
          ],
          [
           563,
           563
          ],
          [
           567,
           567
          ],
          [
           571,
           571
          ],
          [
           575,
           575
          ],
          [
           579,
           579
          ],
          [
           583,
           583
          ],
          [
           587,
           587
          ],
          [
           591,
           591
          ],
          [
           595,
           595
          ],
          [
           599,
           599
          ]
         ]
        }
       }
      }
     },
     "packNote": "harness h5 (the bug fixed). P3 ('no detectable difference') held only because of an API outage and is declined: with the 25 outage episodes re-run, the cloud model is clearly better"
    },
    {
     "id": "tuned-1gb",
     "label": "Our tuned 1 GB model",
     "where": "local",
     "model": "Qwen2.5 1.5B, tuned on our own decisions.",
     "brainVersion": "v3.1",
     "harness": "h5",
     "n": 200,
     "successes": 172,
     "rate": 0.86,
     "ci": [
      0.805,
      0.901
     ],
     "source": {
      "experiment": "EXP-W04",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        201,
        400
       ]
      ],
      "sha256": "dff4707dd2d8f7d6cff4f29001cbe015248d5b26c46f23767be9ee8b937a7665",
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/13",
       "results": {
        "file": ".ai/experiments/EXP-W04-h5-named-args/results/results.jsonl",
        "commit": "96d6f7a3",
        "sha256": "4c7ef3e30415cc7db7ce755cfb24ef6c60e5f9321fbb979b1a19bfdc4cb5ec7e",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 3500.6,
     "modelMsPerTask": 4134.5,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           201,
           240
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           241,
           280
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 35,
        "rate": 0.875,
        "ci": [
         0.739,
         0.945
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           281,
           320
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 36,
        "rate": 0.9,
        "ci": [
         0.769,
         0.96
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           321,
           360
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 29,
        "rate": 0.725,
        "ci": [
         0.572,
         0.839
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           361,
           400
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 35,
        "rate": 0.7,
        "ci": [
         0.562,
         0.809
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           204,
           204
          ],
          [
           208,
           208
          ],
          [
           212,
           212
          ],
          [
           216,
           216
          ],
          [
           220,
           220
          ],
          [
           224,
           224
          ],
          [
           228,
           228
          ],
          [
           232,
           232
          ],
          [
           236,
           236
          ],
          [
           240,
           240
          ],
          [
           244,
           244
          ],
          [
           248,
           248
          ],
          [
           252,
           252
          ],
          [
           256,
           256
          ],
          [
           260,
           260
          ],
          [
           264,
           264
          ],
          [
           268,
           268
          ],
          [
           272,
           272
          ],
          [
           276,
           276
          ],
          [
           280,
           280
          ],
          [
           284,
           284
          ],
          [
           288,
           288
          ],
          [
           292,
           292
          ],
          [
           296,
           296
          ],
          [
           300,
           300
          ],
          [
           304,
           304
          ],
          [
           308,
           308
          ],
          [
           312,
           312
          ],
          [
           316,
           316
          ],
          [
           320,
           320
          ],
          [
           324,
           324
          ],
          [
           328,
           328
          ],
          [
           332,
           332
          ],
          [
           336,
           336
          ],
          [
           340,
           340
          ],
          [
           344,
           344
          ],
          [
           348,
           348
          ],
          [
           352,
           352
          ],
          [
           356,
           356
          ],
          [
           360,
           360
          ],
          [
           364,
           364
          ],
          [
           368,
           368
          ],
          [
           372,
           372
          ],
          [
           376,
           376
          ],
          [
           380,
           380
          ],
          [
           384,
           384
          ],
          [
           388,
           388
          ],
          [
           392,
           392
          ],
          [
           396,
           396
          ],
          [
           400,
           400
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 50,
        "rate": 1,
        "ci": [
         0.929,
         1
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           201,
           201
          ],
          [
           205,
           205
          ],
          [
           209,
           209
          ],
          [
           213,
           213
          ],
          [
           217,
           217
          ],
          [
           221,
           221
          ],
          [
           225,
           225
          ],
          [
           229,
           229
          ],
          [
           233,
           233
          ],
          [
           237,
           237
          ],
          [
           241,
           241
          ],
          [
           245,
           245
          ],
          [
           249,
           249
          ],
          [
           253,
           253
          ],
          [
           257,
           257
          ],
          [
           261,
           261
          ],
          [
           265,
           265
          ],
          [
           269,
           269
          ],
          [
           273,
           273
          ],
          [
           277,
           277
          ],
          [
           281,
           281
          ],
          [
           285,
           285
          ],
          [
           289,
           289
          ],
          [
           293,
           293
          ],
          [
           297,
           297
          ],
          [
           301,
           301
          ],
          [
           305,
           305
          ],
          [
           309,
           309
          ],
          [
           313,
           313
          ],
          [
           317,
           317
          ],
          [
           321,
           321
          ],
          [
           325,
           325
          ],
          [
           329,
           329
          ],
          [
           333,
           333
          ],
          [
           337,
           337
          ],
          [
           341,
           341
          ],
          [
           345,
           345
          ],
          [
           349,
           349
          ],
          [
           353,
           353
          ],
          [
           357,
           357
          ],
          [
           361,
           361
          ],
          [
           365,
           365
          ],
          [
           369,
           369
          ],
          [
           373,
           373
          ],
          [
           377,
           377
          ],
          [
           381,
           381
          ],
          [
           385,
           385
          ],
          [
           389,
           389
          ],
          [
           393,
           393
          ],
          [
           397,
           397
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 49,
        "rate": 0.98,
        "ci": [
         0.895,
         0.996
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           202,
           202
          ],
          [
           206,
           206
          ],
          [
           210,
           210
          ],
          [
           214,
           214
          ],
          [
           218,
           218
          ],
          [
           222,
           222
          ],
          [
           226,
           226
          ],
          [
           230,
           230
          ],
          [
           234,
           234
          ],
          [
           238,
           238
          ],
          [
           242,
           242
          ],
          [
           246,
           246
          ],
          [
           250,
           250
          ],
          [
           254,
           254
          ],
          [
           258,
           258
          ],
          [
           262,
           262
          ],
          [
           266,
           266
          ],
          [
           270,
           270
          ],
          [
           274,
           274
          ],
          [
           278,
           278
          ],
          [
           282,
           282
          ],
          [
           286,
           286
          ],
          [
           290,
           290
          ],
          [
           294,
           294
          ],
          [
           298,
           298
          ],
          [
           302,
           302
          ],
          [
           306,
           306
          ],
          [
           310,
           310
          ],
          [
           314,
           314
          ],
          [
           318,
           318
          ],
          [
           322,
           322
          ],
          [
           326,
           326
          ],
          [
           330,
           330
          ],
          [
           334,
           334
          ],
          [
           338,
           338
          ],
          [
           342,
           342
          ],
          [
           346,
           346
          ],
          [
           350,
           350
          ],
          [
           354,
           354
          ],
          [
           358,
           358
          ],
          [
           362,
           362
          ],
          [
           366,
           366
          ],
          [
           370,
           370
          ],
          [
           374,
           374
          ],
          [
           378,
           378
          ],
          [
           382,
           382
          ],
          [
           386,
           386
          ],
          [
           390,
           390
          ],
          [
           394,
           394
          ],
          [
           398,
           398
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 38,
        "rate": 0.76,
        "ci": [
         0.626,
         0.857
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           203,
           203
          ],
          [
           207,
           207
          ],
          [
           211,
           211
          ],
          [
           215,
           215
          ],
          [
           219,
           219
          ],
          [
           223,
           223
          ],
          [
           227,
           227
          ],
          [
           231,
           231
          ],
          [
           235,
           235
          ],
          [
           239,
           239
          ],
          [
           243,
           243
          ],
          [
           247,
           247
          ],
          [
           251,
           251
          ],
          [
           255,
           255
          ],
          [
           259,
           259
          ],
          [
           263,
           263
          ],
          [
           267,
           267
          ],
          [
           271,
           271
          ],
          [
           275,
           275
          ],
          [
           279,
           279
          ],
          [
           283,
           283
          ],
          [
           287,
           287
          ],
          [
           291,
           291
          ],
          [
           295,
           295
          ],
          [
           299,
           299
          ],
          [
           303,
           303
          ],
          [
           307,
           307
          ],
          [
           311,
           311
          ],
          [
           315,
           315
          ],
          [
           319,
           319
          ],
          [
           323,
           323
          ],
          [
           327,
           327
          ],
          [
           331,
           331
          ],
          [
           335,
           335
          ],
          [
           339,
           339
          ],
          [
           343,
           343
          ],
          [
           347,
           347
          ],
          [
           351,
           351
          ],
          [
           355,
           355
          ],
          [
           359,
           359
          ],
          [
           363,
           363
          ],
          [
           367,
           367
          ],
          [
           371,
           371
          ],
          [
           375,
           375
          ],
          [
           379,
           379
          ],
          [
           383,
           383
          ],
          [
           387,
           387
          ],
          [
           391,
           391
          ],
          [
           395,
           395
          ],
          [
           399,
           399
          ]
         ]
        }
       }
      }
     },
     "packNote": "harness h5 (the bug fixed). P3 ('no detectable difference') held only because of an API outage and is declined: with the 25 outage episodes re-run, the cloud model is clearly better"
    },
    {
     "id": "rules",
     "label": "Hand-written rules",
     "where": "rules",
     "model": "The control: no model at all.",
     "brainVersion": null,
     "harness": "h5",
     "n": 200,
     "successes": 62,
     "rate": 0.31,
     "ci": [
      0.25,
      0.377
     ],
     "source": {
      "experiment": "EXP-W04",
      "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
      "file": "results/results.jsonl",
      "lines": [
       [
        1,
        200
       ]
      ],
      "sha256": "dff4707dd2d8f7d6cff4f29001cbe015248d5b26c46f23767be9ee8b937a7665",
      "pack": {
       "file": "evidence/lab01/telemetry/odin-telemetry-v1.json",
       "commit": "c8b36efd39192a904f0a97ff83935be018a633ba",
       "pointer": "/benchmark/16",
       "results": {
        "file": ".ai/experiments/EXP-W04-h5-named-args/results/results.jsonl",
        "commit": "96d6f7a3",
        "sha256": "4c7ef3e30415cc7db7ce755cfb24ef6c60e5f9321fbb979b1a19bfdc4cb5ec7e",
        "sha256Of": "the file as a Windows checkout reads it (CRLF line endings)"
       }
      }
     },
     "fallbacks": 0,
     "tokensPerTask": 0,
     "modelMsPerTask": 0,
     "breakdown": {
      "world": {
       "a": {
        "n": 40,
        "successes": 17,
        "rate": 0.425,
        "ci": [
         0.285,
         0.578
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           40
          ]
         ]
        }
       },
       "b": {
        "n": 40,
        "successes": 9,
        "rate": 0.225,
        "ci": [
         0.123,
         0.375
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           41,
           80
          ]
         ]
        }
       },
       "c": {
        "n": 40,
        "successes": 13,
        "rate": 0.325,
        "ci": [
         0.201,
         0.48
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           81,
           120
          ]
         ]
        }
       },
       "d": {
        "n": 40,
        "successes": 10,
        "rate": 0.25,
        "ci": [
         0.142,
         0.402
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           121,
           160
          ]
         ]
        }
       },
       "e": {
        "n": 40,
        "successes": 13,
        "rate": 0.325,
        "ci": [
         0.201,
         0.48
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           161,
           200
          ]
         ]
        }
       }
      },
      "kind": {
       "buy": {
        "n": 50,
        "successes": 15,
        "rate": 0.3,
        "ci": [
         0.191,
         0.438
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           4,
           4
          ],
          [
           8,
           8
          ],
          [
           12,
           12
          ],
          [
           16,
           16
          ],
          [
           20,
           20
          ],
          [
           24,
           24
          ],
          [
           28,
           28
          ],
          [
           32,
           32
          ],
          [
           36,
           36
          ],
          [
           40,
           40
          ],
          [
           44,
           44
          ],
          [
           48,
           48
          ],
          [
           52,
           52
          ],
          [
           56,
           56
          ],
          [
           60,
           60
          ],
          [
           64,
           64
          ],
          [
           68,
           68
          ],
          [
           72,
           72
          ],
          [
           76,
           76
          ],
          [
           80,
           80
          ],
          [
           84,
           84
          ],
          [
           88,
           88
          ],
          [
           92,
           92
          ],
          [
           96,
           96
          ],
          [
           100,
           100
          ],
          [
           104,
           104
          ],
          [
           108,
           108
          ],
          [
           112,
           112
          ],
          [
           116,
           116
          ],
          [
           120,
           120
          ],
          [
           124,
           124
          ],
          [
           128,
           128
          ],
          [
           132,
           132
          ],
          [
           136,
           136
          ],
          [
           140,
           140
          ],
          [
           144,
           144
          ],
          [
           148,
           148
          ],
          [
           152,
           152
          ],
          [
           156,
           156
          ],
          [
           160,
           160
          ],
          [
           164,
           164
          ],
          [
           168,
           168
          ],
          [
           172,
           172
          ],
          [
           176,
           176
          ],
          [
           180,
           180
          ],
          [
           184,
           184
          ],
          [
           188,
           188
          ],
          [
           192,
           192
          ],
          [
           196,
           196
          ],
          [
           200,
           200
          ]
         ]
        }
       },
       "deal": {
        "n": 50,
        "successes": 22,
        "rate": 0.44,
        "ci": [
         0.312,
         0.577
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           1,
           1
          ],
          [
           5,
           5
          ],
          [
           9,
           9
          ],
          [
           13,
           13
          ],
          [
           17,
           17
          ],
          [
           21,
           21
          ],
          [
           25,
           25
          ],
          [
           29,
           29
          ],
          [
           33,
           33
          ],
          [
           37,
           37
          ],
          [
           41,
           41
          ],
          [
           45,
           45
          ],
          [
           49,
           49
          ],
          [
           53,
           53
          ],
          [
           57,
           57
          ],
          [
           61,
           61
          ],
          [
           65,
           65
          ],
          [
           69,
           69
          ],
          [
           73,
           73
          ],
          [
           77,
           77
          ],
          [
           81,
           81
          ],
          [
           85,
           85
          ],
          [
           89,
           89
          ],
          [
           93,
           93
          ],
          [
           97,
           97
          ],
          [
           101,
           101
          ],
          [
           105,
           105
          ],
          [
           109,
           109
          ],
          [
           113,
           113
          ],
          [
           117,
           117
          ],
          [
           121,
           121
          ],
          [
           125,
           125
          ],
          [
           129,
           129
          ],
          [
           133,
           133
          ],
          [
           137,
           137
          ],
          [
           141,
           141
          ],
          [
           145,
           145
          ],
          [
           149,
           149
          ],
          [
           153,
           153
          ],
          [
           157,
           157
          ],
          [
           161,
           161
          ],
          [
           165,
           165
          ],
          [
           169,
           169
          ],
          [
           173,
           173
          ],
          [
           177,
           177
          ],
          [
           181,
           181
          ],
          [
           185,
           185
          ],
          [
           189,
           189
          ],
          [
           193,
           193
          ],
          [
           197,
           197
          ]
         ]
        }
       },
       "hand": {
        "n": 50,
        "successes": 10,
        "rate": 0.2,
        "ci": [
         0.112,
         0.33
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           2,
           2
          ],
          [
           6,
           6
          ],
          [
           10,
           10
          ],
          [
           14,
           14
          ],
          [
           18,
           18
          ],
          [
           22,
           22
          ],
          [
           26,
           26
          ],
          [
           30,
           30
          ],
          [
           34,
           34
          ],
          [
           38,
           38
          ],
          [
           42,
           42
          ],
          [
           46,
           46
          ],
          [
           50,
           50
          ],
          [
           54,
           54
          ],
          [
           58,
           58
          ],
          [
           62,
           62
          ],
          [
           66,
           66
          ],
          [
           70,
           70
          ],
          [
           74,
           74
          ],
          [
           78,
           78
          ],
          [
           82,
           82
          ],
          [
           86,
           86
          ],
          [
           90,
           90
          ],
          [
           94,
           94
          ],
          [
           98,
           98
          ],
          [
           102,
           102
          ],
          [
           106,
           106
          ],
          [
           110,
           110
          ],
          [
           114,
           114
          ],
          [
           118,
           118
          ],
          [
           122,
           122
          ],
          [
           126,
           126
          ],
          [
           130,
           130
          ],
          [
           134,
           134
          ],
          [
           138,
           138
          ],
          [
           142,
           142
          ],
          [
           146,
           146
          ],
          [
           150,
           150
          ],
          [
           154,
           154
          ],
          [
           158,
           158
          ],
          [
           162,
           162
          ],
          [
           166,
           166
          ],
          [
           170,
           170
          ],
          [
           174,
           174
          ],
          [
           178,
           178
          ],
          [
           182,
           182
          ],
          [
           186,
           186
          ],
          [
           190,
           190
          ],
          [
           194,
           194
          ],
          [
           198,
           198
          ]
         ]
        }
       },
       "meet": {
        "n": 50,
        "successes": 15,
        "rate": 0.3,
        "ci": [
         0.191,
         0.438
        ],
        "source": {
         "experiment": "EXP-W04",
         "commit": "158b75d116e3a4bb4dd6574a221d468f0b419390",
         "file": "results/results.jsonl",
         "lines": [
          [
           3,
           3
          ],
          [
           7,
           7
          ],
          [
           11,
           11
          ],
          [
           15,
           15
          ],
          [
           19,
           19
          ],
          [
           23,
           23
          ],
          [
           27,
           27
          ],
          [
           31,
           31
          ],
          [
           35,
           35
          ],
          [
           39,
           39
          ],
          [
           43,
           43
          ],
          [
           47,
           47
          ],
          [
           51,
           51
          ],
          [
           55,
           55
          ],
          [
           59,
           59
          ],
          [
           63,
           63
          ],
          [
           67,
           67
          ],
          [
           71,
           71
          ],
          [
           75,
           75
          ],
          [
           79,
           79
          ],
          [
           83,
           83
          ],
          [
           87,
           87
          ],
          [
           91,
           91
          ],
          [
           95,
           95
          ],
          [
           99,
           99
          ],
          [
           103,
           103
          ],
          [
           107,
           107
          ],
          [
           111,
           111
          ],
          [
           115,
           115
          ],
          [
           119,
           119
          ],
          [
           123,
           123
          ],
          [
           127,
           127
          ],
          [
           131,
           131
          ],
          [
           135,
           135
          ],
          [
           139,
           139
          ],
          [
           143,
           143
          ],
          [
           147,
           147
          ],
          [
           151,
           151
          ],
          [
           155,
           155
          ],
          [
           159,
           159
          ],
          [
           163,
           163
          ],
          [
           167,
           167
          ],
          [
           171,
           171
          ],
          [
           175,
           175
          ],
          [
           179,
           179
          ],
          [
           183,
           183
          ],
          [
           187,
           187
          ],
          [
           191,
           191
          ],
          [
           195,
           195
          ],
          [
           199,
           199
          ]
         ]
        }
       }
      }
     },
     "packNote": "harness h5 (the bug fixed). P3 ('no detectable difference') held only because of an API outage and is declined: with the 25 outage episodes re-run, the cloud model is clearly better"
    }
   ]
  }
 ]
}
