A new open-weight model, MiniMax M3.1 Flash, appeared on Ollama Cloud on 19 October 2026, free to use until 22 October 2026 23:59 UTC and priced at $0.15 per million input tokens and $0.60 per million output tokens after that. We wanted to know whether it can find real bugs, so we did not use a public benchmark it may have trained on. We used our own bug history: fixes from our private repositories. That code is not public, so it is unlikely to be in any training set, but we cannot verify what a model has or has not seen.
On the bugs that can be found from the code alone, the free model's first guess landed within two lines of the real fix 63% of the time. Our current default for offloaded work managed 23%; random guessing would score 6.5%. The larger finding came before any model ran: only about 1 in 10 of the small fixes we sampled could be found from the code alone.
The question in ordinary language
Give a model 120 lines of code as they were just before a real bug was fixed, with no commit message and no hint. Can it point at the line that was wrong? A deterministic scorer counts a hit when the answer is within two lines of a line the real fix changed, so no model grades another model on the headline number. A hit means the right place, not necessarily the right diagnosis; a separate blind check below looks at the diagnosis.
What we did
- Cases. Every commit from 1 May to 20 October 2026 in four of our private repositories whose subject starts with
fix, that touches exactly one non-test code file (TypeScript, JavaScript, Python or shell) with a small diff: one to three hunks, one to eight deleted lines, at most twenty added. That gave 357 cases, mined in two batches. - Excerpt. For each case, about 120 lines of the file before the fix, placed at random so the bug is not always in the middle. Only the file path and the numbered excerpt were sent.
- Privacy. 43 cases containing secret-shaped strings, IP addresses or internal host names were never sent to a model. In the first batch (81 cases) the scan ran after tagging, so 9 of them are still in the tag counts below: 2 unusable and 7 needing context, none local. In the second batch (276) the 34 were removed before tagging.
- Strata, tagged blind. Separate AI agents, working only from each fix and before any model answer existed, sorted the cases into local (a skilled engineer could find the bug from the excerpt alone), context (the fix depends on something outside the file) and unusable. Their rule was strict: when in doubt, context. Of 323 tagged cases (81 + 242), 27 were unusable, 266 needed context and 30 were local, so 30 of the 296 usable cases.
- Models. MiniMax M3.1 Flash, its predecessor MiniMax M3, and gpt-oss:120b, which is our current default for offloaded work. All ran on Ollama Cloud with thinking on, temperature 0 and one answer per case.
- Score. The primary metric was hit@1 on the local stratum: the model's first line falls within two lines of a line the real fix changed. The random baseline is the same window's chance for a uniformly random line, averaged per case.
Results
| Model | hit@1 | 95% Wilson interval | Right cause (blind check) |
|---|---|---|---|
| MiniMax M3.1 Flash | 19/30 · 63% | 46–78% | 17/30 |
| MiniMax M3 | 13/30 · 43% | 27–61% | 11/30 |
| gpt-oss:120b, reasoning “high” | 9/30 · 30% | 17–48% | not checked |
| gpt-oss:120b (our default) | 7/30 · 23% | 12–41% | 4/30 |
| Random line | 6.5% | — | — |
The comparison is paired: every model saw the same 30 cases. Flash found 13 bugs that gpt-oss:120b missed, and gpt-oss:120b found one that Flash missed (exact two-sided p = 0.002). Against gpt-oss:120b on its highest reasoning setting, the split was 13 against 3 (p = 0.02). Against its predecessor the difference is not significant at this size: 9 against 3 (p = 0.15), so we do not claim the new MiniMax beats the old one.
A second AI model checked each answer's explanation against the real fix without knowing which model had written it. Its finding agreed with the line score. No miss turned out to name the right bug. A few line hits were lucky: one for Flash (plus one partial) and three for gpt-oss:120b.
The context stratum was run on the first batch only, as the pre-registration's amendment fixed: its 59 sendable context cases, of which 58 were answered by all three models (one call failed). On those 58 cases that need context from outside the file, every model dropped: Flash to 14/58 (24%, interval 15–37%), gpt-oss:120b to 9/58 (16%) and MiniMax M3 to 8/58 (14%), against a 6.6% random baseline. That stratum was pre-registered as secondary.
At the post-promotion list price the 30 local cases would have cost about $0.24 with Flash, under a cent per bug. It thinks for minutes rather than seconds: the median answer took about four and a half minutes while several runs shared the account.
Why it matters
Measured: on private code, a free open-weight model localised most of the bugs that are visible in the code, and clearly beat the model we route offloaded work to today. Expected, not measured: that this carries over to reviewing whole changes. The follow-on study on our review bench tests that directly.
The 1-in-10 figure matters more than the model ranking. Most of the fixes we sampled hinge on something the file does not show: a tuned timeout, a name that must match another system, a policy decision. A better code reader does not close that gap. Context does.
Evidence and its boundary
This is an internally evidenced experiment. The code comes from private repositories, so per-case excerpts, fixes, tags and answers are not published; the counts above are. The plan was written down and hashed before the first model call, and amended once, after exactly one model answer had been seen, to add more cases because the first batch held only eight local ones. One tagging batch was found contaminated by a shared scratch file; it was set aside unscored and re-tagged in isolation. Model calls ran from 20 October 2026 22:30 UTC on https://ollama.com/api/chat.
What remains open
- A human pass on the 30 local cases. The tagger and the semantic judge are AI agents, spot-checked against commit subjects.
- Training exposure. Private code is a strong guard, but not a verified one.
- Larger samples. Thirty cases separate Flash from our default but not from its predecessor.
- Whole-change review rather than line localisation. The review-bench study measures that on a sealed corpus.
- Fixes that span several files. These were excluded by design here.
Local experiment. The protocol was locally frozen, not publicly pre-registered. Private source code; aggregates only. Prices as listed on ollama.com on 20 and 21 October 2026.