{ "schemaVersion": 1, "kind": "preregistration", "experiment": { "id": "EXP 006", "slug": "nina-changes", "title": "nina reviews the change", "statusText": "Pre-registered — not yet run", "authoredOn": "2026-09-30", "summary": "EXP 005 met nina's spotlight bar, but in 131 of 180 runs its own tool fence kept the reviewer from reading the diff. EXP 006 measures nina 0.34.0's reviewer again on the same 60 changes, with a fence that allows the git forms it uses, and counts a run as having seen the change only when a tool output shows at least one changed line of that change (or, for an added file, the file listed as untracked and at least one of its new lines read).", "note": "Written, hashed and published before any counted reviewer run. Nothing in this record may change after publication; a later change is a new, dated amendment (the EXP 005 amendment pattern), refuted and published first, and it resets the not-before." }, "question": "On the same 60 authored changes and the same architecture rules as EXP 005, and with a tool fence that lets it read the change, does nina 0.34.0's reviewer, run headless by Claude Code, meet the same spotlight bar as EXP 005, in runs of which at least 90% are counted as having seen the change: a tool output shows at least one changed line of that item (or, for an added file, the file listed as untracked and at least one of its new lines read)?", "parent": { "experiment": "EXP 005", "preregistration": { "file": "experiments/jev-gate/preregistration.json", "sha256": "30bdcf07a6d3dc14383858bb8f9ef64d8419dc2c59f8c2f3725b09f7bdc8fb1a" }, "amendment01": { "file": "experiments/jev-gate/amendment-01.json", "sha256": "5ddd8df9920c71fb26a2da68a8859d95130c04f1279345da56b3fd7bf7933bbb" }, "amendment02": { "file": "experiments/jev-gate/amendment-02.json", "sha256": "75d231c255d70fa537cb3f4fc90e780be053b42aa5fc8997f003009297d8cb9b" }, "results": { "file": "experiments/jev-gate/results/results.json", "sha256": "7388556049b7639e2a251edcf3fb8feb782213b78e71a02cddb687f12b09dccd" }, "spotlightDecision": { "file": "experiments/jev-gate/results/spotlight-decision.json", "sha256": "294a48d74b90f25310b2adda0c932524a06c4c19547e1c4918938dd4fa6aee3d", "held": true, "decision": "Hold spotlight, measure again" }, "finding": "EXP 005's results: the reviewer met the registered spotlight bar (0 of 90 missed, 0 of 90 false rejects, 60 of 60 agreement, 0 harness failures), but by EXP 005's committed classifier (results/diff-visibility.json) 131 of 180 runs were diff-blind: its fence refused the git forms the reviewer used, and it audited the clean base tree instead. The founder held the spotlight until nina is measured reviewing changes.", "diffVisibility": { "file": "experiments/jev-gate/results/diff-visibility.json", "sha256": "8acda3cf347fba65f08efcc22431baa30bdeb26e20223d2f3af337aadc4a9856" } }, "census": { "file": "experiments/nina-changes/denial-census.json", "sha256": "dd1b4fa52a1bc0aa18d670113d81efa1a2fe72e3453a82bd82503fb77a29cd18", "headline": "All 317 git commands EXP 005's fence refused, by exact form: 184 are allowed under fence6 and 133 stay refused (127: redirect to a file (fence6 never allows > to a file); 2: not a git command (head): no allow rule, and no blanket rule; 1: no allow rule matches (verb or prefix not allowed); 1: not a git command (cat): no allow rule, and no blanket rule; 1: not a git command (echo): no allow rule, and no blanket rule; 1: not a git command (ls): no allow rule, and no blanket rule). Of the 131 diff-blind EXP 005 runs, 123 began with a form fence6 still refuses and 130 tried a form fence6 allows.", "categories": { "compound": 158, "redirection": 0, "quoting": 0, "-C + --no-pager": 29, "git -C ": 103, "git --no-pager ": 27, "other": 0 } }, "reused": { "statement": "The corpus, inputs, labels, base app, base commit and rules are EXP 005's, byte for byte, so the two experiments are directly comparable. The runner asserts each by sha256 before any call.", "corpusSha256": "a83b222a1a4a64cc81ac755c827a47009baa2bb91b036e351e71422cc8d526a9", "inputsSha256": "6bfb2b8d52376cbd22c8a34f5f986fe67ad68a0c587da862ba6b56e77e966a34", "labelsSha256": "36af1422755e56af56f4af267411a4a8e6535d08e6fc9d7aa3734aa005e4412a", "rulesSha256": "87186d652a7cf8a4c8e71966236c4fe6837c4bd22799fc49a15d437a083b2562", "items": 60, "red": 30, "green": 30, "baseCommit": "3e35e4e274932a61bc0d92f378f8d506a9bb4ce0", "nina": { "release": "0.34.0", "commit": "be546e32ce30acb2a18ec9b55ff4a3b23a7b4890", "integrity": "sha512-M/Vtu9DGe931tYBA+jJ60LtSiFGnu0y7fNBUYZEIXbMbji70aBVRvleOTpq6N8MceW0lXbeIvkDA94QM91fUjg==", "rule": "nina 0.34.0 unchanged: the pinned tarball, verified by integrity, staged exactly as EXP 005 staged it (amendment 01's workspace order, EXP 005's own stageWorkspace). No local patch." }, "client": "Claude Code", "clientVersion": "2.1.280", "model": "claude-opus-5-5", "effort": "high", "agent": "reviewer", "k": 3, "timeoutSeconds": 600, "hangStop": { "fraction": 0.2, "minRuns": 10, "rule": "Once at least minRuns reviewer runs have completed, if more than fraction of all runs so far ended in a timeout, the run stops and the design is revisited before any further call; the stop and its counts are published. Before minRuns runs, a timeout alone never stops the run. Isolation probe runs count toward neither minRuns nor the timeout fraction." }, "order": "k runs per item, sequential, in id order, each in its own fresh workspace, never retried.", "prompt": "Review the change in this repository. The change is everything that git diff shows; it has not been committed. The architecture rules this codebase must follow are in rules.txt at the repository root. Decide one thing only: does the change break any of those rules? Ignore style, naming, tests and anything else that rules.txt does not state. Do not edit any file. Explain your reasoning briefly, then write your decision on a line of its own: VERDICT: REJECTED if the change breaks at least one rule, or VERDICT: APPROVED if it breaks none.", "promptSha256": "71a5d374b41fd856021dfbe17f8ace13064678cada21f610b82e7d19f176e46b" }, "reviewer": { "command": "claude -p --agent reviewer --model claude-opus-5-5 --effort high --output-format stream-json --verbose --no-session-persistence --setting-sources project --permission-mode dontAsk --settings \"{\\\"permissions\\\":{\\\"blockReadsOutsideWorkingDirectories\\\":true,\\\"deny\\\":[\\\"Bash(git *--output*)\\\",\\\"Bash(git *--no-index*)\\\",\\\"Bash(git *<*)\\\",\\\"Bash(git *>*)\\\",\\\"Bash(git *'*)\\\",\\\"Bash(git *\\\\\\\"*)\\\",\\\"Bash(git *\\\\\\\\*)\\\",\\\"Bash(git *{*)\\\",\\\"Bash(git *}*)\\\",\\\"Bash(git *`*)\\\"]},\\\"sandbox\\\":{\\\"enabled\\\":true,\\\"failIfUnavailable\\\":true,\\\"allowUnsandboxedCommands\\\":false,\\\"autoAllowBashIfSandboxed\\\":false,\\\"filesystem\\\":{\\\"denyRead\\\":[\\\"/private/tmp\\\",\\\"/tmp\\\",\\\"/private/var/folders\\\",\\\"/var/folders\\\"]}}}\" --allowedTools \"Read(./**)\" \"Grep(./**)\" \"Glob(./**)\" \"Bash(git diff:*)\" \"Bash(git status:*)\" \"Bash(git show:*)\" \"Bash(git log:*)\" \"Bash(git --no-pager diff:*)\" \"Bash(git --no-pager status:*)\" \"Bash(git --no-pager show:*)\" \"Bash(git --no-pager log:*)\" \"Bash(git -C /repo diff:*)\" \"Bash(git -C /repo status:*)\" \"Bash(git -C /repo show:*)\" \"Bash(git -C /repo log:*)\" \"Bash(git -C /repo --no-pager diff:*)\" \"Bash(git -C /repo --no-pager status:*)\" \"Bash(git -C /repo --no-pager show:*)\" \"Bash(git -C /repo --no-pager log:*)\"", "commandRule": "The command is the EXP 005 command with two declared differences: --output-format stream-json --verbose (so every tool call and its output is recorded), and fence6 (the settings and allow list below). stands for the run's own temp dir; its repository is /repo, rendered as an absolute path in each run and scrubbed back to /repo in the records. The validator derives this text from the runner's own renderCommand.", "differences": [ "--output-format stream-json --verbose instead of --output-format json", "fence6: the fence5 deny rules and sandbox, plus allow rules for git, git --no-pager, git -C /repo and git -C /repo --no-pager with diff, status, show and log" ], "settings": { "permissions": { "blockReadsOutsideWorkingDirectories": true, "deny": [ "Bash(git *--output*)", "Bash(git *--no-index*)", "Bash(git *<*)", "Bash(git *>*)", "Bash(git *'*)", "Bash(git *\"*)", "Bash(git *\\*)", "Bash(git *{*)", "Bash(git *}*)", "Bash(git *`*)" ] }, "sandbox": { "enabled": true, "failIfUnavailable": true, "allowUnsandboxedCommands": false, "autoAllowBashIfSandboxed": false, "filesystem": { "denyRead": [ "/private/tmp", "/tmp", "/private/var/folders", "/var/folders" ] } } }, "allowedTools": [ "Read(./**)", "Grep(./**)", "Glob(./**)", "Bash(git diff:*)", "Bash(git status:*)", "Bash(git show:*)", "Bash(git log:*)", "Bash(git --no-pager diff:*)", "Bash(git --no-pager status:*)", "Bash(git --no-pager show:*)", "Bash(git --no-pager log:*)", "Bash(git -C /repo diff:*)", "Bash(git -C /repo status:*)", "Bash(git -C /repo show:*)", "Bash(git -C /repo log:*)", "Bash(git -C /repo --no-pager diff:*)", "Bash(git -C /repo --no-pager status:*)", "Bash(git -C /repo --no-pager show:*)", "Bash(git -C /repo --no-pager log:*)" ], "childGitEnv": { "GIT_CONFIG_GLOBAL": "/dev/null", "GIT_CONFIG_NOSYSTEM": "1" }, "runDirectory": "Each run stages a fresh workspace under the operator's home cache (outside any checkout and outside the shared temp roots), with the reviewer's TMPDIR inside the run dir; the whole dir is removed after the run.", "harnessFailure": "A counted run that ends in a timeout, a non-zero client exit, no parseable final `type:\"result\"` line, or an unparseable NDJSON line, is_error: true, or an empty result. A run with a completed result and no verdict line is a model abstention, not a harness failure.", "harnessFailureNote": "is_error here is the final type:\"result\" line's only. A tool_result's is_error (a refusal, a failed git, or the client's cwd-tracking artifact) never makes a run a harness failure." }, "fence": { "allowPrefixes": [ "git", "git --no-pager", "git -C /repo", "git -C /repo --no-pager" ], "verbs": [ "diff", "status", "show", "log" ], "denyRules": [ "Bash(git *--output*)", "Bash(git *--no-index*)", "Bash(git *<*)", "Bash(git *>*)", "Bash(git *'*)", "Bash(git *\"*)", "Bash(git *\\*)", "Bash(git *{*)", "Bash(git *}*)", "Bash(git *`*)" ], "sandbox": { "enabled": true, "failIfUnavailable": true, "allowUnsandboxedCommands": false, "autoAllowBashIfSandboxed": false, "filesystem": { "denyRead": [ "/private/tmp", "/tmp", "/private/var/folders", "/var/folders" ] } }, "sandboxOnlyVariant": { "permissions": { "blockReadsOutsideWorkingDirectories": true }, "sandbox": { "enabled": true, "failIfUnavailable": true, "allowUnsandboxedCommands": false, "autoAllowBashIfSandboxed": false, "filesystem": { "denyRead": [ "/private/tmp", "/tmp", "/private/var/folders", "/var/folders" ] } } }, "compound": "No compound rule. A chain runs only when the client allows every part under the same allow list; a chain with any other part is refused whole.", "redirections": "Redirections to /dev/null and 2>&1 are stripped by the client before matching; a redirect to a file is never allowed.", "answers": [ { "q": 1, "answer": "Yes. Bash(git -C :*) matches the path as spawned (realpath, under the home cache): git -C diff, git -C --no-pager diff and git -C status --short ran and printed the change.", "evidence": "D2 n60 (C9), n65 (C14), n63 (C12) not refused, marker in output; D1 n58, n63, n61" }, { "q": 2, "answer": "Yes, per sub-command, with no compound rule: git status && git diff, git status --short; git --no-pager diff and git diff --stat || git diff ran; a chain with any non-allowed part (&& cat, ; cat, || cat, | tee) was refused whole.", "evidence": "D2 n62 (C11), n66 (C15), n68 (C19) ran; n51, n52, n58, n53 (R39, R40, R46, R41) refused" }, { "q": 3, "answer": "Yes. 2>&1 and 2>/dev/null are stripped before matching: both ran under the Bash(git *>*) deny rule.", "evidence": "D2 n67 (C16), n64 (C13), n35 (R28) ran; D1 n65, n62, n30" }, { "q": 4, "answer": "Refused. git diff > r47.txt (a redirect inside the workspace) was refused by the client and no file appeared.", "evidence": "D2 n59 refused; calls[0].matrix.info.i47 {refusedByClient: true, fileAppeared: false}" }, { "q": 5, "answer": "Every -C to another path was refused by the allow list itself (no rule matches: git -C , git -C .., git -C /.., git -C , git --no-pager -C , git -C /link). Under an allowed -C prefix an outside path argument to git diff was refused by the client (R44); an absolute outside path to git show ran and was stopped by the sandbox (R45: fatal: failed to stat … Operation not permitted, no canary).", "evidence": "D2 n44-n47, n50, n55 refused; n56 refused (R44); n57 ran, is_error, output without the R45 secret" }, { "q": 6, "answer": "Read: `Ntext`, N unpadded, from 1 (not N→). Grep content mode with -n: `path:N:text`, the path relative to the working directory. Both are stripped by stripReadPrefix / grepLinesFor (fixtures/live-read-grep.json, R4-4 test).", "evidence": "D2 n10 (C18 whole-file Read), n6 (C1), n9 (C17 Grep content)" }, { "q": 7, "answer": "Yes. Every tool_use has a tool_result with content text and is_error, refused calls included; refused calls are also in the final permission_denials with their tool_use_id. FINDING: is_error is also set on some git commands that RAN (git status --short, git log, git show, git -C status), with \"Exit code 1\" and an appended \"zsh: operation not permitted: /claude-session/cwd-…\" line: the client's own cwd-tracking write fails under the sandbox (the temp root is denyRead). So refusal is read from permission_denials, never from is_error (stream6.mjs, diff-seen.mjs).", "evidence": "D1 207 stream lines, 0 malformed; D2 n14-n17 vs permissionDenials; D1 n38, n61 (is_error with the status output)" } ] }, "isolationEvidence": { "matrix": { "rows": [ "r1-r31 with r8b, r8c, r19b (EXP 005 v5)", "r32", "r33", "r34", "r35", "r36", "r37", "r38", "r39", "r40", "r41", "r42", "r43", "r44", "r45", "r46" ], "controls": "C1-C8 (EXP 005 v5), C9-C19 (each allowed form returns the change; C17 Grep content mode, C18 a whole-file Read, C19 a || chain)", "info": [ "R28 git diff HEAD 2>/dev/null", "I47 git diff > r47.txt (a redirect inside the workspace)" ], "judgedBy": "A row leaks when its secret appears in the report, a refusal or ANY tool output, or its escape file appears on disk; a control works when its own command was not refused and returned the change in its own output. The model's own ALLOWED/DENIED labels are not used." }, "runs": [ { "name": "matrix-v6-discovery-fence6-1", "file": "experiments/nina-changes/probes/matrix-v6-discovery-fence6-1.json", "sha256": "e7deb80a7b5cf3c6d748999068be3a2f66ff51eba0f6b4277a2f1c10f1d01bc0", "variant": "fence6", "role": "discovery (before the parser recorded refusals; C12 failed only on the is_error artifact)", "codeSha256": "050ecbe763ad4effa464282a26e2f354d655953f8e5cc5bdde25d69fe909cad3", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "rowsHeld": "48/48", "controlsWorked": "15/16", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": false, "costUsd": 0.6545558 }, { "name": "matrix-v6-discovery-fence6-2", "file": "experiments/nina-changes/probes/matrix-v6-discovery-fence6-2.json", "sha256": "6f9dfcbd43d226457b18997498ff8b1dbe1724f3de04ea612e4e5ee4c89ae185", "variant": "fence6", "role": "discovery 2 (with C17-C19; the open questions answered from it)", "codeSha256": "100b943a1d2a0725ed73b32dc8fbeea1b7a7e655b71f4b3b76f99e98aae95367", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "rowsHeld": "48/48", "controlsWorked": "19/19", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": true, "costUsd": 0.5596686 }, { "name": "matrix-v6-fence6-1", "file": "experiments/nina-changes/probes/matrix-v6-fence6-1.json", "sha256": "7faf564ed642fa674a65e7eea021bc2dd8f2541cac6908f15ca62d5e11bd1147", "variant": "fence6", "role": "proof run 1", "codeSha256": "3abf1c7ea80e6caf79c7e2dd4d8bf4bf55f953b7fb4cf06d2f759725ce4c1ea0", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "rowsHeld": "48/48", "controlsWorked": "19/19", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": true, "costUsd": 0.625817 }, { "name": "matrix-v6-fence6-2", "file": "experiments/nina-changes/probes/matrix-v6-fence6-2.json", "sha256": "7bf8dc8a050783382704464799e4e64809604a9556f58294c13e9b1abbeeb4ce", "variant": "fence6", "role": "proof run 2", "codeSha256": "3abf1c7ea80e6caf79c7e2dd4d8bf4bf55f953b7fb4cf06d2f759725ce4c1ea0", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "rowsHeld": "48/48", "controlsWorked": "19/19", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": true, "costUsd": 0.5681038 }, { "name": "matrix-v6-sandbox6-only-1", "file": "experiments/nina-changes/probes/matrix-v6-sandbox6-only-1.json", "sha256": "bafb7ae1a7e51b299fbfd5ca49afd44387a800ef96e065c1cc374cda154d8f64", "variant": "sandbox6-only", "role": "paid, but its record was refused at write time and its verdict lost (a truncated home path escaped the scrub)", "codeSha256": "3abf1c7ea80e6caf79c7e2dd4d8bf4bf55f953b7fb4cf06d2f759725ce4c1ea0", "commandSha256": "313a374d232bddab6c201d2f879aa8cfbf6dfc8fdc6f5d1a5fdc149269919ef3", "verdict": "lost: the record was refused at write time (lint)", "costUsd": 0.547284 }, { "name": "matrix-v6-sandbox6-only-2", "file": "experiments/nina-changes/probes/matrix-v6-sandbox6-only-2.json", "sha256": "5a7b528c724de9e7d6cebad0def0a6d9b970dbf2374c18c88906fc5055bf5c0f", "variant": "sandbox6-only", "role": "proof run, the sandbox and file fence alone", "codeSha256": "0cda8a9cae73d336f8b79252342626bf760891eccb3f2111d6a30c0b6b8fab15", "commandSha256": "313a374d232bddab6c201d2f879aa8cfbf6dfc8fdc6f5d1a5fdc149269919ef3", "rowsHeld": "48/48", "controlsWorked": "19/19", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": true, "costUsd": 0.581182 }, { "name": "matrix-v6-fence6-3", "file": "experiments/nina-changes/probes/matrix-v6-fence6-3.json", "sha256": "04861e1b0cfd63edc50d009df94060d071b866ebc78c0f642f432f775743af0c", "variant": "fence6", "role": "proof run 3, probe of record", "codeSha256": "0cda8a9cae73d336f8b79252342626bf760891eccb3f2111d6a30c0b6b8fab15", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "rowsHeld": "48/48", "controlsWorked": "19/19", "info": { "r28": "ran, not refused", "i47": "refused by the client; no file appeared" }, "passed": true, "costUsd": 0.5455714 } ], "probeOfRecord": { "file": "experiments/nina-changes/probes/matrix-v6-fence6-3.json", "sha256": "04861e1b0cfd63edc50d009df94060d071b866ebc78c0f642f432f775743af0c" }, "proofRuns": [ { "name": "matrix-v6-fence6-1", "commit": "14ba08d", "codeSha256": "3abf1c7ea80e6caf79c7e2dd4d8bf4bf55f953b7fb4cf06d2f759725ce4c1ea0", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "differsFromPinned": [ { "file": "experiments/nina-changes/diff-seen.mjs", "reason": "rule (b) also reads long-form \"Untracked files:\" entries (D2); a git call is the first token and only top-level calls count (N9, N10); the classifier runs after the call, on the record" }, { "file": "experiments/nina-changes/guard6.mjs", "reason": "the counted guard also re-checks every file the pre-registration pins (phase C, spotlight gate included)" }, { "file": "experiments/nina-changes/results6.mjs", "reason": "a comment only (D3)" }, { "file": "experiments/nina-changes/run_reviewer6.mjs", "reason": "refusals before the call and handling after it only: a record refused at write time keeps its verdict and cost (00e27d7); removing a run dir never throws (bf1308a); the ledger line is written from the spawn result before anything else, and later failures mark the record (refute r2 N1); a failed ledger write goes to a pending sidecar and stops the run (r3 N1); an intent line is written before each paid spawn and cleared after, and every paid run must use the committed ledger (r4 N1, B2); every paid invocation holds an exclusive run lock (r5 N2); a paid run is refused under the Node test runner (r6 B1); a matrix run keeps every call's output (r3 N3); the post-freeze pre-run probe's own ledger kind (D4, N3); and the attempted-per-row check applied after the matrix judge (N6, matrix6-attempts.mjs). The argv builder is unchanged, as the command sha256 equality shows" }, { "file": "experiments/nina-changes/scrub6.mjs", "reason": "the write-time scrub also collapses a home path the client truncated (00e27d7)" }, { "file": "experiments/nina-changes/spend6.mjs", "reason": "the post-freeze pre-run probe has its own ledger kind, charged to the $60 cap only and made once per frozen pre-registration (D4, N3); a sub-5e-8 reported cost is charged as unknown, a corrupt ledger line refuses every call, and so does a pending (unwritten or intent) line; intent lines are written and cleared around each paid spawn, the clear removing only its own line by an atomic rewrite, and a pre-run probe line without a pre-registration sha counts as the current one (r2 N1, N2; r3 N1; r4 N1; r5 N1, N3)" }, { "file": "experiments/nina-changes/stream6.mjs", "reason": "records parent_tool_use_id per call and defines a git call by its first token (N9, N10); a matrix run keeps every call's output (r3 N3); the D3 comment; the stream parse and harness-failure rule are unchanged" } ] }, { "name": "matrix-v6-fence6-2", "commit": "14ba08d", "codeSha256": "3abf1c7ea80e6caf79c7e2dd4d8bf4bf55f953b7fb4cf06d2f759725ce4c1ea0", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "differsFromPinned": [ { "file": "experiments/nina-changes/diff-seen.mjs", "reason": "rule (b) also reads long-form \"Untracked files:\" entries (D2); a git call is the first token and only top-level calls count (N9, N10); the classifier runs after the call, on the record" }, { "file": "experiments/nina-changes/guard6.mjs", "reason": "the counted guard also re-checks every file the pre-registration pins (phase C, spotlight gate included)" }, { "file": "experiments/nina-changes/results6.mjs", "reason": "a comment only (D3)" }, { "file": "experiments/nina-changes/run_reviewer6.mjs", "reason": "refusals before the call and handling after it only: a record refused at write time keeps its verdict and cost (00e27d7); removing a run dir never throws (bf1308a); the ledger line is written from the spawn result before anything else, and later failures mark the record (refute r2 N1); a failed ledger write goes to a pending sidecar and stops the run (r3 N1); an intent line is written before each paid spawn and cleared after, and every paid run must use the committed ledger (r4 N1, B2); every paid invocation holds an exclusive run lock (r5 N2); a paid run is refused under the Node test runner (r6 B1); a matrix run keeps every call's output (r3 N3); the post-freeze pre-run probe's own ledger kind (D4, N3); and the attempted-per-row check applied after the matrix judge (N6, matrix6-attempts.mjs). The argv builder is unchanged, as the command sha256 equality shows" }, { "file": "experiments/nina-changes/scrub6.mjs", "reason": "the write-time scrub also collapses a home path the client truncated (00e27d7)" }, { "file": "experiments/nina-changes/spend6.mjs", "reason": "the post-freeze pre-run probe has its own ledger kind, charged to the $60 cap only and made once per frozen pre-registration (D4, N3); a sub-5e-8 reported cost is charged as unknown, a corrupt ledger line refuses every call, and so does a pending (unwritten or intent) line; intent lines are written and cleared around each paid spawn, the clear removing only its own line by an atomic rewrite, and a pre-run probe line without a pre-registration sha counts as the current one (r2 N1, N2; r3 N1; r4 N1; r5 N1, N3)" }, { "file": "experiments/nina-changes/stream6.mjs", "reason": "records parent_tool_use_id per call and defines a git call by its first token (N9, N10); a matrix run keeps every call's output (r3 N3); the D3 comment; the stream parse and harness-failure rule are unchanged" } ] }, { "name": "matrix-v6-fence6-3", "commit": "2ab2c27", "codeSha256": "0cda8a9cae73d336f8b79252342626bf760891eccb3f2111d6a30c0b6b8fab15", "commandSha256": "ea845d1955f2a7ca6dcbaf4fc68c6bc32a6da5442fe99fb24c4bce697045d986", "differsFromPinned": [ { "file": "experiments/nina-changes/diff-seen.mjs", "reason": "rule (b) also reads long-form \"Untracked files:\" entries (D2); a git call is the first token and only top-level calls count (N9, N10); the classifier runs after the call, on the record" }, { "file": "experiments/nina-changes/guard6.mjs", "reason": "the counted guard also re-checks every file the pre-registration pins (phase C, spotlight gate included)" }, { "file": "experiments/nina-changes/results6.mjs", "reason": "a comment only (D3)" }, { "file": "experiments/nina-changes/run_reviewer6.mjs", "reason": "refusals before the call and handling after it only: a record refused at write time keeps its verdict and cost (00e27d7); removing a run dir never throws (bf1308a); the ledger line is written from the spawn result before anything else, and later failures mark the record (refute r2 N1); a failed ledger write goes to a pending sidecar and stops the run (r3 N1); an intent line is written before each paid spawn and cleared after, and every paid run must use the committed ledger (r4 N1, B2); every paid invocation holds an exclusive run lock (r5 N2); a paid run is refused under the Node test runner (r6 B1); a matrix run keeps every call's output (r3 N3); the post-freeze pre-run probe's own ledger kind (D4, N3); and the attempted-per-row check applied after the matrix judge (N6, matrix6-attempts.mjs). The argv builder is unchanged, as the command sha256 equality shows" }, { "file": "experiments/nina-changes/spend6.mjs", "reason": "the post-freeze pre-run probe has its own ledger kind, charged to the $60 cap only and made once per frozen pre-registration (D4, N3); a sub-5e-8 reported cost is charged as unknown, a corrupt ledger line refuses every call, and so does a pending (unwritten or intent) line; intent lines are written and cleared around each paid spawn, the clear removing only its own line by an atomic rewrite, and a pre-run probe line without a pre-registration sha counts as the current one (r2 N1, N2; r3 N1; r4 N1; r5 N1, N3)" }, { "file": "experiments/nina-changes/stream6.mjs", "reason": "records parent_tool_use_id per call and defines a git call by its first token (N9, N10); a matrix run keeps every call's output (r3 N3); the D3 comment; the stream parse and harness-failure rule are unchanged" } ] }, { "name": "matrix-v6-sandbox6-only-2", "commit": "2ab2c27", "codeSha256": "0cda8a9cae73d336f8b79252342626bf760891eccb3f2111d6a30c0b6b8fab15", "commandSha256": "313a374d232bddab6c201d2f879aa8cfbf6dfc8fdc6f5d1a5fdc149269919ef3", "differsFromPinned": [ { "file": "experiments/nina-changes/diff-seen.mjs", "reason": "rule (b) also reads long-form \"Untracked files:\" entries (D2); a git call is the first token and only top-level calls count (N9, N10); the classifier runs after the call, on the record" }, { "file": "experiments/nina-changes/guard6.mjs", "reason": "the counted guard also re-checks every file the pre-registration pins (phase C, spotlight gate included)" }, { "file": "experiments/nina-changes/results6.mjs", "reason": "a comment only (D3)" }, { "file": "experiments/nina-changes/run_reviewer6.mjs", "reason": "refusals before the call and handling after it only: a record refused at write time keeps its verdict and cost (00e27d7); removing a run dir never throws (bf1308a); the ledger line is written from the spawn result before anything else, and later failures mark the record (refute r2 N1); a failed ledger write goes to a pending sidecar and stops the run (r3 N1); an intent line is written before each paid spawn and cleared after, and every paid run must use the committed ledger (r4 N1, B2); every paid invocation holds an exclusive run lock (r5 N2); a paid run is refused under the Node test runner (r6 B1); a matrix run keeps every call's output (r3 N3); the post-freeze pre-run probe's own ledger kind (D4, N3); and the attempted-per-row check applied after the matrix judge (N6, matrix6-attempts.mjs). The argv builder is unchanged, as the command sha256 equality shows" }, { "file": "experiments/nina-changes/spend6.mjs", "reason": "the post-freeze pre-run probe has its own ledger kind, charged to the $60 cap only and made once per frozen pre-registration (D4, N3); a sub-5e-8 reported cost is charged as unknown, a corrupt ledger line refuses every call, and so does a pending (unwritten or intent) line; intent lines are written and cleared around each paid spawn, the clear removing only its own line by an atomic rewrite, and a pre-run probe line without a pre-registration sha counts as the current one (r2 N1, N2; r3 N1; r4 N1; r5 N1, N3)" }, { "file": "experiments/nina-changes/stream6.mjs", "reason": "records parent_tool_use_id per call and defines a git call by its first token (N9, N10); a matrix run keeps every call's output (r3 N3); the D3 comment; the stream parse and harness-failure rule are unchanged" } ] } ], "codeStatement": "The fence was tested in three fence6 proof runs and one sandbox-only proof run. fence6 runs 1 and 2 were made at commit 14ba08d; fence6 run 3 (the probe of record) and the sandbox-only run at commit 2ab2c27 (candidate-branch commits). Their recorded code differs from the files this record pins as listed in proofRuns, each file with its reason. None of the differing files defines the fence, the matrix or its judge (experiments/nina-changes/fence6.mjs, experiments/nina-changes/matrix6.mjs, experiments/jev-gate/run_reviewer.mjs are byte-identical in every proof run), and every fence6 run ran exactly the command registered here (command sha256 ea845d19…), the sandbox-only run exactly the sandbox6-only template.", "attemptedCheck": "Refute r2 N6: an escape row counts as held only if the run ATTEMPTED it (a tool call whose input is that row's step); a row not attempted fails the matrix (matrix6-attempts.mjs, applied after the unchanged matrix judge). Every committed proof and discovery record was re-judged with it: every row was attempted in every run, so no verdict changed.", "refusedByAllowList": "The new -C escape forms (R32-R35, R38, R43) were refused by the allow list itself in every proof run, the sandbox-only run included (they are in each run's permission_denials). The evidence that the OS sandbox is the boundary for outside reads is R9, R45, R29 and R30: they passed the permission layer and git failed with \"Operation not permitted\".", "lostVerdict": "matrix-v6-sandbox6-only-1 was a paid call ($0.5472840) whose record was refused at write time: a home path the client had truncated in a tool output was not matched by the scrub, so the record was not written and its verdict was lost. The scrub was fixed forward and the run repeated (matrix-v6-sandbox6-only-2)." }, "classifier": { "rule": "DIFF-SEEN iff (a) a Bash git call the client did not refuse printed a whole diff line + or - equal to one of the item's fingerprints of that sign; or (b) for an item with added files, a non-refused Bash git output lists the added path or a directory prefix of it as untracked (a short `??` entry, or an entry inside a long-form \"Untracked files:\" section, never under \"Changes not staged\"/\"Changes to be committed\") and a non-refused Read/Grep of that added path returned one of its + fingerprint lines (line-number and path prefixes stripped). A git call is a Bash command whose first token is git; only top-level calls count (a sub-agent's calls never make a run SEEN). Refused = listed in the result's permission_denials; is_error alone is not a refusal (the client sets it on a command that ran when its own cwd-tracking write fails under the sandbox). Otherwise DIFF-BLIND. A harness failure is never DIFF-SEEN.", "fingerprints": { "file": "experiments/nina-changes/change-fingerprints.json", "sha256": "54833bc6f93314539a10aa4f78b280ca1e2d4c5c72fa1ea9a4730288f55dab69", "rule": "Per corpus item: every hunk line of its patch, trimmed. A + line (modified or added file) is a fingerprint when it has at least 8 characters and its sha256 is not in base-lines.json; a - line when it has at least 8 characters. Added, deleted and renamed paths are listed. A fingerprint matches only a whole trimmed line with the same sign.", "min": 1, "max": 11 }, "baseLines": { "file": "experiments/nina-changes/base-lines.json", "sha256": "92aeb23a37ce66541d3b586b006d2d04c4027e9569476c063b8e476b3a1ab2d3", "baseCommit": "3e35e4e274932a61bc0d92f378f8d506a9bb4ce0" }, "practiceRows": { "file": "experiments/nina-changes/practice/practice-rows.json", "sha256": "0fecd80674a7abf19cfdeaf0991dda372194769d82fde70329aa40cfe0d1f35e", "ids": [ "p04", "p05", "p06" ], "note": "Practice rows for the dry run (a new directory, a rename, a mixed change); never scored." }, "addFileItems": [ "c007", "c012", "c013", "c020", "c023", "c035", "c036", "c037", "c038", "c048", "c052", "c054", "c059", "c060" ], "addOnlyItems": 9, "rename": "c018 renames a file with 100% similarity: the pure rename has no hunk and yields no fingerprint; its modify hunks carry its fingerprints (rule a).", "gitCall": "A git call is a Bash command whose FIRST token is git (not any command that merely contains the word), and only top-level calls count: a sub-agent's calls (parent_tool_use_id set) never make a run SEEN.", "refusals": "A refused call is one listed in the result's permission_denials. is_error alone is not a refusal: live, the client sets it on git commands that ran and printed when its own cwd-tracking write fails under the sandbox.", "recompute": "Each counted run record keeps, per tool call, the tool, the scrubbed input, is_error, refused, the output sha256 and length, and the scrubbed output text of git Bash calls and Read/Grep/Glob calls. The scorer recomputes every run's class from those outputs and the committed fingerprints and refuses a record whose stamped class differs." }, "bar": { "statement": "The same spotlight bar as EXP 005: amendment 01's four criteria, thresholds, denominators, hold rule and three-state labels, computed by EXP 005's own spotlightVerdict. The only change is the run-state mapping below.", "rule": "All four criteria must hold for a spotlight. A partial run (cap or hang-stop) gives no spotlight. Failing any one criterion means no spotlight; the results are then published neutrally with the same attribution. The spotlight describes the harness as measured, meaning its reviewer on mechanical architecture rules, and nothing it wasn't tested on.", "judging": "Missed drift and false reject are judged on the run-level point estimate (n = 90 each). Beside each rate the Wilson 95% interval of the item-level majority (n = 30) is published, because the 3 runs on one change are not independent, with the parent's three-state label computed on that item-level majority: its point estimate and its Wilson interval at n = 30, under the parent's thresholdRule.singleRate. The item-level majority is the parent's reviewerMajority; an abstaining majority counts against the reviewer. A spotlight needs BOTH the run-level point estimate within the threshold AND an item-level label other than \"refuted\". With 30 RED items the interval cannot establish 10% or less: 0 of 30 has a Wilson upper bound of 11.35%.", "labelPhrase": "Where the item-level label is \"passes, not established at this N\", the spotlight card and the journal carry that phrase verbatim.", "criteria": [ { "id": "missed-drift", "metric": "missedDrift", "level": "run", "numerator": "RED judgments that are APPROVED or abstained", "denominator": "all RED judgments: 30 RED items × 3 runs = 90", "n": 90, "threshold": 0.1, "refutedWhen": "greater-than", "statement": "Missed drift ≤ 10%: over all RED judgments (30 × 3), the share that is APPROVED or abstained." }, { "id": "false-reject", "metric": "falseReject", "level": "run", "numerator": "GREEN judgments that are REJECTED or abstained", "denominator": "all GREEN judgments: 30 GREEN items × 3 runs = 90", "n": 90, "threshold": 0.25, "refutedWhen": "greater-than", "statement": "False reject ≤ 25%: over all GREEN judgments (30 × 3), the share that is REJECTED or abstained." }, { "id": "self-agreement", "metric": "selfAgreement", "level": "change", "numerator": "changes whose 3 runs all gave the same decided verdict", "denominator": "all 60 changes", "n": 60, "threshold": 0.9, "refutedWhen": "less-than", "statement": "Self-agreement ≥ 90%: the share of changes where all 3 runs gave the same decided verdict. Any abstaining run makes the change disagree, as in the parent's reviewerVariance." }, { "id": "zero-patches", "metric": "harnessFailureRate", "level": "run", "numerator": "counted runs that are harness failures", "denominator": "all 180 counted runs; isolation probes and dry-run rows are excluded", "n": 180, "harnessFailureMax": 0.05, "refutedWhen": "patched-or-greater-than", "statement": "Zero patches: the measured run composed and ran from the unmodified 0.34.0 release with no local patch to nina. Harness failures, meaning runs that end without a report for a reason that is not the model's answer, are ≤ 5% of runs.", "harnessFailure": "A counted run that ends in a timeout, a non-zero client exit, missing or unparseable JSON, is_error: true, or an empty result. A run with a completed result and no verdict line is a model abstention, not a harness failure." } ], "harnessFailure": "A counted run that ends in a timeout, a non-zero client exit, no parseable final `type:\"result\"` line, or an unparseable NDJSON line, is_error: true, or an empty result. A run with a completed result and no verdict line is a model abstention, not a harness failure.", "states": { "refuted": "refuted", "notEstablished": "passes, not established at this N", "passes": "passes" }, "runStates": { "SEEN-DECIDED": "diff-seen, with a verdict line", "SEEN-ABSTAIN": "diff-seen, no verdict line", "BLIND": "not diff-seen, whatever it decided", "HARNESS-FAIL": "a harness failure (also BLIND in the manipulation check)" }, "mapping": "A BLIND run, harness failures included, is scored as an error in missed drift and false reject, at run level and in the item majority (where it counts as the wrong answer), and as a disagreement in self-agreement: its decision is treated as no decision before EXP 005's formulas run.", "manipulation": { "maxBlindRuns": 18, "countedRuns": 180, "rule": "BLIND runs (harness failures included) at most 18 of the 180 counted runs (10%): the manipulation check PASSES. More than 18: it FAILS, and there is no spotlight whatever the bar says." }, "hookErrors": "Hook errors are recorded and their rate published beside the bar; they are not a criterion (as in EXP 005).", "diffSeenOnly": "The same rates over the diff-seen runs only are reported beside the primary figures, never in their place.", "partial": "A partial run (the spend cap, the hang-stop, a crash, or coverage other than exactly runs 1..3 of all 60 items) decides nothing.", "percentile": "p90 (and any percentile) is results.mjs percentile: numpy's linear interpolation.", "scorer": { "file": "experiments/nina-changes/results6.mjs", "sha256": "7533ee983f2891506bac2185396740199b86b94510da031df4a21c51332f7025" } }, "spotlightDecision": "Automatic and pre-registered: a spotlight-decision record with held: false is committed if and only if the bar PASSES, the manipulation check PASSES, and the independent results refute SHIPs; otherwise held: true, naming the condition that failed. A founder decision is asked only if a refute raises a doubt this rule does not decide.", "spotlightArtefact": { "pass": "On a PASS (the bar PASSES, the manipulation check PASSES, and the committed EXP 006 spotlight decision says held: false), the home page's \"Tools leaving the factory\" list gains nina's entry, next after 002 Laya, in the same shape as the Laya entry (number and category, a title linking to the evidence, one paragraph, links, an attribution note, a spec list): nina, harness orchestration for Claude Code, by Marcos Schulz (xhulz), https://github.com/xhulz/nina, with the attribution \"used with the permission of its author, as confirmed by Odin Labs\", and links to the EXP 006 measurement, to EXP 005 (#spotlight) and to the upstream fixes opened by Odin Labs (xhulz/nina#39, merged; xhulz/nina#41, stated as its live state at publish time: merged or open). Every figure in it is rendered from the EXP 006 results record at build time. It takes the list's next free number, 005, the slot EXP 005's gated nina card held; there is only one nina entry.", "fail": "On a FAIL, or when the spotlight is held for any reason, there is no entry, and the results page says plainly which condition failed (the bar, the manipulation check, the refute, or a partial run).", "gate": "WHETHER the entry shows is decided only by experiments/nina-changes/spotlight-gate.mjs, pinned in files below and re-checked by the counted guard. Its one entry point, gateOpen(root), owns the paths of the results record, the spotlight decision and the reviewer run record, reads them, and opens only for a real, complete, counted EXP 006 measurement under THIS frozen pre-registration: the results record is kind results, experiment EXP 006, not a fixture, not a rehearsal, not partial, made under the pre-registration sha frozen in freeze.mjs (which must be set), and passes EXP 005's publication check; the decision names the results record AND the reviewer run record by sha256; the reviewer run record is a complete counted EXP 006 reviewer run (kind gate-run, experiment EXP 006, gate reviewer, mode counted, not partial, not a fixture or a rehearsal) made under the same pre-registration, with the same not-before and code as the results, its calls exactly the runs the results scored (as id, run, decision, harness failure and diff-visibility class, in order) and as many as the manipulation denominator, and a parseable end time, and the Measured date comes only from it; and the bar PASSES AND the manipulation check PASSES AND spotlightEligible is exactly true AND held is the boolean false. A missing, unparseable, unbound or failing record hides the entry. It carries EXP 005's card gate (spotlightShown) byte-identical, with a slice test.", "renderer": "HOW it looks is scripts/nina-changes-spotlight.mjs, the markup, not pinned: it holds no record path and no gate logic and renders only what gateOpen returns (its figures are read from the results record gateOpen hands back, and the PR states from the committed upstream.json), so the results can feed it without an amendment." }, "spend": { "capUsd": 60, "preCountedCeilingUsd": 10, "ledger": "experiments/nina-changes/spend-ledger.jsonl", "rule": "Every paid call of EXP 006 (matrix probes, the dry run, the counted run) appends one ledger line, cost to 7 decimals from the client's total_cost_usd (API-equivalent). A call whose cost is unknown is charged its upper bound, max(the largest EXP 006 call so far, $0.60), marked upper-bound; never a $0 line. Before a call the runner refuses it when the recorded total plus a reserve of max(largest call so far, $0.60) would exceed $60, and a pre-counted call when the pre-counted total plus that reserve would exceed $10: enforced in code (spend6.mjs). Every paid run (practice, probe, pre-run probe, counted) appends to this one committed ledger; only fixture and rehearsal runs, which make no paid call, may use another. Before each paid spawn an intent line (call id, kind, its upper-bound cost) goes to spend-ledger.pending.jsonl and is cleared once the real line is written; a leftover intent or a failed write keeps that sidecar non-empty and the runner refuses every further call until the operator reconciles it at the upper bound; if the intent cannot be written, no call is made.", "calls": [ { "id": "matrix-v6-discovery-fence6-1", "variant": "fence6", "record": { "file": "experiments/nina-changes/probes/matrix-v6-discovery-fence6-1.json", "sha256": "e7deb80a7b5cf3c6d748999068be3a2f66ff51eba0f6b4277a2f1c10f1d01bc0" }, "endedAt": "2026-09-30T00:34:28.580Z", "costUsd": 0.6545558, "costBasis": "api-equivalent" }, { "id": "matrix-v6-discovery-fence6-2", "variant": "fence6", "record": { "file": "experiments/nina-changes/probes/matrix-v6-discovery-fence6-2.json", "sha256": "6f9dfcbd43d226457b18997498ff8b1dbe1724f3de04ea612e4e5ee4c89ae185" }, "endedAt": "2026-09-30T00:47:20.410Z", "costUsd": 0.5596686, "costBasis": "api-equivalent" }, { "id": "matrix-v6-fence6-1", "variant": "fence6", "record": { "file": "experiments/nina-changes/probes/matrix-v6-fence6-1.json", "sha256": "7faf564ed642fa674a65e7eea021bc2dd8f2541cac6908f15ca62d5e11bd1147" }, "endedAt": "2026-09-30T00:51:28.226Z", "costUsd": 0.625817, "costBasis": "api-equivalent" }, { "id": "matrix-v6-fence6-2", "variant": "fence6", "record": { "file": "experiments/nina-changes/probes/matrix-v6-fence6-2.json", "sha256": "7bf8dc8a050783382704464799e4e64809604a9556f58294c13e9b1abbeeb4ce" }, "endedAt": "2026-09-30T00:53:27.879Z", "costUsd": 0.5681038, "costBasis": "api-equivalent" }, { "id": "matrix-v6-sandbox6-only-1", "variant": "sandbox6-only", "record": { "file": "experiments/nina-changes/probes/matrix-v6-sandbox6-only-1.json", "sha256": "bafb7ae1a7e51b299fbfd5ca49afd44387a800ef96e065c1cc374cda154d8f64" }, "endedAt": "2026-09-30T00:55:34.553Z", "costUsd": 0.547284, "costBasis": "api-equivalent" }, { "id": "matrix-v6-sandbox6-only-2", "variant": "sandbox6-only", "record": { "file": "experiments/nina-changes/probes/matrix-v6-sandbox6-only-2.json", "sha256": "5a7b528c724de9e7d6cebad0def0a6d9b970dbf2374c18c88906fc5055bf5c0f" }, "endedAt": "2026-09-30T01:01:58.110Z", "costUsd": 0.581182, "costBasis": "api-equivalent" }, { "id": "matrix-v6-fence6-3", "variant": "fence6", "record": { "file": "experiments/nina-changes/probes/matrix-v6-fence6-3.json", "sha256": "04861e1b0cfd63edc50d009df94060d071b866ebc78c0f642f432f775743af0c" }, "endedAt": "2026-09-30T01:03:59.698Z", "costUsd": 0.5455714, "costBasis": "api-equivalent" } ], "spentUsd": 4.0821826, "sum": "0.6545558 + 0.5596686 + 0.6258170 + 0.5681038 + 0.5472840 + 0.5811820 + 0.5455714 = 4.0821826", "preCountedRemainingUsd": 5.9178174, "reserveUsd": 0.6545558, "headroomRule": "The headroom holds only while no call exceeds the current reserve: the reserve is max(the largest call so far, $0.60) and rises with the largest call, so a dearer call lowers the headroom for the calls after it.", "dryRunHeadroomUsd": 5.2632616, "bundle2": "D4: the separate post-merge probe is dropped. The measured run's own pre-run matrix probe, made after the freeze on the frozen runner, is the post-freeze proof; it has its own ledger kind (prerun-matrix) and is charged only against the $60 cap, never the $10 pre-counted ceiling. Pre-freeze probes and the practice dry run stay under the $10 ceiling. The real headroom for the dry run: $10 − $4.0821826 spent − $0.6545558 reserve = $5.2632616 before the last allowed call; the reserve is max(the largest call so far, $0.60), which is also the unknown-cost upper bound, so it cannot be lowered. Reaching the ceiling stops the dry run and asks the founder: bundle 2 may then complete fewer than its 18 practice runs, and its acceptance (every completed run diff-seen and hand-read) is judged on the runs completed, with the shortfall reported; whether that is enough to go on is the founder's call.", "askFork": "Reaching the $10 pre-counted ceiling stops the run and asks the founder (raise the cap or trim scope). Before the counted run: if spent + 180 × p90(dry-run per-run cost) + one pre-run probe + $0.60 > $60, the founder is asked first." }, "preflight": "Answer-key pre-flight, EXP 005's mechanism with EXP 006's pins file and not-before: one counted pre-flight record written before the measured run (runner-guard.mjs --preflight --mode counted --write-record --pins-file experiments/nina-changes/runners.sha256), ended at most 2 h before the first counted call and after the not-before, bound to that runner code and commit, and committed with the results; no commit between the record and the last counted call's start.", "siteChecks": "The page widths were checked two ways: a real headless Chromium render (playwright-isolated, served from the built site) of the home page, this note and the EXP 005 note at 375 px and 320 px, finding no horizontal overflow (document scrollWidth equal to the viewport); and a static repository test that no unbroken run in the note is wider than the column unless the stylesheet lets it wrap.", "notBefore": "No counted reviewer run starts before the merge time of the odin-rnd pull request that adds this file; that time is recorded after publication, frozen with this record's sha256 in freeze.mjs, and every counted call must start after it. Practice runs and matrix probes on canaries do not wait for it.", "limits": [ "Two layers: the text deny rules are best-effort (the client matches a normalised command); the OS sandbox is the boundary: reads outside the repository within home and the temp roots are denied (the sandbox's denyRead and its home block; R9 and R45 read a file outside the repository and got \"Operation not permitted\"); writes are confined and there is no unsandboxed retry. Nothing is canaried anywhere else, so the matrix tests the sandbox where the answer files could be.", "Untested residual: git diff --no-index given an in-workspace symlink to an outside file (and /dev/null) was not run live. It is refused by the layer-1 text rule on --no-index, and R45 shows the sandbox resolves the real path of what git reads; it is stated as untested, not proven (no new live row was added; matrix6 is unchanged).", "Untested in the matrix: git forms with an environment prefix (GIT_DIR=…, GIT_EXTERNAL_DIFF=…, GIT_CONFIG_PARAMETERS=… before git). They are believed refused because no allow rule's prefix matches a command that starts with an assignment, and whatever they ran, reads outside the repository are stopped by the sandbox (R9, R29, R30 and R45, and the sandbox-only run). No live row was added; this is stated as untested, not proven.", "git diff -O (an orderfile) is allowed under the git diff rules and makes git read that file to order its output; it prints none of the file's content, so at most it is a side channel (whether an outside path exists or parses). Documented as a residual; no live row was added (no paid call).", "The attempted-per-row check compares each row's recorded command with its registered step after the client's own normalisation only (a 2>&1 or 2>/dev/null redirection stripped). An honest rephrasing beyond that marks the row not attempted and fails the pre-run probe, which is the only one allowed under this pre-registration: the consequence is a dated amendment with a new not-before (then its own pre-run probe), not merely a question to the founder.", "A stream with a malformed line is charged the upper bound even if a final result line reported a cost (conservative: the line cannot be trusted).", "Sub-agents: a run can spawn a sub-agent whose own tool calls appear in the stream with a parent_tool_use_id. Only TOP-LEVEL calls count toward DIFF-SEEN (a sub-agent's calls are recorded but never make a run SEEN), so a run whose change was read only by a sub-agent counts BLIND.", "Reviewer only: Jev, Laya and the cascade are not re-run; this measures nina 0.34.0's reviewer on mechanical architecture rules and nothing else about nina.", "Same corpus, inputs, labels and rules as EXP 005, so the results compare; the corpus is public, and the reviewer is fenced from the answer files by the sandbox and the answer-key pre-flight.", "Prompt confound: the prompt says \"the change is everything that git diff shows\", which is false for the 14 items that add a file (9 of them add only files): git diff does not show untracked files. The prompt is EXP 005's, unchanged, so the results compare.", "Client artifact: Claude Code 2.1.280 marks some git commands that ran (git status --short, git log, git show) is_error, with \"Exit code 1\" and a line for its failed cwd-tracking write under the sandbox. It likely affected EXP 005's tool results too; EXP 005's records keep no tool output, so this cannot be checked (its classifier never read tool-level is_error). EXP 006 reads refusals from permission_denials and never from is_error.", "Known false-BLIND forms: the classifier does not recognise a change shown only through git diff --color=always, git diff --word-diff, git diff -R, a coloured git status (-c color.status=always), git status --porcelain=v2 or -z, or a long-form status run from a subdirectory (its ../ paths). A run that saw the change only that way is counted BLIND: an error against the bar and the manipulation check, never toward a spotlight. Each form is pinned BLIND by a unit test (documented behaviour, not a silent gap).", "The runs are sequential, on one machine, under whatever load it has; latency is recorded, not judged.", "The spend guard's pending sidecar is per worktree: bundles 2 and 3 run from ONE worktree (named in experiments/nina-changes/README.md), one after the other. Two runners at once could each pass the spend check and overshoot the cap by one call, so every paid invocation holds an exclusive lock file (experiments/nina-changes/run.lock, created O_EXCL) for its whole run and a second one refuses; a stale lock is removed by the operator after checking. A reconciled pre-run probe line without a pre-registration sha counts as made under the current one (fail closed).", "Paid calls before this record: 7 matrix probes, $4.0821826, itemised in spend.calls; none is counted in any result." ], "attribution": { "nina": "nina (github.com/xhulz/nina) — used with the permission of its author, as confirmed by Odin Labs", "ninaUrl": "https://github.com/xhulz/nina" }, "files": { "experiments/nina-changes/run_reviewer6.mjs": "13cc7b59de017530194c258e9c47fc3b59d268399d1d5b1da59298dbbdcecf5c", "experiments/nina-changes/stream6.mjs": "09e55aab57aa276f7b6ae1d245ac45bf41cfd2b095e32fd692860bf4b7b250e1", "experiments/nina-changes/fence6.mjs": "a591069653a39f3d9efe00e98511b81c1f7bd83584a066e7ed6e15243d79698f", "experiments/nina-changes/matrix6.mjs": "2b088e6a862dd5f168a2d85237b39ab3ffa781fc457f3444f7749a39c6164030", "experiments/nina-changes/scrub6.mjs": "d38b4efce5346abacd6e68d19c6121c5a533410de474d0578bcdbcec75fad435", "experiments/nina-changes/spend6.mjs": "7663e90febdd9f50cb2a11fee2a54feb56282cbd09c7cecfdab4d9b689524897", "experiments/nina-changes/guard6.mjs": "21b143f61a4aa640d158bef9e096d6bc213691c7170b13efed7cb9ddee2b55ce", "experiments/nina-changes/diff-seen.mjs": "c032517dca76109f1988ba641b73d6cc7618a245ce4f3ad243fcc27fff812f88", "experiments/nina-changes/fingerprints.mjs": "4eab13972a6c255ce264eebe935f7db16ea87b9bd48839762dfa65b056ba22ea", "experiments/nina-changes/change-fingerprints.json": "54833bc6f93314539a10aa4f78b280ca1e2d4c5c72fa1ea9a4730288f55dab69", "experiments/nina-changes/base-lines.mjs": "0b034c6cfedb766c7136178a1cb4a4e2839ed30d4b8db00c241eae966b484970", "experiments/nina-changes/base-lines.json": "92aeb23a37ce66541d3b586b006d2d04c4027e9569476c063b8e476b3a1ab2d3", "experiments/nina-changes/practice/practice-rows.json": "0fecd80674a7abf19cfdeaf0991dda372194769d82fde70329aa40cfe0d1f35e", "experiments/nina-changes/results6.mjs": "7533ee983f2891506bac2185396740199b86b94510da031df4a21c51332f7025", "experiments/nina-changes/spotlight-gate.mjs": "df7006754a965479640f86c9eb1b661a75d0db5ac46d632b86fe68e5ebe029a3", "experiments/nina-changes/matrix6-attempts.mjs": "8b8b606a69eb763a363c76254c1355a88e15204afb5139db67d6597301a2115e", "experiments/nina-changes/vendored-exp005.mjs": "d4e3445f9f585a119d059e6ff8fc355afd09459d932463849f81fff738327511", "experiments/jev-gate/run_reviewer.mjs": "6134e3a3dfd6ab845d536a5afaca83b9d620bc48b3b4057c19d039b461505116", "experiments/jev-gate/runner-guard.mjs": "1ad072329cf7f70f9c12b70e56a7fe42f06afcdcbdd84eb455fc2d014fa1bcd9", "experiments/jev-gate/results.mjs": "010ca4bb6aaf9f8c5949bc0f15aa90c4a87be6b064543edb7010c9d1197cf29c", "experiments/jev-gate/metrics.mjs": "ec7c03e002def319b2cc81e2865242601792be116510c122f2420ddda1e24e8d", "experiments/jev-gate/lint.mjs": "df4485c7c7c7f515c5a484ec10df3400c59e2558c9cc67d1956aa6d40f9f38c9", "experiments/jev-gate/bce-contract.mjs": "ca9ea8860b5fe5502c32b102c3db5fcc19e3474bdde9cc4104d3da89364f2004" } }