Test Witness

The test passed.
The defect survived.

An agent can write a test that makes its own change look correct. Before accepting that work, ask the test to detect a mistake whose answer you already know.

Inspect the recorded evidence Explore a scoped pilot

One cent exposes the difference.

The fixture rounds 1.005 to two decimal places. The declared HALF_UP result is 1.01. We deliberately change the implementation so it returns 1.00.

Authored Java fixture · cases from the recorded experiment below
Candidate	What actually happens	Recorded evidence
Weak suite	`PASS · exit 0` Checks 4.00 = 4.00. Never checks the rounding case.	Inspect this case
Stronger suite	`FAIL · exit 1` Rounding assertion: actual 1.00, expected 1.01. The other four assertions pass.	Inspect this case
Always-pass suite	`PASS · exit 0` Prints all five expected values without observing the implementation. Rejected.	Inspect this case

Here, the red test is the useful result.

The strong suite detects the exact seeded fault. A generic failure would not be enough: a crash, missing assertion or contradictory summary cannot stand in for that detection. The acceptance check verifies assertion identities, values, coverage and process status.

These examples test the acceptance mechanism. They do not measure an agent’s performance on unseen work.

Inspect the recording.

Recorded fixture run; the execution environment is identified below. These Java examples, candidate tests and corrections were authored for this demonstration. Selecting a case reads saved evidence; it does not execute code or an agent.

Recorded fixture run · GitHub Actions · 2026-10-11T05:47:44.220Z

Run 29bb03f4-8bc4-4d79-9e8e-fc92a604d1f2 · base source revision 6a375f0b5f0c0c26c9563613e5aa0cbb808c94b9

Uncommitted working tree; base revision does not identify these edits. Input hashes identify executed sources.

GitHub Actions recording · attempt 1

Five authored invoice assertions and three seeded defects; acceptance of test evidence only.

Runtime and experiment boundaries

{
  "node": "v22.23.3",
  "java": "openjdk version \"21.0.12.1\" 2026-08-18 LTS\nOpenJDK Runtime Environment Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS)\nOpenJDK 64-Bit Server VM Temurin-21.0.12.1+1 (build 21.0.12.1+1-LTS, mixed mode, sharing)",
  "javac": "javac 21.0.12.1"
}

No generated tests or held-out performance evaluation.
No measured time savings or customer ROI.
Trusted local Java fixtures; temporary execution is not a security sandbox.
Hashes establish consistency, not malicious-tamper authenticity.

Raw execution record · Source hashes and recording manifest

defects-detected: 3 / 3 eligible seeded defects
Strong-suite assertion failures uniquely attributed to rounding, threshold or validation seeds; unresolved excluded.
unresolved: 2 / 12 executions
Missing-assertion and unknown crash controls retained as inconclusive, never detections.

Recorded case

rounding

rejected

Weak suite passes while omitting the assertion for the seeded defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 1,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    }
  ],
  "exitCode": 0
}

Actual command transcript

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-5: java -cp <workdir> Witness rounding weak
cwd: <workdir>
completed; exit 0; 47 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
SUMMARY|0

stderr:

Acceptance concerns the stated evidence check. An expected rejection can demonstrate that the check works; it does not mean the tested behavior passed. Fixture results are not customer savings, held-out performance or a guarantee of broader correctness.

All recorded observations, including negative controls

correct-strong · accepted

Correct implementation must pass every declared assertion

Expected

All five assertions pass

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "PASS",
      "actual": "1.01",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "PASS",
      "actual": "90.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "PASS",
      "actual": "rejected",
      "expected": "rejected"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-4: java -cp <workdir> Witness correct strong
cwd: <workdir>
completed; exit 0; 49 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|PASS|1.01|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|PASS|90.00|90.00
ASSERT|validation|PASS|rejected|rejected
SUMMARY|0

stderr:

rounding-weak · rejected

Weak suite passes while omitting the assertion for the seeded defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 1,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-5: java -cp <workdir> Witness rounding weak
cwd: <workdir>
completed; exit 0; 47 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
SUMMARY|0

stderr:

rounding-strong · accepted

Detection requires precisely the assertion ID assigned to the seeded defect

Expected

Only rounding assertion fails

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "FAIL",
      "actual": "1.00",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "PASS",
      "actual": "90.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "PASS",
      "actual": "rejected",
      "expected": "rejected"
    }
  ],
  "exitCode": 1
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-6: java -cp <workdir> Witness rounding strong
cwd: <workdir>
completed; exit 1; 52 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|FAIL|1.00|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|PASS|90.00|90.00
ASSERT|validation|PASS|rejected|rejected
SUMMARY|1

stderr:

threshold-weak · rejected

Weak suite passes while omitting the assertion for the seeded defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 1,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-7: java -cp <workdir> Witness threshold weak
cwd: <workdir>
completed; exit 0; 45 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
SUMMARY|0

stderr:

threshold-strong · accepted

Detection requires precisely the assertion ID assigned to the seeded defect

Expected

Only threshold assertion fails

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "PASS",
      "actual": "1.01",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "FAIL",
      "actual": "100.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "PASS",
      "actual": "rejected",
      "expected": "rejected"
    }
  ],
  "exitCode": 1
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-8: java -cp <workdir> Witness threshold strong
cwd: <workdir>
completed; exit 1; 53 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|PASS|1.01|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|FAIL|100.00|90.00
ASSERT|validation|PASS|rejected|rejected
SUMMARY|1

stderr:

validation-weak · rejected

Weak suite passes while omitting the assertion for the seeded defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 1,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-9: java -cp <workdir> Witness validation weak
cwd: <workdir>
completed; exit 0; 46 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
SUMMARY|0

stderr:

validation-strong · accepted

Detection requires precisely the assertion ID assigned to the seeded defect

Expected

Only validation assertion fails

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "PASS",
      "actual": "1.01",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "PASS",
      "actual": "90.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "FAIL",
      "actual": "allowed",
      "expected": "rejected"
    }
  ],
  "exitCode": 1
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-10: java -cp <workdir> Witness validation strong
cwd: <workdir>
completed; exit 1; 63 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|PASS|1.01|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|PASS|90.00|90.00
ASSERT|validation|FAIL|allowed|rejected
SUMMARY|1

stderr:

correct-always-pass · rejected

Unconditional suites cannot qualify: they ignore implementation behavior

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "PASS",
      "actual": "1.01",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "PASS",
      "actual": "90.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "PASS",
      "actual": "rejected",
      "expected": "rejected"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-11: java -cp <workdir> Witness correct always-pass
cwd: <workdir>
completed; exit 0; 47 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|PASS|1.01|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|PASS|90.00|90.00
ASSERT|validation|PASS|rejected|rejected
SUMMARY|0

stderr:

rounding-always-pass · rejected

Unconditional suites cannot qualify: they ignore implementation behavior

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "PASS",
      "actual": "4.00",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "PASS",
      "actual": "1.01",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "PASS",
      "actual": "99.00",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "PASS",
      "actual": "90.00",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "PASS",
      "actual": "rejected",
      "expected": "rejected"
    }
  ],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-12: java -cp <workdir> Witness rounding always-pass
cwd: <workdir>
completed; exit 0; 46 ms
stdout:
ASSERT|ordinary|PASS|4.00|4.00
ASSERT|rounding|PASS|1.01|1.01
ASSERT|below-threshold|PASS|99.00|99.00
ASSERT|threshold|PASS|90.00|90.00
ASSERT|validation|PASS|rejected|rejected
SUMMARY|0

stderr:

correct-always-fail · rejected

Unconditional suites cannot qualify: they ignore implementation behavior

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 5,
  "assertions": [
    {
      "id": "ordinary",
      "status": "FAIL",
      "actual": "unconditional-failure",
      "expected": "4.00"
    },
    {
      "id": "rounding",
      "status": "FAIL",
      "actual": "unconditional-failure",
      "expected": "1.01"
    },
    {
      "id": "below-threshold",
      "status": "FAIL",
      "actual": "unconditional-failure",
      "expected": "99.00"
    },
    {
      "id": "threshold",
      "status": "FAIL",
      "actual": "unconditional-failure",
      "expected": "90.00"
    },
    {
      "id": "validation",
      "status": "FAIL",
      "actual": "unconditional-failure",
      "expected": "rejected"
    }
  ],
  "exitCode": 1
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-13: java -cp <workdir> Witness correct always-fail
cwd: <workdir>
completed; exit 1; 52 ms
stdout:
ASSERT|ordinary|FAIL|unconditional-failure|4.00
ASSERT|rounding|FAIL|unconditional-failure|1.01
ASSERT|below-threshold|FAIL|unconditional-failure|99.00
ASSERT|threshold|FAIL|unconditional-failure|90.00
ASSERT|validation|FAIL|unconditional-failure|rejected
SUMMARY|5

stderr:

correct-missing · inconclusive

Execution or assertion evidence missing; never a detected defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 0,
  "assertions": [],
  "exitCode": 0
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-14: java -cp <workdir> Witness correct missing
cwd: <workdir>
completed; exit 0; 27 ms
stdout:
No assertions executed

stderr:

correct-crash · inconclusive

Execution or assertion evidence missing; never a detected defect

Expected

Cannot qualify as an adequate test suite

Actual

{
  "assertionCount": 0,
  "assertions": [],
  "exitCode": 1
}

Execution receipt

command-3: javac -d <workdir> <repo>demos/test-witness/fixtures/Witness.java
cwd: <workdir>
completed; exit 0; 644 ms
stdout:

stderr:


command-15: java -cp <workdir> Witness correct crash
cwd: <workdir>
completed; exit 1; 26 ms
stdout:

stderr:
Exception in thread "main" java.lang.RuntimeException: unknown execution failure
	at Witness.main(Witness.java:14)

Reproduce on your machine.

From the repository root, with Node 22 and JDK 21 installed. Runs trusted bundled fixtures locally.

node demos/test-witness/run.mjs --out /tmp/test-witness.json

Read the demo source and limitations

Bring one decision to the bench.

A proposed pilot for a team deciding whether this evidence would improve its engineering review.

Proposed delivery cap: 20 engineering hours for this bounded scope, subject to input review and agreement. A planning hypothesis, not a service promise or measured saving.

Prepare a pilot brief Download Test Witness proposal

Good fit: A small Java module whose behavior can be expressed as concrete examples.
You bring: A trusted, runnable module; existing tests; approved expected values and representative defect scenarios.
You take away: An acceptance suite, seeded-defect comparison and a portable report showing what the tests detect and miss.
Acceptance to agree: Correct code passes; each agreed seed triggers its specified assertion; missing checks and unexplained failures remain visible.
Outside this pilot: Automatic test generation, exhaustive correctness, untrusted code execution and production changes.

Measure whether it earns its place.

Agree a comparable baseline, acceptance criteria and a stop condition before work starts. Record setup, execution, human review and rework time, plus tool costs and the full cost of labor. Compare total effort and accepted outcomes; disclose unresolved work.

The demo establishes a mechanism to inspect. Whether it reduces your team's cost or risk is a pilot question. Agree scope, fees, access, data handling and delivery terms before sharing code or data. The brief is a manual preparation step; nothing is sent automatically.

Which mistakes must your agent-generated tests be able to catch?

Bring one component and the failures your reviewer would refuse to ship. We can build the acceptance check around those examples, compare it with your current suite, and record both missed defects and reviewer effort.

The proposed work is the acceptance system around your workflow: an explicit contract, checks that can reject a plausible but wrong result, and a record another engineer can reproduce. The small Java example above makes one part of that approach inspectable.

Discuss your acceptance problem

Part of a larger engineering problem.

When agents produce more changes, someone still has to decide which changes deserve acceptance. Odin explores that decision from the rule to the observed outcome.

Blueprint Conformance Engine: Published architecture constraints checked against code. Public, reproducible examples.
A passing pipeline was not yet a verified release: Our internal release account follows coverage artifacts, a qualified image and a user journey. Author-reported; its private receipts are not independently inspectable here.
These Witness experiments: Public adversarial examples: tests that miss defects, matching implementations that share a bug, and corrections that lose required checks.

Test Witness

The test passed.The defect survived.

One cent exposes the difference.

Here, the red test is the useful result.

Inspect the recording.

rounding

Expected

Actual

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Expected

Actual

Execution receipt

Reproduce on your machine.

Bring one decision to the bench.

Measure whether it earns its place.

Which mistakes must your agent-generated tests be able to catch?

Part of a larger engineering problem.

The test passed.
The defect survived.