To tell whether an AI-written test can actually fail, make the code it guards wrong on purpose and run the test again. If it still passes, it is not protecting that behavior, however green the suite looks. A passing run only shows that the test and the current code agree.

You can do this by hand today: introduce one small bug on the line the test claims to guard, re-run only that test, confirm it fails on an assertion about behavior, then restore the code and confirm it is green again. Mutation testing tools such as Stryker, PIT and mutmut automate the same idea across many small changes.

TL;DR

  • Green is not evidence. A useful test fails when the behavior it describes is broken.
  • Fastest check: break the guarded line, re-run the one test, expect an assertion failure, restore.
  • A crash, import error or compile error is not the test catching the bug.
  • Inverting the expected value only proves the assertion runs, not that the test detects wrong code.
  • Mutation testing automates "break it and watch". Equivalent mutants mean 100% is not always reachable.

Why can a green AI-written test be worthless?

Because a test that passes on correct code may also pass on broken code. Coverage does not close the gap: PIT's documentation notes that line coverage measures only which code is executed, not whether tests can detect faults in it.

Patterns to look for in generated tests:

  • Shape-only assertions: not null, right type, no exception, but the value is never checked.
  • Expected values copied from the code: computed with the implementation's own formula, or pasted from what the current code printed. The test then agrees with the bug.
  • Mocking the unit under test: the assertion checks the mock, not the function.
  • Asserting a call, not an outcome: "save() was called" passes even if nothing was stored correctly.
  • Happy path only: one typical input, no boundary or rejection case.
  • Tests that never ran: skipped, filtered or uncollected tests are not passes. Check the reported count, not only the exit code.

How do I check that a test can fail without a special tool?

Make one deliberate bug in the production code the test is about, and require the test to fail on it.

  1. Start clean. Commit or stash your work so you can restore exactly. Note the test and the line it should guard.
  2. Break one thing on that line: >= to >, - to +, a different constant, or an early return of a wrong value. Keep the code compiling.
  3. Run only that test with your runner's normal test selection.
  4. Read the failure. It should be an assertion about behavior (wrong value, wrong status). A syntax or import error means the check never happened; make a smaller change.
  5. Restore and re-run. Confirm your diff is clean and the same test passes.
  6. Record it: test name, what you changed, the failure message, the version.

For a bug fix, run the new test against the commit before the fix instead; it should fail there and pass on the final diff. How to Verify an AI-Generated Bug Fix covers that red-then-green record.

Inverting the expected value (== 60 to == 61) is a weaker check. It catches tests that are skipped or never collected, but you changed the test, not the code, so it says nothing about whether a wrong implementation would be noticed.

A small teaching drill

Teaching only, not a Boloop test suite. The weak test checks shape; the strong test checks the value from the rule (25% off 80 is 60). Save as a file and run with python3.

def apply_discount(price, percent):
    return round(price * (1 - percent / 100), 2)

def apply_discount_mutant(price, percent):
    return round(price * (1 + percent / 100), 2)  # deliberate bug

def test_weak(fn):
    result = fn(80, 25)
    assert result is not None
    assert isinstance(result, float)

def test_strong(fn):
    assert fn(80, 25) == 60.0

def check(test, fn):
    try:
        test(fn)
        return "pass"
    except AssertionError:
        return "FAIL"

for name, fn in [("original", apply_discount), ("mutant", apply_discount_mutant)]:
    print(f"{name}: weak={check(test_weak, fn)} strong={check(test_strong, fn)}")

Output when run with Python 3.13 on 2026-11-08:

original: weak=pass strong=pass
mutant: weak=pass strong=FAIL

Both tests are green on correct code. Only one notices the bug.

When does mutation testing help, and when is it extra?

It helps when you want the break-it check applied to many lines, not just the one you picked. The tools insert small faults (mutants), run your tests against each, and report whether a test failed (killed) or all passed (survived).

  • [Stryker](https://stryker-mutator.io/docs/): StrykerJS (JavaScript/TypeScript), Stryker.NET (C#), Stryker4s (Scala).
  • [PIT](https://pitest.org/): Java and the JVM. It mutates bytecode; it runs from the command line, Ant or Maven, with Gradle via third-party integration.
  • [mutmut](https://mutmut.readthedocs.io/en/latest/): Python. mutmut run runs pytest and mutmut browse shows mutants. It needs fork support, so on Windows it runs inside WSL. Commit before applying a mutant to disk.

Reading results, per the same docs:

  • Survived means tests ran over the mutated code and none failed. No coverage means no test reached it at all.
  • Stryker leaves runtime-error and compile-error mutants out of the mutation score.
  • Equivalent mutants behave like the original, so no test can kill them. Stryker says there is no definitive way for it to find them. Do not chase 100%.
  • Cost: PIT recommends running frequently against only changed code; Stryker's FAQ suggests mutating only what is needed.

It is extra for one-off scripts, for a suite that does not run reliably yet, or when the risky logic is a few lines you can check by hand. Fix flaky and skipped tests first.

Who is this for?

Developers and reviewers accepting tests written by Claude Code, Codex, Copilot or a similar agent, especially tests added with a fix or a new rule.

It is not a testing strategy. It does not replace checks of the real user path, security review or performance testing. A test that can fail can still assert the wrong rule; the expected value must come from the agreed requirement.

How does this fit Boloop's other guides?

It expands one step. The AI-Generated Code Review and Acceptance Checklist lists "prove a test can fail" among four AI-specific checks; this page shows how. How to Verify an AI-Generated Bug Fix applies it as red-then-green. To have an agent design cases or prepare a red-test handoff, the test-case Skill separates those modes.

FAQ

Is high code coverage enough?

No. Coverage shows which lines ran; a test without a meaningful assertion still covers them.

Can the agent run the fail-check itself?

As a draft, yes. Then re-run it yourself on the final diff and read the failure. An agent's summary is not a test result.

My test failed with an import error. Does that count?

No. It never reached its assertion. Make a smaller change that keeps the code loading.

What mutation score should I aim for?

The docs read for this page set no universal target. Treat survivors in changed code as review items.

Sources

  1. Stryker docs: What is mutation testing?, Mutant states and metrics, Equivalent mutants, FAQ.
  2. PIT: home, Quickstart, Basic concepts, Mutators.
  3. mutmut documentation.

Limits

Boloop has not run Stryker, PIT or mutmut for this article and reports no mutation scores. Tool claims come from the official docs above, read on 2026-11-08; behavior can change between versions. The only code executed was the teaching drill, with the output shown.