Imagine a preferences screen with one switch: “Weekly email.” You turn it off, click Save, and see “Saved.” A browser test finds the toast and passes. Open the preferences again in a fresh tab, though, and the switch is back on. These two observations can coexist. The interface reported success, but the new value was missing on readback.
Evan X. Merz wrote about shallow tests and code that becomes hard to maintain as AI-generated changes accumulate. That is a useful concern to test on the next change. His account describes what he saw in his own work. It gives us no estimate of how often this happens elsewhere.
Ask three questions about the same save
- What did the user see? Use the ordinary preferences screen. Change one value whose old value you know. Observe whether the interface reports success, failure or an uncertain outcome.
- What can you read back? Reload or reopen the screen in a clean context that cannot reuse the edited form state. Check the exact value for the same user. If the product promises synchronization, check a second session only after the documented sync point.
- What happens when the reply is lost? In an authorized test environment, let the service store the change while withholding the response from the browser. Reopen the screen. If the value is already off, a retry of the same intent should not trigger a second promised side effect, such as sending another notification.
Separate the two failure cases
A request that fails before storage should leave the old value in place. A response that disappears after storage can leave the new value in place. Both can look like a timeout at the screen. The action you take next depends on which state you can read back. For a switch, setting the same value twice may be harmless. For a save that also sends email or creates another record, check that effect separately.
Capture the starting value, the final value, and what happened when you tried again. Ask the agent to explain which test would fail if the save silently stopped persisting. The reviewer can then reproduce the result from the same entry point. If you are reviewing more than this one path, use the AI-generated code acceptance checklist.
Sources
- AI usage generates a fragile codebase by Evan X. Merz, November 7, 2026.
- Discussion on Hacker News.
The preferences screen and the accompanying diagram are illustrative examples by Boloop.