How to Verify an AI-Generated Bug Fix

Verify an AI-generated fix by showing that the original failure is detectable before the change, disappears after it, and stays absent on affected paths. Record the exact version, inputs, observations, and remaining gaps. A successful build is useful evidence; it does not establish that the reported bug is fixed.

Red, then green (or pause)

Verification is ordered evidence, not a green CI screenshot:

1. Red first: On the known-bad version (or an isolated replay of the defect), the focused check must show the reported failure.
2. Green second: On the repaired final diff, the same check and the original user path must pass without weakening the assertion.
3. Affected paths: Regress only what the cause can break; record anything still NOT RUN.
4. Pause: If you cannot produce the red signal safely, stop at diagnosis — do not claim a verified repair. See the section “When verification fails or cannot run” below.

For locating and repairing the first broken rule under explicit authorization, use the Root-Cause Bug Fixing Skill. For accepting a feature (not only a bug claim), use the AI-Generated Code Acceptance Checklist.

Start with one testable claim

Write the expected behavior before evaluating the patch: “After the permitted user saves this field and reopens the record, the saved value is returned.” State the role, normal entry point, input, environment, and version. If the expected rule is disputed, resolve that product decision first.

Keep the scope explicit: diagnosis, verification, code repair, test editing, historical-data repair, and deployment are different actions. A verification request does not authorize all of them. Read the repository's instructions and inspect test-command side effects before running anything.

1. Preserve the failure before the fix

Save the original steps and the actual result. Run a check that asserts the reported symptom, not merely that a response exists. For a browser failure, start from the user's normal page and role; an internal API call alone cannot replace that journey. For an API or background-job failure, use the relevant execution boundary without adding an irrelevant browser step.

Where safely available, run the same regression check against the known-bad version in an isolated environment. Keep the command, version, inputs, expected result, and output. A reduced test can aid diagnosis, but retain the unreduced scenario for final verification. Do not run destructive historical behavior against production just to obtain a red test.

2. Connect the repair to the cause

Trace the actual path: entry, validation or transformation, storage or external operation, readback, and user-visible result. Identify the first boundary that breaks the agreed rule. A plausible explanation or changed line is a hypothesis until the observations support it.

Inspect the final diff for a repair at the responsible layer. Check other effects of the same cause and other places using the same faulty pattern. Avoid unrelated refactoring. A code fix does not itself repair existing bad records; keep historical-data correction separately scoped and authorized.

3. Replay the original path after the fix

Use the repaired version with the same relevant inputs, role, and conditions. Record any necessary differences. Rerun the original complete scenario and the regression check. For saved data, refresh or reopen and compare against the authoritative readback; a success toast is insufficient. For rejected operations, verify that no prohibited write or external side effect occurred.

Do not report a local result as deployed. Keep implementation, local verification, deployment, and acceptance on the live path distinct. If the patch changes after review, rerun affected checks against the final version.

4. Regress the paths the cause can affect

Select checks from the mechanism, not a fixed number of cases:

• Shared validation: valid input, the relevant boundary, and rejection behavior
• Permissions: the allowed role and applicable denied role or resource, including side effects
• Persistence: save, readback, reopen, and applicable stale-state behavior
• Async behavior: relevant retry, cancellation, duplicate, or out-of-order paths
• Shared callers: other entry points using the changed rule

Reuse evidence only when the implementation, data binding, and failure conditions are genuinely equivalent. Similar-looking pages do not establish equivalence. Keep independent branches uncovered unless checked. Run required lint, build, integration, or end-to-end checks according to the repository and change impact. Review the latest diff; disclose missing independent review.

A synthetic walkthrough: save succeeds, old value returns

This is an invented teaching scenario, not a Boloop incident or an executed test.

Assume the confirmed requirement says an authorized profile edit must persist. Before the patch, the planned reproduction is: open the profile through the normal route, change the field, save, refresh, and compare the returned field. Capture the request and authoritative readback for the same record and timeline. Do not assume caching is the cause merely because refreshing exposes the symptom.

After a proposed repair, repeat those steps. Add the relevant rejected-save check and another caller of the changed rule if one exists. Leave every result NOT RUN until observed. This example supplies a verification plan, not a diagnosis, patch, or success claim.

Executable teaching drill (cursor tie-break)

Teaching only — not a Boloop production incident and not a run against your repository. Copy the Python below into a local file and run python3 on that file. It covers a sorted-cursor tie: the known-bad implementation fails the assertion; the repaired one passes the same assertion.

Expected stdout from this teaching run (captured 2026-11-06):

known-bad: RED [(101, 43)]
repaired: GREEN
TEACHING_DRILL_OK

"""Teaching drill: red-then-green on a sorted-cursor tie.

Not a production test and not a run against any live application.
Copy into a local file and run: python3 <that-file>
"""
from __future__ import annotations


def next_page_buggy(rows, after, limit):
    # Broken cursor: compares only the first field, drops ties incorrectly.
    ordered = sorted(rows)
    if after is not None:
        t, _ = after
        ordered = [row for row in ordered if row[0] > t]
    return ordered[:limit]


def next_page_fixed(rows, after, limit):
    ordered = sorted(rows)
    if after is not None:
        ordered = [row for row in ordered if row > after]
    return ordered[:limit]


def test_tie_keeps_next_id(fn) -> None:
    rows = [(100, 41), (100, 42), (101, 43)]
    result = fn(rows, after=(100, 41), limit=1)
    assert result == [(100, 42)], result


def main() -> None:
    try:
        test_tie_keeps_next_id(next_page_buggy)
        raise SystemExit("expected red on buggy")
    except AssertionError as e:
        print("known-bad: RED", e)

    test_tie_keeps_next_id(next_page_fixed)
    print("repaired: GREEN")
    print("TEACHING_DRILL_OK")


if __name__ == "__main__":
    main()

Keep the expectation derived from the ordering rule, not from the buggy implementation’s output. If the red step cannot be shown safely, pause and do not claim a verified repair.

A real Boloop case: a six-second timeout looked like a Google failure

On October 20, 2026, Boloop's analytics email could read sitemap information but could not return the six page-indexing checks. The honest UI said the page checks were unavailable. That was a reporting failure, not evidence that Google had indexed zero pages.

The first question was where the request stopped. The same read-only account and the same six URLs were used for the comparison:

• With the original deadline, all six requests were cancelled after roughly 6,002–6,011ms with AbortError.
• In a temporary diagnostic run with a longer deadline, the same six requests returned HTTP 200 after roughly 7,042–7,907ms.

Those observations supported a local timeout as the immediate cause. They did not prove that every earlier provider failure had the same cause. There was no evidence from this comparison that broader Google permissions were needed.

The repair was deliberately narrow. URL Inspection received its own 15-second deadline. Token requests, GA4, search-performance reports and sitemap reads kept their six-second deadline. The six inspections remained parallel; response-size limits, partial results, caching and read-only scopes stayed in place. No retry loop or indexing submission was added.

Before calling the repair complete, the checks covered three different outcomes:

• An eight-second inspection response now returns all six page results.
• A request that never returns is still cancelled at fifteen seconds, while the successful sitemap result remains available.
• Other reporting requests still stop at six seconds.

These are controlled automated regressions with simulated provider responses and a test clock. They test timeout behavior; they do not measure Google's current response time. The eight-second regression failed against the earlier implementation and passed after the repair. It was rerun for this write-up along with the stalled-request check.

The real-provider readback was separate: the repaired call returned all six results in 8,188ms. After the authorized deployment, the normal production analytics screen also displayed all six rows. The English homepage was indexed; the remaining fixed URLs were reported as unknown to Google at that time. Fixing the report did not itself get those pages indexed.

This case closes a bounded claim: the short local deadline incorrectly rejected valid responses in the observed runs, and the dedicated deadline restored those checks. It does not establish a general speed improvement, guarantee future provider availability or prove a successful scheduled email after deployment. Those outcomes require their own observations.

To reuse the method, write down the original symptom, the first boundary that fails, the before/after evidence and the nearby failure paths. Keep local regression, real-provider readback and production acceptance separate in the result template below.

When verification fails or cannot run

Pause rule: no safe red signal ⇒ no “verified fix” claim. Keep diagnosis, evidence gaps, and the next authorized action explicit.

• Original symptom still present: record FAIL and return to diagnosis; do not weaken the expected result to make the test pass
• Missing access, data, or environment: record BLOCKED with the exact prerequisite and next action
• A planned check not attempted: record NOT RUN, not PASS or N/A
• Intermittent failure: record attempts, failures, timing conditions, and uncertainty; a clean sample does not prove absence
• Original reproduction unavailable: keep the root cause provisional, preserve available logs and inputs, and seek independent supporting evidence. Without a regression signal sensitive to the defect, stop at diagnosis rather than claiming a verified repair

Mocks can isolate a mechanism but cannot establish an unobserved real-system outcome. Do not substitute synthetic acceptance data without permission. Redact secrets and personal data from evidence before sharing it.

Stop when the evidence supports the scoped claim

Close verification only when the original scenario and applicable affected paths have matching evidence for the final version, required checks and review are complete, and no unresolved defect invalidates the claim. Record any remaining limits and the shortest rollback. Passing a subset supports only that subset.

If an important prerequisite is missing, stop dependent work and say what would unblock it. If evidence is complete, stop adding repetitive checks that cannot change the conclusion. The goal is a bounded, defensible result, not an endless green-test count.

Record the result and take the next step

Use the companion Markdown result template to record before/after evidence, regression results, review, status boundaries, and rollback.

For an existing failure, start with $bug-fixing. For an independent read-only review of the final diff, use $code-review-multiagent. For authorized validation against confirmed requirements, choose the explicit $test-case mode. These are workflow instructions; they supply neither permissions nor execution tools.

Next: fill in the expected behavior, known-bad version, and original replay steps before asking an agent to assess the repair.

Related guide: after the bug claim is verified (or paused), use AI-Generated Code Review and Acceptance Checklist when you still need feature-level acceptance, persistence readback, or a merge decision beyond this defect.

Skill pages: bug-fixing · code-review-multiagent · test-case

Download the Markdown template