Playbook: Testing AI-Generated Code

Objective

Leave every agent-built change with tests that would fail if the bug came back — not a green suite that only proves mocks were wired. Agents write tests eagerly; most of that output needs a human quality bar, not more prompting volume.

When to use

During or immediately after implementation PRs from Claude Code (or any agent). Also before refactors (Legacy Refactor) and as the verification half of Add a Feature. For gate wiring itself, see Quality Gates.

Inputs required

• The spec or bug report (acceptance criteria / repro).
• Commands: unit, integration, make check.
• Known risky boundaries (DB, clock, payments, authz).
• Explicit ban list from CLAUDE.md (e.g. no network in unit tests).

Step-by-step process

1. Decide the test pyramid for this change

Change type	Prefer	Avoid as sole proof
Pure domain logic	Fast unit tests with real examples from the spec	Snapshot of implementation details
HTTP handler	Request/response tests through the real router stack	Mocking the handler internals
DB query	Integration test against real DB (or ephemeral test DB)	Mocked query builder assertions only
Legacy before edit	Characterization of current behavior	Aspirational tests that fail on main
Bug fix	Regression test that fails before the fix	"Added coverage" that never failed

Write this choice into the prompt so the agent doesn't default to mock-heavy unit tests for everything.

2. Demand failing tests first for bugs and new acceptance criteria

For <bug or acceptance criterion>:

1. Add a test that fails on current main for the right reason.
2. Paste the failing output.
3. Stop. Do not implement the fix until I confirm the failure mode is correct.

No production changes in this step except a minimal test harness hook if required
(ask first).

If the agent can't get a red test, it doesn't understand the bug yet — don't let it "fix" forward.

3. Prompt for tests that encode the spec

Add tests for specs/<nnn>-*.md acceptance criteria <list>.

Requirements:
- Each acceptance criterion maps to at least one test by name or comment.
- Exercise real boundaries for <DB/HTTP/auth> as applicable.
- Do NOT write mock-only tests that only assert mock call counts.
- Do NOT test private implementation details that would break under refactor.
- Prefer examples from the spec's examples section.

Paste `make check` output when green.

4. Review tests like production code

Use Reviewing AI Tests. Quick reject table:

Pattern	Why reject	Replace with
Assert mock called with expect.any(Object)	Tautology	Assert meaningful fields / outcomes
90% coverage of getters	Metric theater	Branch on risk
Snapshot of entire HTML/JSON blob	Brittleness + hidden intent	Targeted assertions
Same happy path three ways	Padding	One happy + real edge cases from spec
Test duplicates production bug	Copy-paste of wrong logic	Independent expected values from spec

5. Separate builder and reviewer when stakes are high

Same session that wrote the code will defend weak tests. For payment/auth/data-loss paths:

Review the test files in <paths> as a skeptical reviewer.
You did not write them. Goal: find tests that would stay green if the
production bug returned. List each weak test with a concrete improvement.
Do not edit yet.

Then a second pass to apply fixes — builder/reviewer split, Multi-Agent Patterns.

6. Wire evidence into the PR

Require pasted commands in the PR body. CI green is necessary; local evidence plus your spot-run of one critical test is the bar for agent PRs. See Verification Strategy.

Human review points

1. Red-then-green for bug fixes — you saw the red output.
2. Acceptance ↔ test mapping — every criterion has a fang.
3. No mock-only as sole proof on boundary changes.
4. Characterization not silently rewritten during refactors.
5. Flakes — agent-added sleep/retry loops are suspects; fix properly.

Expected artifacts

Regression or acceptance tests · mapping to spec criteria · green make check with evidence · (optional) reviewer notes on weak tests addressed.

Common failures

Symptom	Root cause	Early detection	Fix
CI green, prod bug returns	Tests never failed for that bug	No red-first step	Delete weak test; write failing repro; fix
Suite slow / flaky after agent PR	Real clock, real network, shared state	New sleeps, jest.setTimeout hikes	Isolate; fake timers only where justified
Coverage up, confidence flat	Padding tests	Review asserts	Cull; add risk-based cases
Characterization edited in refactor	Behavior change smuggled	Diff on expected values	Restore; separate behavior PR
Agent tests implementation private APIs	Over-coupling	Imports of non-exported symbols	Test through public API

Recovery strategy

If the suite is green theater: quarantine new tests, restore confidence with one real failing repro, then rebuild coverage deliberately. If agents keep producing mock-only tests, add a never-do to CLAUDE.md with an example of the rejected pattern and the preferred one. Broader mess: Recovery.

Acceptance criteria

• [ ] Bug fixes include a test that failed before the fix (evidence retained)
• [ ] Spec acceptance criteria map to tests
• [ ] Boundary changes have non-mock-only coverage
• [ ] make check green with pasted evidence
• [ ] No flake-prone sleeps added without justification
• [ ] Refactors did not weaken characterization assertions

Related

• Reviewing AI Tests — deep reject/accept patterns.
• Verification Strategy — layers of proof.
• Verification Checklists — copy-paste lists.
• Quality Gates — making failure loud in CI.
• Shallow Implementation — code that passes weak tests.
• Reviewing AI PRs — pass 3 is evidence/tests.