Verification Strategy
Agent output is optimized to look finished: typed, named, commented, green. Human reviewers treat that surface as a proxy for correctness because for human authors the two correlate. Agents break the correlation. A polished invoice proration module can round currency wrong, skip leap-year billing, and still ship with 94% coverage of tests that assert the mocks were called. Verification is not "read the diff carefully." It is a pipeline that proves behavior against intent, independent of how convincing the code looks.
Why AI code looks correct but isn't
Three mechanisms produce the illusion:
| Mechanism | What you see | What's wrong |
|---|---|---|
| Fluency ≠ correctness | Idiomatic TypeScript, clean names | Wrong business rule encoded fluently |
| Self-confirming tests | Green suite written by the same session | Tests encode the same wrong assumptions as the code |
| Local coherence | Every touched file is internally consistent | Hidden coupling two modules away is broken |
The review skill that matters is behavior against intent, not craftsmanship. Craftsmanship is free now and signals nothing. See Why Agents Fail and Hallucination and False Confidence.
The verification pipeline
| Pass | Question | Cheap signal of failure |
|---|---|---|
| Scope | Did it only touch files the task requires? | Unrelated files in the diff; see Scope Failures |
| Architecture | Did it reuse existing patterns and respect boundaries? | New parallel abstraction next to an existing one; see Architecture Erosion |
| Behavior | Do tests prove the acceptance criteria, not the implementation? | Mock-only assertions; missing named edge cases |
| Attack | Does a cold second agent find holes the builder missed? | Zero findings from a reviewer that shared the builder's context |
| Gates | Do CI, types, lint, security, and smoke pass? | Agent "fixed" a gate by weakening it |
Passes 1–2 are human-minutes on the file list and a quick architecture skim. Pass 3 is where most agent bugs die — if you skip it, you are reviewing prose. Pass 4 is optional for trivial diffs and mandatory for auth, money, and data integrity. Pass 5 is mechanical; treat a green gate as necessary, never sufficient. Full gate design: Quality Gates.
Test layers for AI-assisted work
Agents will happily generate whichever layer you ask for — and will default to the cheapest layer that turns green. You choose the mix; the agent fills it in.
| Layer | What it proves for agent work | Agent failure it catches |
|---|---|---|
| Unit | Pure logic against enumerated cases | Wrong formula, off-by-one, bad branching |
| Integration | Real collaborators (DB, queue, HTTP) wired correctly | Mock-shaped code that never worked end-to-end |
| Contract | Boundary shapes between services/clients | Hallucinated response fields; silent schema drift |
| Regression | Previously fixed bugs stay fixed | "Cleanup" that deletes the edge case you paid for |
| Smoke | Deployed system answers critical paths | Config/env mistakes invisible in unit green |
Rule of thumb for agent PRs: at least one integration or contract test per new boundary, and one regression test per bug the agent "fixed." Unit-only PRs on money, auth, or persistence are incomplete by default.
Tests before code
Writing tests after the agent has implemented the feature is how you get characterization of whatever it built — including the bugs. Invert the order when the acceptance criteria are clear:
From docs/specs/<feature>.md, write failing acceptance tests only.
Do not implement production code. Do not read any existing
implementation of this feature if one exists.
Cover every acceptance criterion and every named edge case in
the spec. Use real collaborators where the spec's behavior
crosses a boundary; mock only external systems listed in
CLAUDE.md as mockable.
Stop when the suite fails for the right reasons. List any
spec ambiguities you hit — do not invent answers.Why the order matters: the tester role that hasn't seen the implementation cannot inherit the builder's shortcuts. Same idea as the tester role and the tester-first pattern. After the suite exists, a separate builder session implements until green — and is forbidden from weakening assertions to get there.
Second-agent attack
A builder that reviews its own work re-derives the same wrong assumption and calls it verified. The attack pass needs a fresh context: diff + spec + CLAUDE.md, never the builder transcript.
You did not write this code. Review git diff main...<branch>
against docs/specs/<feature>.md and CLAUDE.md.
Your job is to find the three worst problems, ranked by
production blast radius. Hunt specifically for:
- silently narrowed requirements vs the spec
- weakened authz / validation / error handling
- tests that assert mocks or call counts instead of outcomes
- missing edge cases the spec names
- unrelated file changes
Output: severity · file:line · what's wrong · which spec line
it violates. Zero findings is suspicious — state what you
checked and found clean. Do not modify files.In Claude Code, spawn this as a read-only subagent or a separate session in a worktree. Independence is the product; see Agent Roles and Coordination.
Weak vs strong verification ask
Weak:
Make sure everything works and write some tests.Strong:
Implement docs/specs/invoice-proration.md on branch
feat/proration. Acceptance is green only when:
1. test/integration/proration.test.ts covers the six cases
named in the spec (including leap-year and TZ edge).
2. No new mocks inside src/services/.
3. npm run typecheck && npm test && npm run lint pass.
4. You paste the failing-then-passing test output — do not
claim green without evidence.
Do not expand scope. List assumptions before coding.The delta: the strong version makes "done" an evidence claim, not a vibe. Agent confidence without pasted proof is noise — treat it as such in Quality Gates.
Related
- Reviewing AI Tests — how to audit the suite the agent wrote for you.
- Quality Gates — CI, types, lint, security, and performance as hard stops.
- Verification Checklists — copy-paste gates for accept / merge / deploy.
- Reviewing AI PRs — the three-pass human review that sits on top of this pipeline.
- Hallucination and False Confidence — mock-only tests and confident-but-wrong claims.
- Responsibility Split — why "is this done?" never delegates.