Production Debugging with an Agent

An agent is a pattern-matcher with a very large library of famous bugs. In an incident, that's both its value and its danger: it will generate plausible root causes faster than any human, and it will latch onto the most famous cause rather than your cause with equal confidence. The playbook below exists to extract the speed without inheriting the overconfidence: evidence in, ranked hypotheses out, discriminating test for each hypothesis before anyone writes a fix.

Objective

Turn production telemetry into a verified root cause and the smallest safe fix, with a regression test and a postmortem note — without the agent freelancing a refactor mid-incident.

When to use

• A production error, latency regression, or data anomaly with real telemetry available (logs, traces, error payloads, metrics).
• You can reproduce, or plausibly reproduce, in staging or locally.

Not for: active sev-1 mitigation. Mitigate first (rollback, feature flag off, scale up) with humans in charge. Bring the agent in for diagnosis once the bleeding is controlled — an agent editing code during an outage is a second incident waiting to happen.

Inputs required

Input	Why it matters
Raw log excerpts, stack traces, error payloads	The agent reasons dramatically better from verbatim evidence than from your summary of it
Incident timeline (first occurrence, deploys, config changes, traffic shifts)	Correlation with changes is the highest-signal discriminator
What changed recently (git log --oneline since last-known-good, infra changes)	Most incidents are caused by the last change; hand the agent the suspect list
Blast radius (which endpoints/users/jobs affected, which are NOT)	Negative evidence kills hypotheses fastest
Repo access	The agent must read the actual code paths, not guess at them

Don't sanitize the evidence into a narrative. Your summary already contains your hypothesis, and you'll bias the agent toward it. Paste raw material; redact secrets, not detail.

Step-by-step process

1. Assemble the evidence pack

Five minutes of collection saves an hour of hallucinated diagnosis. Drop everything into a single file (incident/2026-07-10-checkout-500s.md or a scratch doc) so the whole investigation shares one source of truth and survives a /clear.

2. Run the triage prompt

Copy verbatim, fill placeholders:

You are diagnosing a production incident. Do NOT propose or write any fix yet.

EVIDENCE:
- Symptom: <one-line symptom, e.g. "POST /api/checkout returns 500 for ~3% of requests since 14:20 UTC">
- Timeline: <first occurrence, deploys/config changes near it, current status>
- Logs/traces (verbatim):
<paste raw excerpts>
- Error payload / stack trace (verbatim):
<paste>
- What is NOT affected: <endpoints, regions, user segments that are healthy>
- Recent changes: <git log excerpt, infra/config changes>

TASK:
1. Read the code paths implicated by the stack trace before hypothesizing.
2. Produce a ranked list of hypotheses (most likely first). For EACH:
   - Mechanism: exactly how it produces THIS symptom, referencing specific
     files/lines in this repo.
   - Evidence for / evidence against, from the material above.
   - A discriminating test: the cheapest observation or experiment that would
     confirm or eliminate it (a log query, a curl, a unit test, a staging repro).
3. Explicitly flag which hypotheses fail to explain any piece of evidence,
   especially the "not affected" list.
4. Stop. Do not modify any code.

The two load-bearing constraints: read the code first (otherwise you get textbook diagnoses untethered from your repo) and discriminating test per hypothesis (a hypothesis without a cheap falsifier is an opinion, and it forces the agent to state what evidence would prove it wrong).

3. Human review point: audit the hypothesis list

Before running any test, check:

• Does hypothesis #1 explain all the evidence, including what's not broken? An agent-favorite failure: a hypothesis that explains the stack trace but not why only 3% of requests fail.
• Does each mechanism cite real files and lines? Spot-check one. If the agent references src/payments/retry.ts and that file doesn't exist, the whole list is contaminated — restate the evidence and rerun.
• Is any hypothesis suspiciously famous? "Node.js event loop blocked", "N+1 query", "race condition in connection pool" — these are real bug classes, but they're also the agent's greatest hits. Demand repo-specific evidence before accepting one.

4. Run discriminating tests, cheapest first

Execute the tests yourself or have the agent run the read-only ones (log queries, targeted greps, reading traces). Eliminate hypotheses out loud in the conversation — the agent should update its ranking as evidence lands, not defend its original pick. If it defends a dead hypothesis, that's context poisoning; restate surviving evidence in a fresh message or fresh session.

5. Reproduce before fixing

A root cause you can't reproduce locally or in staging is still a hypothesis. Have the agent write the reproduction:

Hypothesis confirmed: <one-line root cause>.
Write a minimal reproduction as a failing test in <test path>. It must fail for
the same reason production fails (assert on the actual error/behavior, not a
proxy). Do not fix anything yet. Show me the failing test output.

The failing test is your regression test — you get it for free, and it's derived from the bug, not from the fix.

6. The fix-constraint prompt

Now, and only now, the fix. Copy verbatim:

The failing test in <test file> reproduces the root cause. Now fix it under
these constraints:
- The fix must not touch anything outside <file or directory, e.g. src/checkout/tax.ts>.
- The regression test stays exactly as written and must pass.
- Smallest change that makes the test pass. No refactoring, no renaming, no
  drive-by cleanup, no dependency changes, no "while I'm here" improvements —
  even if the surrounding code is bad. Note cleanup candidates in a comment in
  your summary instead.
- If the correct fix requires touching files outside the boundary, STOP and
  tell me which files and why. Do not proceed.

The escape hatch in the last line matters: without it, an agent that genuinely needs to touch a second file will either silently violate the boundary or contort the fix into something wrong-but-compliant.

7. Verify and write the postmortem note

Run the regression test, the surrounding test suite, and — per your deploy process — verify in staging against the original telemetry signature. Then:

Write a postmortem note in <incident file>: symptom, root cause (with file/line),
why it wasn't caught earlier, the fix (commit ref), the regression test, and one
prevention item (a lint rule, a monitor, a spec/CLAUDE.md addition). Max 25 lines. Facts only,
no process theater.

If the prevention item is a repo convention the agent violated, put it where the agent will see it next time — see CLAUDE.md and AGENTS.md and Docs as Memory.

The loop at a glance

Expected artifacts

• Incident evidence file (the shared source of truth)
• Ranked hypothesis list with discriminating tests (in the conversation or the incident file)
• Failing-then-passing regression test, committed
• Minimal fix commit, boundary respected
• Postmortem note with one concrete prevention item

Common failures

Symptom	Root cause	Early detection	Fix
Confident diagnosis that matches a famous bug but cites no code in your repo	Agent pattern-matched to training data, not your evidence	Mechanism section names generic causes, no file/line refs; hypothesis ignores your "not affected" list	Reject the list; rerun triage prompt with "reference specific files and lines" enforced; spot-check citations
Fix makes the symptom vanish but the mechanism is unexplained	Agent fixed the symptom, not the cause (added retry, widened timeout, wrapped in try/catch)	Fix diff contains retries/catches/timeout bumps with no change to the causal code path	Revert. Demand the mechanism first: "explain the exact sequence of events, then fix the cause"
Diff touches 14 files during a 1-file bug	Agent refactored mid-incident ("this code was confusing, so I restructured it")	Diff size disproportionate to the bug; renames and moves in the diff	Revert everything outside the boundary; rerun with the fix-constraint prompt
Agent keeps defending an eliminated hypothesis	Context poisoned by its own earlier confident claims	It re-cites evidence you've already refuted	Fresh session; paste only surviving evidence and eliminated-hypothesis list — see Stuck Loops and Thrash
Regression test passes even when you re-introduce the bug	Test asserts a proxy (mock called, no exception) rather than the actual failing behavior	Manually revert the fix; test must fail	Rewrite the test against the production failure signature — see Reviewing AI Tests

Recovery strategy

If the fix ships and the symptom returns, or the diagnosis collapses: treat it as a new incident with better evidence, not a continuation. Revert the fix (it's noise now), start a fresh session, and add "we previously believed X; it was wrong because Y" to the evidence pack — the failed hypothesis is itself high-value negative evidence. If the agent produced a sprawling diff you can't trust, go straight to Recovering from Bad Agent Output.

Acceptance criteria

• [ ] Root cause has a stated mechanism referencing real files/lines, and it explains all evidence including what wasn't affected
• [ ] Bug was reproduced (test failed) before the fix was written
• [ ] Fix diff touches only the declared boundary
• [ ] Regression test fails when the fix is manually reverted
• [ ] Postmortem note exists with one concrete prevention item landed (monitor, lint rule, or CLAUDE.md/spec update)

Related

• Recovering from Bad Agent Output — when the fix attempt itself goes wrong
• Hallucination and False Confidence — the failure class behind famous-bug pattern-matching
• Stuck Loops and Thrash — recognizing and breaking a poisoned debugging session
• Reviewing AI Tests — making sure the regression test actually guards the regression
• Maintenance Prompts — more debugging and diagnosis prompt variants
• Docs as Memory — turning postmortem lessons into durable agent context