Case Study: Prototype to Production

Background

Queuebird — async PDF report generation for mid-market analytics teams. Founder + two contractors built a prototype in ten days to close a pilot: Express API, BullMQ workers, Postgres, S3 for artifacts. It worked in demos. The pilot customer planned a rollout to ~200 seats in six weeks. The prototype had: no structured logging, errors as res.status(500).send(e.message), retries that duplicated side effects, secrets in .env committed once (rotated, but the pattern remained), and a worker that console.log'd stack traces containing customer query SQL.

Mandate: harden to production before the 200-user date. Agents encouraged for speed.

Initial vague prompt

Harden this prototype for production. Add error handling, logging,
retries, validation, rate limiting, and security best practices across
the API and workers. Keep behavior the same.

Claude Code opened a PR titled "production hardening" — 89 files, ~3.1k lines. Description claimed: "Added comprehensive error handling, pino logging, zod validation, rate limits, and safer BullMQ retries."

Why it failed

The PR was too large to review. They merged to a staging branch and probed:

Claim	Reality
Error handling	Central middleware on HTTP; workers still threw into BullMQ with default retry that re-sent emails
Logging	pino on API; workers still console.log; SQL still in logs on the failure path in reportWorker.ts
Validation	Zod on 4 of 11 routes — the ones the agent touched while "exploring"
Rate limiting	Global IP limiter; missing on the expensive POST /reports that the pilot would hammer
Retries	attempts: 5 added; no idempotency key on S3 put or email send

"Added error handling" in a PR title is a cross-cutting claim. Agents assert them globally; they implement them locally on the files in context. Reviewing per-PR narrative fails; you must verify per surface.

The improved spec

They reverted the mega-PR and wrote a hardening matrix in specs/prod-harden.md:

## Surfaces × Controls

| Surface | Errors | Logs | Validation | Authz | Idempotency | Rate limit |
|---|---|---|---|---|---|---|
| HTTP routes (11) | middleware + typed AppError | pino redaction | zod per route | session | — | per-route |
| reportWorker | no throw after side effect | pino child | job schema | — | jobId = reportId | concurrency cap |
| emailSender | swallow+metric | no PII body | — | — | provider key | — |
| s3Artifact | retry only on 5xx | key names only | — | — | if-none-match | — |

### Process
- One control column per PR (or one surface row) — never "all hardening"
- Done when the matrix cell is checked by a human with evidence (file:line)

Each agent session filled cells, not vibes. Example prompt:

Read specs/prod-harden.md. Implement ONLY the Validation column for HTTP
routes. List every route file. For each, add zod and a test. Do not change
logging, retries, or workers. End with a table of route → schema file.

Agent workflow

Failure injection mattered: they killed Redis mid-job and proved emailSender did not double-send after the idempotency cell was filled. Before that cell, it double-sent — despite the earlier mega-PR's "safer retries."

They also kept a running "evidence log" in the PR description:

| Cell | Evidence |
|---|---|
| HTTP validation | routes listed in PR; each has `*.schema.ts`; test names … |
| reportWorker idempotency | reportWorker.ts:88 uses jobId; chaos test `worker-redis-kill.test.ts` |

Without the evidence log, reviewers fell back to trusting the PR title again — the exact failure mode they were escaping.

Human intervention points

1. Matrix authorship — humans define surfaces; agents don't get to invent the inventory from a partial tree walk. First agent inventory missed emailSender entirely (it lived under lib/notify/).
2. Evidence review — every green cell needs path:line or a test name, not a PR adjective.
3. Secret/PII pass — human grepped logs for SQL and email bodies after logging work. Found authorization headers still logged on a debug branch the agent left in.
4. Load test + chaos — not delegated; agents can write the script, humans watch the outcome. Target: 50 concurrent POST /reports without duplicate emails.
5. Accept residual risk — they explicitly deferred request signing for webhooks to post-pilot; wrote it down so the next agent wouldn't "finish" it mid-hotfix.

Mistakes made

• Merging the 89-file hardening PR to a long-lived branch. Even unmerged, it poisoned reviews and wasted 1.5 days of bisecting which parts to keep. Cost: calendar slip.
• Believing worker error handling came free with API middleware. Classic prototype lie; cost: duplicate pilot emails in staging (lucky: not prod).
• Rate limiter on the wrong dimension. IP limits don't protect expensive per-account report generation. Fixed after the pilot's CI job got a 429 from a shared NAT — two hours of confusion.
• Redaction config that redacted password but not token or sql. Agent used a common pino redaction snippet. Human expanded the list after reading one real error log.
• Closing matrix cells based on "looks done" in review. One cell (S3 idempotency) was marked done because the code comment said if-none-match; the header wasn't set. Chaos test would have caught it — they had skipped writing that test to "save time." Cost: 4 hours later under load.

Final outcome

Hardening took 13 engineer-days against a 10-day fantasy. Matrix ended with 3 deferred cells documented. At 200 seats: one Sev-3 (S3 throttle, mitigated by backoff cell that had been marked done with evidence). No duplicate emails in prod. The mega-PR's only lasting gift was a couple of zod schemas they cherry-picked (~200 LOC). Final hardened surface: ~8k LOC app + workers, with the matrix checked into docs/prod-harden-matrix.md and updated when new surfaces appear.

Lessons

1. Cross-cutting claims ("added error handling") must be verified per surface via a matrix, not accepted from a PR description.
2. Split hardening into column-or-row PRs; mega-hardening PRs are unreviewable and falsely complete.
3. Workers are a separate surface from HTTP — prototype debt concentrates there.
4. Idempotency is part of retries; retries without it are a duplicate-side-effect machine.
5. Grep production-like logs for PII after any logging change; default redaction snippets are incomplete.
6. A cell is not done without evidence a hostile reviewer can check in under a minute.

Related

• Playbook: Production Hardening — structured hardening path
• Verification Strategy — evidence over claims
• Quality Gates — what must be green before traffic
• Shallow Implementation — locally applied "global" fixes
• Playbook: Production Debugging — when a skipped cell bites later