Case Study: Add Authentication

Background

Briefly — a scheduling tool for freelance creatives. React SPA, Fastify API, Postgres. Launched with a shared "magic link" prototype: email a token, set a cookie, no refresh rotation, no session revocation, roles as a string column nobody checked. 30 paying customers, ~180 weekly actives, revenue real enough that a breach would be existential for a three-person team.

They needed real auth before an agency deal: email/password + OAuth Google, refresh tokens, logout-everywhere, admin vs member roles enforced on every mutating route.

Initial vague prompt

Add proper authentication to our Fastify app. Support email/password and
Google OAuth, JWT access tokens, refresh tokens, password reset, and
role-based access. Write tests. Don't break existing users.

Claude Code delivered in one long session: authPlugin, passport-ish Google flow, JWT middleware, a users.password_hash migration, and 22 new tests — all green.

Why it failed / would have failed

Security review (external, half-day) before the agency deal found:

Finding	Detail
Refresh tokens stored as plaintext in Postgres	Theft of DB backup = permanent account takeover
role checked only on /admin/* routes	DELETE /v1/projects/:id still trusted client-sent userId
Password reset token in query string logged by CDN	24h window of tokens in access logs
Google OAuth state not bound to session	Basic CSRF on the callback
Migration set password_hash NULL for existing magic-link users	Fine — but login path treated NULL hash as empty-string bcrypt match failure with a user enumeration timing leak the tests never covered
Tests	Register → login → access protected route. No revocation, no role confusion, no token reuse

Happy-path tests are how agents declare victory on auth. They prove the demo works. They do not prove the threat model holds. With 30 paying customers, "don't break existing users" also meant: magic-link users must still get in during migration, sessions must invalidate on password change, and you cannot email everyone "please reset" without a support plan.

The improved spec

They reverted the auth PR to a branch and wrote specs/014-auth-retrofit.md with an explicit threat model. Abridged:

## Auth retrofit — constraints

### Threats in scope
- Session theft via XSS (httpOnly, secure, sameSite)
- Refresh token theft at rest (hash at rest, rotate on use, family revoke)
- IDOR on project/org resources (authz from session, never from body)
- OAuth CSRF (state param, server-side store)
- User enumeration on login/reset (constant responses + timing budget)

### Migration
- Existing magic-link users: keep link login for 30 days OR set password
- On password set: revoke all sessions
- Cookie name change: dual-read for one deploy, then drop legacy

### Tests required (minimum)
- Refresh reuse detection revokes family
- Member cannot hit admin routes AND cannot delete other orgs' projects
- Reset token single-use; CDN-log scenario documented as accepted residual if POST-only form used
- Google callback rejects bad state

### Out of scope
- SAML, passkeys, MFA (track in PLAN.md for post-agency)

They also added a human-owned checklist from Verification Checklists adapted for auth — not generated by the agent.

Agent workflow

Auth was split into three PRs on purpose. A single "add auth" PR is unreviewable; agents make them large because the prompt is large.

Human intervention points

1. Threat model authored by humans — agent may expand it, never invent it as the source of truth.
2. PR1 review focused on crypto/storage — refresh hashing, cookie flags, token TTL constants.
3. Authz audit — human listed every mutating route; agent was tasked to add checks against that list, then human grepped for request.body.userId and request.query.userId.
4. Staging — password reset and OAuth tested on real Google client IDs; agent local stubs were not trusted.
5. Customer comms — magic-link deprecation email written by founders, not the model.

Mistakes made

• Shipping agent tests as the security suite. Cost: would have been the agency deal failing the customer's questionnaire. External review was the save.
• Allowing userId in bodies "for convenience" in two legacy routes. Agent preserved them and "secured" them with a comment. Comments are not authz. Two hours to find; four to fix call sites in the React app.
• Reset-via-GET from an old blog snippet the agent recalled. Caught because the spec required POST. Without the spec, it would have looked normal.
• Underestimating migration support load. Even with a clean technical path, 7 of 30 customers needed hand-holding. Process, not prompts.

Final outcome

Auth shipped in 11 calendar days (3 of coding, rest review/migration). Post-deploy: one incident — Google users with plus-addressing got duplicate accounts because the agent normalized email inconsistently between OAuth and password paths. Fixed in six hours with a linking tool. No session leaks found in the following quarter's review.

Test count: 22 agent tests kept (rewritten), 19 human negative tests added. The negative tests are the ones that failed first when a later agent "simplified" middleware.

Lessons

1. Auth is the worst feature to hand an agent with a capability list and "write tests." Specify threats, migration, and negative cases.
2. Happy-path tests hide security holes by construction; budget human time for revocation, reuse, IDOR, and CSRF cases.
3. Split storage/crypto PRs from route-wiring PRs so review can actually happen.
4. Grep for client-supplied identity (userId, orgId in body/query) after every auth retrofit — agents preserve legacy shortcuts.
5. Email normalization and account linking are product decisions; if unspecified, the agent will pick one and you will discover it in production.

Related

• Playbook: Add Feature — retrofit pattern with migration constraints
• What Not to Delegate — threat models stay human
• Verification Strategy — negative paths as first-class
• Hallucination and False Confidence — green tests, wrong guarantees
• Quality Prompts — adversarial review prompts