Case Study: AI-Built Product
Background
Inkline — collaborative writing tool for small legal teams (clause libraries, redlines, export to DOCX). Solo technical founder + one part-time domain expert (ex-paralegal). Stack: Next.js App Router, Postgres, Claude Code locally and in CI (headless review on PRs). Over four months to public beta, roughly 90% of commits by line weight were agent-typed. The founder did not "barely touch the keyboard" — they touched different keys: specs, review, verification, product calls.
This study is about that job shift, not about whether 90% is impressive.
Initial vague prompt
Build a full legal writing product with Next.js and Postgres. Clause
library, document editor, redlining, comments, export to DOCX, auth,
billing. Ship something I can charge for. Use Claude Code for everything.Month one produced a large surface: editor, auth, Stripe, clause CRUD, a DOCX export that worked on happy paths. Demoed well to two friendly lawyers. First real matter: export dropped comment threads, redline attributions were wrong, and a "restore version" path silently duplicated clauses across matters (IDOR-ish coupling through a shared clauseId the agent had "simplified").
Why it failed / would have failed as a process
The failure wasn't that agents write bugs — humans do too. The failure was operating like a typist supervisor ("make it build, make the demo work") while the system required an architect-reviewer-verifier. At 90% agent authorship:
| Human job people imagine | Actual job at 90% |
|---|---|
| Occasional course correction | Continuous spec authorship |
| Skim PRs for style | Adversarial review for authz, data model, export fidelity |
| Trust CI | Design CI to catch agent blind spots |
| Add features daily | Refuse features until verification debt is paid |
Green CI on agent-written tests of agent-written features is a hall of mirrors. The founder learned this when export "tests" asserted zip structure existed, not that comments survived round-trip.
The improved spec
From month two, every capability needed a spec with verification that a human would perform once before paying customers saw it:
## Spec 022 — DOCX export fidelity
### In scope
- Export body paragraphs, clause inserts, comments, redline marks
- Round-trip check: export → pandoc/docx diff fixture against golden
### Verification (human)
- [ ] Open export in Word desktop (not only LibreOffice)
- [ ] Comment anchors point at correct text for fixture matter M-017
- [ ] Tracked changes show author display names from users.table
- [ ] File size under 5MB for 50-page fixture
### Agent must not
- Mark done based only on unit tests that mock the DOCX builder
- Change clause ownership rules while touching exportCLAUDE.md gained: "You do not decide product done-ness. Spec verification checkboxes are human-gated." CI ran agents for implementation and for a reviewer pass that posted findings; merge still required the founder.
Agent workflow
Headless CI review caught style and some bugs; it never replaced the Word-desktop checkbox. The founder stopped asking agents "are we ready to ship?" — a question they will answer yes to.
A concrete week in month three (representative):
| Activity | Hours |
|---|---|
| Writing/cutting specs | 12 |
| Running human verification checklists | 10 |
| Reviewing agent PRs (incl. CI review comments) | 14 |
| Steering agents / clarifying sessions | 6 |
| Typing code by hand | 3 |
| Customer/domain calls | 5 |
That distribution is the product. If your calendar still looks like "mostly coding," you are not actually running an AI-built product — you are pair-programming with autocomplete and calling it leverage.
Human intervention points
- Spec gate — no implementation session without a spec file path in the prompt.
- Verification checklists — performed personally on legal-sensitive paths (export, sharing, authz). Calendar blocks, not "when I get to it."
- Data model changes — founder owned ER decisions; agents proposed migrations, did not invent matter/clause ownership.
- Saying no — ~40% of agent-suggested "quick wins" deferred; velocity without refusal becomes product sludge.
- Incident review — when duplication bug hit, founder wrote the postmortem and the regression fixture; agent implemented the fix against that fixture.
Mistakes made
- Month one without human verification rituals. Cost: two pilot matters of cleanup credibility; nearly lost the domain advisor.
- Letting agent review replace human review because CI posted long comments. Volume ≠ coverage. Authz IDOR lived through three "LGTM"-ish agent reviews. The reviewer agent praised "consistent use of matterId from the session" on a route that still read
clause.matterIdfrom the client body. - Skipping PROJECT_CONTEXT updates. Agents re-introduced a rejected "global clause pool" design twice. Writing the rejection down once stopped the third attempt. See Docs as Memory.
- Measuring progress in PRs merged. Switched to "specs verified this week." PR count made the dangerous month look like the best month.
- DOCX golden fixtures generated by the same code under test. First round-trip test used the exporter to create the golden — tautology. Goldens now come from Word-saved files checked in as binaries; agent may update them only with human approval.
Final outcome
Public beta at month four: 28 paying teams, export fidelity acceptable, sharing authz reviewed by an external contractor for one week (worth it). Codebase ~42k LOC, founder estimates 90% agent-typed, 100% human-accountable. The founder's calendar: ~30% specs/product, ~40% review/verification, ~20% agent steering, ~10% actual typing. Harder cognitive load than writing a smaller app by hand — different muscle, not a vacation.
The honest pitch to other solo founders: agents remove the typing bottleneck and move the bottleneck to judgment. If you were bottlenecked on judgment already, you will feel slower before you feel faster.
Lessons
- At high agent authorship, the scarce skill is spec + adversarial verification, not prompting cleverness.
- Agent-written tests of agent-written features need human-designed fixtures and at least one non-mocked check on critical paths.
- CI agent review is an input to human review, not a substitute — especially for authz and export fidelity.
- Record rejected designs in PROJECT_CONTEXT; agents will recreate them otherwise.
- Optimize the week for verified specs shipped, not for volume of agent commits.
- Golden files must not be produced by the code under test — especially for document export.
Related
- Responsibility Split — what stays human as agents type more
- Specs over Prompts — operating model for AI-built products
- Reviewing AI PRs — adversarial review under volume
- Verification Checklists — human-gated done-ness
- Learning Path: Solo Founder — calendar and rituals