Agentic Engineering

Appearance

Case Study: Node Backend Refactor ​

Background ​

Ledgerly — B2B invoicing. Core API: Express + MySQL, ~45k LOC, five years of growth, three lead engineers over that span, currently owned by a team of four. The billing module (src/billing/, ~11k LOC) had become the place everyone feared: tax rules in three files, proration logic duplicated, a legacyInvoiceRebuild path still hit by 12% of traffic.

Goal: extract a cleaner billing domain, keep MySQL, stay on Express for now, stop the "touch billing and take a week of PTO" culture. They brought Claude Code in for the mechanical moves after two failed human-only attempts that stalled on fear.

Initial vague prompt ​

text
Refactor src/billing to be cleaner. Extract a BillingService, remove
duplication, modernize the tax and proration code, and add better tests.
Don't break production.

The agent produced a 2.4k-line PR in one session: new BillingService, collapsed three tax helpers into one, "fixed" proration to always round half-up, deleted legacyInvoiceRebuild as dead code (it wasn't — the static analysis missed dynamic require from a cron entrypoint).

Why it failed ​

Staging soak found it in four hours:

SymptomCause
3% of renewal invoices off by $0.01–$0.03Agent "corrected" banker's rounding that finance had encoded as a feature for a specific enterprise contract
Nightly rebuild job 500slegacyInvoiceRebuild deleted; cron still called it via string path
New unit tests all greenTests asserted the new rounding behavior, not production behavior

The vague prompt asked for improvement. The agent improved. Finance did not want improvement — they wanted identical invoices under a new structure. "Don't break production" is not a spec; it is a wish.

The improved spec ​

They stopped the PR, restored main, and spent three days on characterization before any redesign. Abridged from specs/billing-extract-phase1.md:

markdown
## Phase 1 — Extract without behavior change

### Invariants (must hold bit-for-bit on fixtures)
- Given fixtures/invoices/*.json (847 cases from prod anonymized),
  `computeInvoice(input)` output must deep-equal recorded output
- Rounding: preserve current half-even on tax lines; DO NOT "fix"
- Keep legacyInvoiceRebuild; mark deprecated; add deprecation log

### Allowed changes
- Move functions into src/billing/domain/* with re-exports from old paths
- Add characterization suite under src/billing/__characterization__/
- No new features, no tax formula edits, no deletion of cron entrypoints

### Forbidden
- "Cleaning up" magic numbers without an ADR
- Changing error messages (clients parse substrings — yes, really)

They generated fixtures by recording production inputs/outputs for one week into a side table, then anonymizing. The agent was only allowed to move code once the characterization suite was green and owned by humans.

Agent workflow ​

Phase 1 prompt pattern:

text
Read specs/billing-extract-phase1.md and CLAUDE.md.
Move computeTax and computeProration into src/billing/domain/ without
changing their bodies. Update imports. Run npm run test:characterization.
If any test fails, revert your changes and report which fixture failed —
do not fix the fixture or the code.

Only after phase 1 merged did they open phase 2 specs for intentional tax cleanup — each change with an ADR and a fixture update reviewed by finance.

Human intervention points ​

  1. Fixture curation — engineers picked the 847 cases; the agent did not invent golden files.
  2. Diff review for silent logic edits — any changed constant or branch in a "move" PR was a reject, even if tests passed.
  3. Finance sign-off before phase 2 rounding changes.
  4. Cron inventory — human grepped for dynamic requires and string-based job names the agent had missed.

Mistakes made ​

  • First PR trusted "better tests." Agent-authored tests encoded the rewrite. Cost: ~6 hours staging debug + trust damage with finance.
  • Assuming static analysis finds all callers. Dynamic require(jobName) in jobs/runner.js hid legacyInvoiceRebuild. Cost: would have been a Sev-2 at 02:00.
  • Letting the agent "document" magic numbers by renaming them. TAX_FUDGE_0_004 became MUNICIPAL_SURCHARGE_RATE with a wrong comment. The rename PR looked helpful; the comment was a hallucination. Caught in review because a senior had written the original fudge in 2021.
  • Characterization gaps on multi-currency. First fixture set was USD-only. EUR invoices drifted in phase 2. Added 120 EUR fixtures after a near-miss.

Final outcome ​

Phase 1 took 2.5 weeks calendar time (mostly fixture work). Net LOC in billing: still ~11k, but with a domain boundary and 847 characterization tests. Phase 2 (real tax cleanup) took one week with zero invoice mismatches in a two-week shadow deploy comparing old vs new compute on live traffic.

The team rule that stuck: no agent refactor PR without a characterization suite that fails when you deliberately mutate one line of business logic.

Lessons ​

  1. Pin current behavior — including bugs and ugly rounding — before an agent is allowed to improve anything. Improvement is a product decision, not a code-style decision.
  2. Agent-written tests of agent-written refactors are not safety; they are confirmation. Characterization fixtures must come from production or human-authored expectations.
  3. "Dead code" deletions require a runtime caller inventory, not just IDE find-references — especially in Node job runners.
  4. Split move-only phases from behavior-change phases; review them with different checklists.
  5. Magic numbers need ADRs, not confident new names from the model.

A field manual for AI-native software engineering.