Case Study: CI/CD Pipeline

Background

Northwind Analytics — six engineers, Node services on ECS, infra drifting between a hand-maintained console setup and a half-finished Terraform repo. Goal: one pipeline path — GitHub Actions → Docker build → ECR → Terraform apply to staging/prod — with required checks, OIDC to AWS, and no long-lived cloud keys in GitHub secrets.

They asked Claude Code to "stand up CI/CD" while the team finished a product sprint. Estimated: two days. Actual drama: two weeks.

Initial vague prompt

Set up a full CI/CD pipeline with GitHub Actions, Docker, and Terraform.
Build, test, push to ECR, deploy to staging on merge to main, deploy to
prod on tag. Use best practices. Make it secure.

The agent created .github/workflows/deploy.yml (420 lines), a multi-stage Dockerfile, and Terraform for ECS/ALB/IAM. First green run on a fork: spectacular. First run on the real org: less so.

Why it failed

Symptom	Root cause
Prod apply from a PR branch	Workflow used pull_request_target + checkout of PR code with cloud credentials
Terraform state in S3 bucket with public ACL block off	Copied from an outdated snippet; agent didn't know the org's baseline module
terraform apply -auto-approve on staging and prod	Prompt said "deploy"; agent obliged
Actions using npm install not npm ci	Lockfile ignored; supply-chain drift
Docker base node:latest	Non-reproducible builds; broke two days later
Secrets: still had AWS_ACCESS_KEY_ID in repo secrets	OIDC code existed in a comment; keys were "easier to get working"

Pipeline code is production code that deploys production code. A vague "make it secure" does not encode your org's trust boundaries. Agents optimize for a green workflow run — the demo of CI — the same way they optimize for a polished UI demo.

The improved spec

specs/cicd-oidc.md abridged:

## Pipeline requirements

### Trust
- OIDC to AWS only — delete static keys after cutover
- Never use pull_request_target with checkout of untrusted code + cloud creds
- fork PRs: run tests only, no AWS

### Apply gates
- staging: apply on merge to main, after tests + terraform plan uploaded as artifact
- prod: apply on annotated tag v*, requires manual environment approval in GitHub
- No -auto-approve without environment protection rules

### Reproducibility
- Pin Actions by SHA
- Pin container digests
- npm ci; Dockerfile COPY package-lock.json

### Review
- Same PR rules as app code: two reviewers for workflows/** and infra/**
- Plan output reviewed before first prod apply

They also added a CODEOWNERS entry for .github/ and infra/ — social gate matching the technical gate.

Agent workflow

Sessions were split: test workflow ≠ deploy workflow ≠ IAM Terraform. The failure mode of one mega deploy.yml is that review cannot hold the whole threat model in their head.

First staging apply still surprised them: Terraform wanted to replace the ALB because the agent had named resources differently from the console-imported state. They spent five hours on a terraform import plan the agent drafted — then a human rewrote it after the agent tried to import the wrong target group ARN. Import/move surgery is where agents are confident and wrong; treat those plans as hostile until proven.

Human intervention points

1. Threat model for CI — who can run what against which account; written before YAML. Include fork PR behavior explicitly.
2. First terraform plan for prod — read by a human who knows the ECS service, not skimmed. Diff size was 400+ lines; they scheduled a 90-minute review, not a Slack LGTM.
3. GitHub Environment protection — configured in UI/API by humans; agent can document, not be trusted to have "set it up" without verification. Founder clicked the settings page and screenshot-attached to the ticket.
4. Deletion of static AWS keys — explicit checklist item after OIDC proven. Keys lingered a week because "we'll delete them after the sprint."
5. Pin audit — monthly script ensuring Actions aren't floating @v4 tags on critical paths; first run found two unpinned third-party actions the agent added for Slack notify.

Mistakes made

• Celebrating the first green workflow. Cost: the pull_request_target pattern was in that green run. ~8 hours to notice, longer to feel safe again.
• Letting the agent "match existing Terraform style" when existing style was the half-finished drift. It propagated bad state backend config. Humans should have pointed at the org's private module registry.
• Auto-approve on staging. Staging shared a data store with a partner integration; a bad apply paused real work for three hours. Staging is not a scratchpad when it is shared.
• Under-reviewing IAM policies the agent generated. ecs:* on * slipped through until a security engineer grepped. Cost: emergency policy tighten + panic Slack thread.
• Slack notify action from a random marketplace publisher. Agent picked a popular action; supply-chain review was not in the original spec. Added "Actions must be from GitHub-verified orgs or our mirror" to the pipeline spec after the fact.

Final outcome

Pipeline stabilized in week three. Deploy time to staging: 14 minutes, reproducible. Prod: manual approval, ~20 minutes, two incidents in six months (both human-approved plans that under-estimated ALB draining — not CI bypasses). Static keys deleted. CODEOWNERS on infra caught three risky workflow edits from well-meaning app engineers using agents — including one that reintroduced -auto-approve on prod "to unblock a hotfix." The CODEOWNERS reject was the win; the agent had done what the engineer asked.

Rough LOC: ~600 lines Actions YAML, ~2.2k Terraform, ~80 Dockerfile. Agent-typed share high (~80%); human-reviewed share 100% on .github/ and infra/.

Lessons

1. Pipeline-as-code needs the same review gates as app code — CODEOWNERS, small PRs, explicit trust specs.
2. A green Actions run proves syntax and permissions you granted; it does not prove a safe trust boundary.
3. Ban pull_request_target + untrusted checkout + cloud creds in CLAUDE.md; agents reproduce this pattern from the internet.
4. Pin Actions and base images by digest/SHA; "latest" and floating tags are silent breakages.
5. Split test, identity, staging apply, and prod apply into separately reviewable changes.
6. Terraform import/state moves are human-led; agent-drafted import scripts need hostile review.

Related

• Quality Gates — what CI must enforce
• Reviewing AI PRs — reading generated YAML/IAM
• What Not to Delegate — trust boundaries
• Constraints and Validation — encoding org rules
• Stuck Loops and Thrash — when agents chase green checks