Agentic Engineering

Appearance

Case Study: Internal Platform ​

Background ​

Platform team of five at a ~120-engineer company. Mandate: reduce time-to-first-deploy for new services from ~2 weeks of copy-paste to under one day via a golden path — Go service template, Kubernetes Helm chart, CI workflow, and a Backstage software catalog + scaffolder. Internal users are other engineers: higher tolerance for rough edges than external customers, near-zero tolerance for "the template generates a cluster-noncompliant YAML."

They used Claude Code heavily. It went well after they learned that internal ≠ "skip conventions."

Initial vague prompt ​

text
Build a Backstage portal with a scaffolder that creates a new Go
microservice with Kubernetes manifests and CI. Make it the golden path
for our company.

The agent generated a Backstage app, a cookiecutter-ish template, a generic Helm chart, and a README saying "customize to your org." The scaffolder produced services that failed admission controllers on first deploy: missing required labels, no network policy, wrong ingress class, Dockerfile as root.

Why it failed ​

Internal platforms fail differently from product features. The demo (Backstage UI spins, scaffolder runs) succeeds while the contract with the platform fails:

Generated artifactPlatform reality
Labels app: onlyAdmission requires app.kubernetes.io/name, owner, team, cost-center
ingressClassName: nginxCluster uses internal-nginx only
Container runs as rootPSS restricted; must run as non-root
CI used kubectl applyOrg standard is GitOps via Argo CD app-of-apps
Go module path github.com/example/...Must be github.com/<org>/... from scaffolder params

Agents are fast at Backstage boilerplate. They do not know your admission policies unless you encode them as the template's acceptance tests.

The improved spec ​

They paused UI work and wrote docs/golden-path-contract.md plus a reference service (services/golden-ref) that already deployed cleanly.

markdown
## Golden path contract v1

### Must pass before scaffolder ships
- kubectl apply --dry-run=server of generated manifests in ci-kind cluster
- Conftest / OPA policies: labels, non-root, no latest tags, network policy present
- Argo CD Application manifest generated, not raw kubectl
- Backstage catalog-info.yaml with required annotations
- Go: go.mod module path from {{cookiecutter.org}}/{{cookiecutter.name}}

### Non-goals v1
- Multi-region, service mesh toggles, autoscaling beyond HPA defaults

### Source of truth
- Copy structure from services/golden-ref — do not invent new chart layout

Prompt pattern shifted to: "Generate the scaffolder so that its output matches services/golden-ref structure and passes make verify-template."

Agent workflow ​

Agents edited the template; humans owned policy tests. When generated manifests failed OPA, the answer was never "relax admission for golden-path services."

make verify-template rendered the scaffolder into a temp dir, ran Conftest, spun kind, and server-dry-ran the manifests. That target became the definition of done for every template PR. An agent that "fixed" a failure by deleting the network-policy template file was caught because Conftest required the object to exist — the test failed closed. Without that, the deletion would have looked like a clean diff.

Backstage work resumed only after three consecutive green verify-template runs on different parameter combinations (team A/B, criticality low/high). The portal is a thin form over a template that already proved itself.

Human intervention points ​

  1. golden-ref built and deployed by platform engineers — the exemplar cluster state, including the Argo Application and sealed-secrets pattern the org actually uses.
  2. Policy-as-code authored/reviewed by platform — agents may propose Conftest rules; humans approve. One proposed rule banned all ClusterIP services; wrong — rejected.
  3. Scaffolder parameter design — team, cost-center, criticality; product decisions for the platform. Agents must not invent new parameters mid-flight.
  4. Pilot feedback — two teams scaffolded for real; platform watched first Argo sync live. Secrets and DNS were the actual blockers, not Go code.
  5. Versioning the contract — breaking template changes required a changelog engineers actually read, plus a #golden-path Slack post with the migrate command.

Mistakes made ​

  • UI-first Backstage. Cost: ~1 week of portal work before the first generated service could deploy. Reverse the order: contract → template → portal.
  • Letting the agent "improve" the Helm chart beyond golden-ref. It introduced a shared subchart abstraction that no one else used; broke on values schema. Regenerated from golden-ref; deleted the abstraction. ~6 hours lost.
  • Skipping kind+Conftest in the scaffolder CI. Relied on "we reviewed the template." A label rename in admission broke 14 generated services' next deploys. Cost: two days of platform firefighting.
  • Assuming internal users read README customization notes. They don't. If it's required, it's in the template or the verify step fails.
  • Golden-path exception culture. After the label incident, one senior asked to exempt "platform-generated" apps from admission. Platform refused. Exceptions would have made the agent-generated path a second-class citizen that bit everyone later.

Final outcome ​

v1 golden path shipped in five weeks (two wasted on the UI-first attempt). Median time-to-first-deploy for new Go services dropped to ~4 hours (mostly Argo sync + secrets). Adoption: 38 services in two quarters. Agent contribution: high on Backstage React/TS and template glue (~70% of those LOC); low on policy and chart shape — by design. Template repo itself: ~4k LOC, of which the verify harness (~800 LOC) was the highest-leverage human-written piece.

Internal tools do tolerate agent speed on undifferentiated glue. They do not tolerate agent invention of cluster conventions. The speed win showed up as platform engineers spending cycles on contract evolution instead of ticket-driven YAML copy-paste for each new service.

Lessons ​

  1. Conventions and a deployed reference service come before scaffolder UI — generate against an exemplar that already passes admission.
  2. Encode platform rules as tests (Conftest/kind) on rendered template output; don't rely on README or review alone.
  3. When the template drifts from golden-ref, regenerate — don't layer clever Helm abstractions via agent.
  4. Internal users need stricter automation gates, not looser ones; they will skip docs.
  5. Agents accelerate Backstage/glue; humans own the contract with the cluster.
  6. Never grant admission exemptions for "agent-generated" manifests — that path must be the most constrained, not the least.

A field manual for AI-native software engineering.