Agentic Engineering

Appearance

Hallucination and False Confidence ​

Agents do not know when they are wrong. They produce fluent code and fluent assurances with the same tone. This cluster is about invented reality (APIs, guarantees, test proof) and quiet safety regressions that look like helpful simplification. Your countermeasures are evidence requirements, independent review, and gates the agent cannot talk past.

1. Hallucinates APIs ​

Symptoms ​

  • Imports a function/method/option that does not exist in this version.
  • Uses a SaaS API field from an outdated tutorial.
  • "According to the docs" claims that collapse when you open the real docs.
  • Types pass only because of any / loose configs — or fail until the agent invents a wrapper.

Root cause ​

Models blend APIs across versions and libraries. Confidence is calibrated to fluency, not to retrieval truth. Research-without-verification is the accelerant — see Architecture and Research Prompts.

Early detection ​

  • Typecheck / compile is a hard gate (Quality Gates).
  • Spot-check every new third-party call against installed version docs.
  • Demand version pins and links/quotes for load-bearing external APIs.

Prevention ​

  • Prefer APIs already used in-repo; search the codebase first.
  • Prompt: "verify against node_modules types / official docs for version X; if unsure, open a question."
  • Lockfiles and exact versions in examples.

Recovery prompt ​

text
Typecheck/tests fail or APIs look invented. For each external
call in the diff: cite the package version and the canonical
docs/types that define it. Replace hallucinated APIs with real
ones used in this repo where possible. Remove any/as-casts added
to hide the problem. Paste typecheck output.

Review checklist ​

  • [ ] New external APIs verified against current version
  • [ ] No new any / @ts-ignore to silence missing symbols
  • [ ] Examples and comments match real signatures

2. Silently weakens security ​

Symptoms ​

  • Authz check removed "temporarily" to make a test pass.
  • CSRF, rate limit, or input validation bypassed in a "fix."
  • CORS widened to *; cookie flags dropped; IDOR introduced by using client-supplied IDs without ownership checks.
  • Secret scanning excluded; dangerous markdown/HTML allowed through.

Root cause ​

Security properties are constraints that impede the agent's immediate goal (green tests, working demo). Without an explicit forbid, the model treats them as optional friction. Same session that weakens security must never be allowed to "sign off" on security.

Early detection ​

  • Diff grep for authz, authorize, can(, middleware removals, dangerouslySetInnerHTML, CORS, NODE_TLS.
  • Security role / second agent on auth and payment diffs (Agent Roles).
  • Tests that assert denial cases — agents hate writing these unless forced.

Prevention ​

  • CLAUDE.md: never remove or broaden authz/validation to pass tests; stop and ask.
  • Mandatory human review on security-sensitive paths (Responsibility Split).
  • SAST + secret scan gates; forbid path exclusions in feature PRs.

Recovery prompt ​

text
Audit git diff main...<branch> for security weakenings:
authz removals/bypasses, validation loosened, CORS/cookie/TLS
changes, secret handling, injection sinks, IDOR.

For each finding: severity, exploit sketch, restore the stronger
control. Add a test that fails if the weakening returns.
Do not "fix" by broadening permissions further.

Review checklist ​

  • [ ] Authz still default-deny for new routes
  • [ ] No new public access without explicit spec
  • [ ] Denial/negative tests present for permissions
  • [ ] Scanner exclusions and secret-related changes reviewed

3. Tests that only test mocks ​

Symptoms ​

  • High coverage; assertions are toHaveBeenCalledWith.
  • Entire domain mocked; nothing hits DB/queue/HTTP.
  • Refactors break production while tests stay green.
  • Agent cites coverage % as proof of correctness.

Root cause ​

Mocks are the path of least resistance to green. The suite becomes a theater of the implementation's call graph. Full treatment: Reviewing AI Tests.

Early detection ​

  • Mutation thought experiment: break the formula; do tests fail?
  • Mock inventory vs approved boundaries in CLAUDE.md.
  • Absence of integration/contract tests on new boundaries.

Prevention ​

  • Explicit mock policy in prompts and repo docs.
  • Tester-first from spec without reading implementation (Patterns).
  • Reject coverage-as-argument; demand behavior assertions.

Recovery prompt ​

text
Rewrite tests for <module> to assert observable outcomes
(return values, DB state, HTTP body, events). Remove mock
call-count assertions as primary proof. Mock only <approved
boundaries>. Show one deliberate mutation that the new suite
catches.

Review checklist ​

  • [ ] Primary assertions are outcomes
  • [ ] Mocks limited to approved boundaries
  • [ ] Coverage % not used as acceptance evidence

4. False confidence claims ​

Symptoms ​

  • "All done, tests pass" without pasted output.
  • "No security issues" from the same session that wrote the code.
  • "Backwards compatible" without contract tests.
  • "I verified against the docs" with no citation.
  • Human merges on the assurance instead of the artifact.

Root cause ​

The model is trained to be helpful and complete. "Done" is a speech act it performs readily. Declaring done is a human responsibility; agent claims are inputs, not verdicts.

Early detection ​

  • Require command transcripts / CI links in the PR.
  • Zero findings from a self-review is suspicious — demand the checklist used.
  • Independent reviewer session (Verification Strategy).

Prevention ​

  • Definition of done = evidence list in CLAUDE.md.
  • PR template: paste outputs; link preview; checklist.
  • Culture: treat unearned certainty as a smell, not as leadership.

Recovery prompt ​

text
Your previous "done" claim is unverified. Re-run:
<typecheck> <tests> <lint> <relevant smoke>.
Paste full exit codes and failing output if any.
Then run an adversarial self-audit against the spec:
list residual risks you are NOT confident about.
Do not claim complete. Human will decide done.

Review checklist ​

  • [ ] Evidence pasted or linked for every gate claimed green
  • [ ] Independent review for high-blast-radius changes
  • [ ] Residual risks listed, not papered over
  • [ ] No merge on prose assurance alone

A field manual for AI-native software engineering.