Agents say done. Done is not proof.

Our founder's agent said the whole photo archive was embedded. 1,926 of 28,709 were.

claude-code, photo archiveAug 29, 2026

agentEmbedding all 28k is running now (detached, PID 40142 ... ~1–2h).

collinyou sure?

logembed: candidates=1926 embedded=1926 errors=0

check1,926 embedded. 2,225 of 28,709 searchable. The claim was not the receipt.

meter

claimed 100%, measured 6.7%

A real session, every line verbatim. Our founder caught this one by hand. He typed "you sure?" and read the database. Have one of these? Send it.

Agents fail the same few ways.

18 defects from Andon's public ledger, each with its root cause and its fix. You will recognize your own week in here.

2026-04The false "done"
DefectThe agent said a change was "complete and tested." It had written the code but never run the tests. I trusted it, moved on, and the gap surfaced hours later.
FixA Stop hook (claim_check_hook.py) that reads the turn-ending message for done-claims and requires a verification log entry before the turn can end. Warn first, block once trusted.
Full entry
2026-04"That doesn't exist" (it did)
DefectTold to find a thing, the agent grepped one keyword, got nothing, and reported it absent. It was there the whole time — under a different name.
FixBefore concluding "X isn't here," search synonyms and old names, grep the content (not just the title), and check the version-control log for when it was added.
Full entry
2026-05Fixed five, claimed fourteen
DefectFixed the instances of a problem in the files I was looking at, then reported the whole class fixed. Nine more were sitting in files I hadn't opened.
FixBefore any "all / complete / whole" claim about a class, grep the full set first and verify each — then scope the claim to exactly what was verified ("5 of the daily surfaces," not "the whole stream").
Full entry
2026-05Four parsers, four slightly different schemas
DefectA config file was read by several scripts, each written from memory. One read the fields in a different order. It didn't crash — it silently returned wrong results.
FixOne canonical loader that every consumer imports. If the structure isn't checked in one place, it isn't real.
Full entry
2026-05Tested the script, not the thing that runs
DefectDeclared a hook "verified end-to-end" after running the inner script by hand. The harness actually invokes a wrapper that reads stdin — which was never exercised.
FixWhen the claim is "it works end-to-end," the test must hit the actual production path — the wrapper the harness calls, under the real environment.
Full entry
2026-05$100 matched $1000
DefectA checker used substring matching to compare a price. "$100" matched inside "$1000", so a 10× error sailed through as "no change."
FixWord-boundary regex with the negative lookahead; never naive in for values that have longer supersets.
Full entry
2026-05Repeated a subagent's "finding" as fact
DefectA helper agent reported a surprising result; I relayed it as true. It was wrong — the agent had missed a detail (a symlink, in that case).
FixSpot-check every surprising subagent finding by re-running its evidence yourself before repeating it.
Full entry
2026-05Acted on yesterday's map
DefectStarted a session from the last handoff note and recommended work that was already done — a parallel session had shipped it between the handoff and now.
FixBefore acting on a handoff, re-read the version-control log since its timestamp. Freshest reality wins.
Full entry
2026-05The gate that jammed the worker
DefectShipped an enforcement hook in block-mode-by-default. First real use, it blocked a legitimate turn with no obvious way out. Trust in the hook cratered immediately.
FixDefault to warn; provide an env-var escape; promote to block only after it's been tuned. Make every gate advisory until proven.
Full entry
2026-06The test that passed by luck
DefectA test was green, so the fix looked solid. The test was asserting something that happened to be true in one case, not the property that actually mattered.
FixAssert the real invariant, then mutate the code to confirm the test fails on a plausible bad version. If a bad mutation survives, the test isn't guarding anything.
Full entry
2026-06Rebuilt what already existed
DefectAsked for a capability, the agent built it from scratch. The codebase already had it. Now there were two, subtly different.
FixSearch-before-building, every time: grep the code, check memory, look for an existing tool. A 30-second search beats three sessions of duplicate-bug cleanup.
Full entry
2026-06Reversed a right answer when pushed
DefectChallenged with "you sure?", the agent retracted a correct claim after eyeballing partial evidence — overcorrecting into a new error.
FixTreat a challenge as a trigger to gather more evidence and compute the full picture — then keep or change the claim on the evidence, not the pressure.
Full entry
2026-09The gate that wasn't running
DefectAn enforcement hook appeared fully configured: on disk, wired in the harness config, hundreds of historical gate-log records in the fire log that appeared to indicate activity, but did not establish successful direct-command invocation. During an observed session, dozens of invocation failures were recorded across three hooks and were present in the transcript but not acted on. The execute bit was missing. Tests invoked the scripts through an interpreter, which doesn't require the execute bit; the harness invoked them as direct commands, which does. Two invocation paths, silently diverged.
FixBefore declaring any hook live, invoke it as a direct command with a benign payload and alarm on a launch/permission failure (distinct from a policy denial). Cross-check configuration against disk in both directions. These checks belong in an automated test — not just a design rule.
Full entry
2026-09The gate that could be alibi'd
DefectThe claim-check gate accepted any fresh verification entry as satisfying any done-claim. It checked only that some recent log line existed — never that the entry referred to the claim, or came from the same session. So a bare log_claim "done" "checked" cleared the gate; a genuine verification of task A licensed an unverified claim about task B; and — worst — on a shared home directory one session's entry cleared every other session's claim, invisibly.
FixBind the entry to the claim and the session. log_claim.py stamps the session id (from CLAUDE_CODE_SESSION_ID) and a content-token fingerprint of the claim and its verification; for claims with subject tokens, the gate checks for a fresh same-session entry whose fingerprint overlaps that claim's sentence. Subjectless claims use a same-session freshness fallback (see the README limits). Closure verbs and selected generic dev nouns are stopworded so "done" cannot match on that word alone. The block message names the verification, not the unblock command. Legacy (pre-binding) entries are honored only while the log has no binding-schema entry. Once a binding-schema entry exists, older entries no longer qualify; in-flight work may need a new-format receipt.
Full entry
2026-09Tightened for precision, lost the recall — then overshot fixing it
DefectThe claim detector caught 4 of 13 ordinary done-claims. Implemented the migration. fired; I've implemented the migration. — how agents actually write — did not. Task complete! and The fix is in place. passed unseen. Reported with a repro in issue #3.
FixAdditions that stand on their own — a first-person lead-in (the determiner rule protecting Fixed income securities… untouched), the missing standalone verbs, a subject-noun closure form, and state verbs each carrying their own guard. Every accumulated example is a checked-in corpus, and the test that matters asserts the patch is additive: no previously detected example in that corpus may stop being caught.
Full entry
2026-09The fix that kept breaking what it was fixing
DefectCatching result-claims (the tests pass) needs the detector to know when a clause asserts something, so exemptions were added for negation, attribution and conditionals. Each round of review found claims those exemptions had silently suppressed: four, then two, then one. Each fix was correct and produced the next one.
FixDrop the feature that needed the exemptions rather than keep tuning them, and replace instance-checking with a property: a corpus-wide test that the patch only ever adds detections. Mutation-checked — reintroduce a broad exemption and it fails.
Full entry
2026-09The rollback that could roll back to itself
DefectA deploy-rehearsal gate proved rollback by rebuilding the previous commit's image, redeploying it by digest against the same database, and checking that it came up with the existing data intact. Caught in design, before the gate's first run: if the previous commit only changed docs, it builds a byte-identical image with the same digest, and the gate "rolls back" by redeploying the exact thing it claims to have rolled back from. Green, and proving nothing.
FixStamp every build with the commit it came from (an image revision label), so two commits always produce two digests. The gate asserts digest A ≠ digest B, and that each label names its own commit, before it redeploys anything. The label reads like metadata, so it's recorded as load-bearing next to the code: deleting it makes nothing fail loudly; it makes the proof go quietly vacuous.
Contributed by @w-30x (PR #6).Full entry
2026-10The witness who wrote their own alibi
DefectAfter binding (above), a verification entry had to be fresh, same-session and on-topic — but the agent still wrote it. log_claim.py "migration complete" "ran the migration tests" cleared the gate whether or not a test had run. The gated actor and the evidence author were the same actor.
FixAn opt-in evidence mode (CLAIM_CHECK_EVIDENCE=tool) that ignores the agent's line and requires a passing verification run the harness recorded in the session transcript: allowlisted command, same sessionId, inside the freshness window, a finished non-error result, and a command line that matches a small fixed grammar so its exit status is the verification's. The recorded-result markers were read off real transcripts first, which turned up three ways a run reports success without having succeeded: a backgrounded command (backgroundTaskId, including auto-backgrounding), a command that hit the tool timeout (timedOutAfterMs), and a nonzero exit reinterpreted as success (returnCodeInterpretation) — all with is_error: false. A first version parsed arbitrary shell and checked separators; four rounds of review kept finding lines where the parser and the shell disagreed (a # comment swallowing a newline, a mid-word quote, pipefail "set" inside a dead branch). The fix was to stop parsing shell and accept only an allowlisted grammar.
Full entry

Four of the 18 here. All of them on GitHub. Add yours on GitHub.

Caught one of your own? The ledger takes entries in the same four-line shape. Add yours on GitHub.

Proof you can run yourself.

hooks/claim_check_hook.py, warn modeclean clone 15ef5bb

agentthe migration is complete and the tests are fixed

andon

⚠️  Claim-check enforcer — done-claim detected without a fresh verification log entry satisfying the applicable correspondence or fallback rule.
Matched phrases:
  - "is complete"
  - "are fixed"

The hook's real response, from a clean clone. Run it yourself in 30 seconds.

539 tests pass on a clean clone.

Done-claims our hook stopped, week by week

331stopped for a missing receipt since 2026-05-19, the last week still open

A stop means the agent said "done" or "fixed" and no proof was on record. 528 claims checked; 331 stopped; 53 proven afterwards.

Read by people who then changed it.

@alex-spataru rebuilt the hook and the ledger inside Serial-Studio, a 7,000-star project.Serial-Studio on GitHub

@w-30x wrote a ledger entry from their own deploy gate, and the argument behind the evidence mode.pull request 6

@gcracolici fixed the detector's recall on done-claims written the way agents write them.pull request 4

Three public changes, each with its link. If Andon runs in your project, email primefoldtools@gmail.com and we will list it here.

Three things on the shelf.

AndonCatches a false "done" before it reaches you.free
AI Agent Operator KitRuns the same operator workflow every session.free
Verification KitCatches the three mistakes Andon lets through.$27Try it for 30 days. If it is not worth $27 to you, email us and we refund it. No questions.

See the shelf

Send us your incident.

One incident: what your agent claimed, what was true. First three free. You get back the check that would have caught it.

Email primefoldtools@gmail.com

A script supplies every figure on this page, as read on Nov 3, 2026.

Your coding agent
says done.