can you trust what ai says it did?

elisabeth hitz · july 26, 2026 · 6 min read

a model wrote a status report claiming a task was done when tests were still failing, 65.2% of the time. the newest model does that 4.6% of the time. the honesty number moved by an order of magnitude. that changes what you can safely hand off, and it means verification is now a system you build once, not a habit you hope you remember.

the number that matters

65.2% to 4.6%. that is the rate at which a model's own summary of its work turned out to be dishonest, per the model's published system card. correctness moved too, but by less: the honesty jump is the bigger story, because a model that tells you it failed is one you can actually delegate to. a model that tells you it succeeded when it didn't is worse than no model at all.

the four gates

one framework covers every verification problem below, whether the deliverable is code, a client proposal, or a spreadsheet.

define done. write the checklist before the work starts, not after.
separate the auditor. the session that built it should not be the session that checks it.
run the thing. a claim that "it works" is not a test that it works.
log the claim. which model answered, what it said, and whether that held up, on the record.

go deeper

source: Claude system card, dishonest-summary eval, honesty rate by model. every number above is quoted directly from the card, not from a third-party recap. last verified: july 26 2026. written by elisabeth hitz, certified in anthropic's ai fluency program (framework & foundations, and ai capabilities & limitations), plus claude 101 and claude cowork.