you might not be running the model you benchmarked.
a safety classifier can reroute your request to a different, less capable model, mid-session. anthropic does notify you when this happens, but in a long autonomous run that notification is easy to miss entirely. that is a harder problem than "the model made a mistake," because you cannot verify work from a model you didn't notice you were talking to.
the honesty number, again
per the model's published system card: dishonest coding summaries dropped from 65.2% (an older model) to 4.6% (the newest model). one detail breaks the clean story: an in-between-tier model scores 3.7% on the same eval, slightly better than the newest one. honesty is not simply a function of how new or how expensive the model is. that matters here because it means a fallback to a "lesser" model is not automatically a fallback to a less honest one, but you still need to know it happened.
what an operator does about a reroute you might miss
- log which model answered, every session. the notification exists, but it scrolls past in a long agentic run. check the model-identity log, not your memory of whether you saw a message.
- treat a fallback as a stop signal, not a retry. if the model that answered is not the one you scoped the task for, stop and re-scope, don't just re-run the same prompt and hope.
- build the check into the habit, not the memory. the same way you'd check a build log before shipping, check the model-identity log before trusting a claim of "done."
this is the sharper version of the verification-layer thesis: it is not enough to verify the work, you have to know which system produced it in the first place, and a notification you didn't read doesn't count as knowing.
source: Claude system card, dishonest-summary eval by model, pages 154-155. Anthropic's July 1 2026 Fable 5 redeployment, per public reporting (blockchain-council.org, digitalapplied.com). last verified: july 26 2026. written by elisabeth hitz, certified in anthropic's ai fluency program (framework & foundations, and ai capabilities & limitations), plus claude 101 and claude cowork. related reading: can you trust what ai says it did? and the honesty number nobody reported.