Trust calibration: why agents should show their doubt

The problem isn't that agents are wrong

It's that they're wrong in the same voice they're right in. Ask an agent a question it knows cold and one it's guessing at, and you'll usually get the same fluent, declarative tone back. That flat affect is the real trust problem in agent interfaces — not the error rate, but the fact that the interface gives you no way to tell a solid answer from a shaky one before you act on it.

Trust calibration is the fix: getting an agent's expressed confidence to track its actual reliability, and then surfacing that gap to the user instead of hiding it behind smooth prose. It sounds like a model-training problem. It's also very much an interface problem — confidence has to be rendered, not just computed.

Where the miscalibration comes from

Base language models, measured on raw next-token probabilities, are reasonably well calibrated — their stated confidence tracks their accuracy fairly closely. Alignment training tends to break this. A 2025 analysis, "Restoring Calibration for Aligned Large Language Models" (accepted for a calibration-aware fine-tuning approach called CFT), argues the mechanism directly: preference optimization rewards confident-sounding answers regardless of correctness, so post-RLHF models get more fluent and more overconfident at the same time. The paper's framing is useful because it locates the cause precisely — not "LLMs are bad at knowing what they know" in the abstract, but a specific side effect of how these models are tuned to be helpful. See the paper on OpenReview.

That matters for interface design because it means calibration isn't a solved problem you can assume away. A model can verbalize "I'm not entirely sure about this" in one breath and take an irreversible tool call in the next as if it were certain — the confidence language and the confidence-driving-behavior are two different circuits, and they don't automatically agree.

What "calibrated" looks like when a vendor ships it

Anthropic's system card for Claude Sonnet 5 treats calibration as one of several distinct honesty behaviors it evaluates for, separate from things like sycophancy or hallucination rate: the model is trained to express more doubt when its basis for a claim is thin and more confidence when it's solid, rather than defaulting to a uniform confident register. The same card notes a related and slightly uncomfortable finding — verbalized evaluation awareness (the model noticing it's plausibly being tested) is measurably higher in Sonnet 5 than in prior models, which is worth knowing if you're using eval performance as a proxy for real-world calibration. See the Claude Sonnet 5 system card.

The practical takeaway for anyone building on top of a model, rather than training one: don't assume the hedge language you see in a chat transcript survives into agentic behavior. Test calibration at the point of action, not just at the point of speech.

The agent-specific version: knowing when to quit

Chat calibration is about wording. Agent calibration is about action — specifically, whether the agent will stop and ask instead of proceeding on a guess. A NeurIPS 2025 workshop paper, "Check Yourself Before You Wreck Yourself," tested this directly: it evaluated twelve models on the ToolEmu benchmark and measured what happens when agents are explicitly prompted to quit (abstain from acting) once their confidence drops below some threshold, rather than pushing forward. Agents given quitting instructions improved a 0–3 safety score by an average of 0.39 across all models, and by 0.64 among proprietary models specifically, with only a small drop in helpfulness. Full details are in the arXiv paper.

That's a small, legible result, but it says something an interface designer should sit with: the single highest-leverage calibration behavior for an agent isn't a better-worded caveat. It's a bailout — a clean, well-defined way to stop and hand back to the human instead of guessing forward under uncertainty. An agent that can't quit gracefully will eventually act confidently on a bad guess, no matter how well-hedged its running commentary sounds.

Designing the surface, not just the model

Microsoft's Guidelines for Human-AI Interaction — a synthesis of interaction research now packaged as the HAX Toolkit — put this as its second guideline: "make clear how well the system can do what it can do," meaning help the user build an accurate mental model of the error rate up front, not just per-answer. See the guidelines overview. Applied to an agent interface, that argues for confidence signals at two different grains:

Action-level signals are the ones that actually change behavior, because they arrive at the moment a person decides whether to trust the next step. A generic confidence disclaimer in onboarding gets read once and forgotten. A specific hedge attached to a specific claim — "I'm inferring this from the error message, not from the actual config file" — gives the user something to act on immediately: verify, or proceed.

What to build

A few patterns fall out of the research above, in order of leverage:

• Make abstention a first-class action, not a failure mode. If quitting looks like an error in your logs or your UI, the agent (and your team) will be biased against using it. Treat "I don't have enough confidence to proceed" as a normal, successful outcome, not a bug report.
• Tie hedges to the specific claim, not the whole response. "I'm not sure" attached to nothing is noise; "I'm not sure the port is 8080 — I inferred it from a comment, not the config" is a fact a user can check in five seconds.
• Route low confidence into your approval gate, not around it. If your agent already has a sign-off step for consequential actions, wire uncertainty into the trigger for that gate — a shaky action should need sign-off even if it would normally run on autopilot.
• Test calibration where the action happens, not just where the words do. A model can hedge accurately in conversation and still act confidently on a bad tool call. Evaluate the two separately.

None of this requires a new protocol or a confidence API nobody ships yet. It requires treating "I don't know" as a designed, tested output — as deliberately built as the happy path — rather than whatever falls out of a model's default register. The pattern tracker has more on where uncertainty currently plugs into the approval gate, and where it still doesn't.