The AI outputs a confidence score. Can I set a threshold and auto-approve anything above it?
Not until you have calibrated it on your own data. A confidence number a language model writes about itself is generated text, not a measured probability — it can be consistently overconfident. Bucket a few hundred past outputs by stated confidence, measure how often each bucket was actually right, and see whether the curve tracks. Until then, use confidence to rank what humans review first, not to decide what humans skip.
Three different things get called confidence
They behave differently and only one of them starts out meaningful.
- Self-reported confidence. The model was asked to rate its own certainty and produced a number the way it produces any other token. It correlates with something, but it is not derived from the model's internal uncertainty.
- Classifier probability. A trained classifier emitting a real probability over labels. Closer to meaningful, still often miscalibrated on data that differs from its training distribution.
- System-level agreement. The same input scored several times, or by two different configurations, with agreement used as the signal. Expensive, but the most honest of the three, because it measures stability rather than asserting it.
The calibration test
Take a sample of outputs that already have a verified human answer. Group them by the confidence the system reported — high, medium, low. In each group, compute the share that turned out correct.
A calibrated system shows a clean ordering: the high bucket is right more often than the medium bucket, which beats the low bucket, and the stated numbers roughly match the observed rates. An uncalibrated one shows a high bucket that is wrong about as often as the rest. That is a system telling you nothing, loudly.
Why miscalibration is the dangerous direction
Under-confidence wastes review time. Over-confidence quietly ships errors. The asymmetry matters because the outputs a model is most confidently wrong about tend to be the ones that look most conventional — the routine-seeming call that was actually a cancellation, the invoice that looks normal but belongs to a different job.
Those are exactly the items a confidence threshold would wave through. If you are going to automate on a threshold, the useful question is not "how accurate is it" but "what does it look like when it is wrong at high confidence," and you only learn that by reviewing high-confidence items on purpose.
What confidence is genuinely good for
Ranking. Even a poorly calibrated score is usually better than random at ordering a review queue, which means your reviewers spend their attention where it pays. That is real value and it requires no trust in the absolute numbers.
It is also a good drift detector. If the distribution of confidence scores shifts noticeably after a model or prompt change, something moved even if the outputs still look fine. We wire that kind of monitoring into operational AI deployments rather than treating scores as decoration on a dashboard.
Topics: confidence · calibration · human review · accuracy
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.