How do you decide how confident an AI has to be before it acts on its own?

Confidence is not accuracy

A model's stated confidence is its own estimate of how likely it is to be right, and that estimate is itself a prediction that can be miscalibrated. A system can be confidently wrong in a consistent way, especially on inputs unlike anything it was tuned on.

So the first job is not choosing a cutoff. It is checking whether confidence correlates with correctness at all on your data. Bucket a few hundred labeled examples by reported confidence and look at the actual hit rate in each bucket. If the buckets do not separate, the score is not usable as a gate and you need a different signal.

Two errors, two very different prices

Every threshold trades false actions against missed actions, and the two rarely cost the same.

• False action. The system did something it should not have. Cost depends almost entirely on reversibility and whether it was visible to a customer.
• Missed action. The system declined and the item went to a queue, or nowhere. Cost is the delay plus the human time to handle it.
• The asymmetry sets the direction. When false actions are expensive, set the bar high and accept a big review queue. When missed actions are expensive and errors are trivially reversible, set it low.

Per action, not per system

Within one workflow, tagging a call by topic, flagging it for a manager's attention, and drafting a customer text are three different risk levels. Giving them one shared threshold means either over-gating the cheap action or under-gating the expensive one.

In call intelligence work this shows up constantly: topic classification can run unattended, while anything that reaches the customer sits behind review. The same logic applies to scoring used in coaching, where output informs a manager's conversation rather than triggering anything by itself.

Thresholds drift

A threshold calibrated in March is a statement about March's data and March's model version. Change either and the curve moves. Put a recheck on the calendar tied to model or prompt changes, not to the quarter, and keep the labeled sample you used the first time so the comparison is apples to apples.

Keep a record of what each threshold was and when it changed, stored next to the outputs it governed. Otherwise a shift in automation rate six months from now reads as a change in the business when it was a configuration value somebody nudged on a Friday afternoon and nobody wrote down.