What happens when the AI is wrong inside a live workflow?
That depends entirely on what you let it touch, which is a design decision rather than a property of the model. Assume a wrong output on every path and ask what it costs. A bad note is noise. A bad tag skews a report. A bad customer message is a real problem. A bad status change breaks dispatch. Contain the blast radius first, improve accuracy second.
Errors are certain; consequences are optional
Any system that interprets language will be wrong sometimes, on accents, on sarcasm, on a call where two things happen at once, on a record somebody typed badly in 2019. Planning for zero errors is planning for a system that does not exist.
So the engineering question is not how to prevent every mistake. It is how much a mistake costs, how quickly it is noticed, and how fast it can be undone. Those three properties are entirely under your control, unlike model accuracy.
A blast-radius checklist for any new automation
- Who sees it? Internal only, or a customer? Customer-visible output raises the bar sharply.
- Can it be undone? If yes, in one click or through a support ticket? The difference matters more than it sounds.
- Does anything downstream key off it? A field that drives dispatch, payroll or a commission calculation propagates the error into places you will not think to check.
- How many records at once? A per-event write fails small. A nightly bulk job fails at scale, so bulk paths need dry runs and row-count guards.
- How long until someone notices? If the answer is 'at month end,' add a check that runs sooner.
Make errors visible and cheap to fix
Label AI-generated content so a human reading it knows what it is. Keep the evidence attached, so a disputed classification can be checked against the actual transcript in seconds rather than argued from memory. Keep a ledger of writes so a bad run can be identified and reversed as a set.
One more habit worth adopting: when a person corrects the system, capture the correction as data. Corrections are the highest-quality signal you will ever get about where the logic is weak, and most systems throw them away.
The category that deserves the most caution
Anything touching a person's employment. Evaluation output should be treated as evidence for a manager, not as a verdict. Rubric analysis applies your standards consistently across every call, which is genuinely fairer than sampling, but the conclusion about a person belongs to the human who knows the context. Framing it that way is not just ethics; it is accuracy, because the system cannot see the shift someone covered or the customer who was abusive.
Customer-facing communication sits close behind. Draft with AI, release with a person, and widen autonomy only where you have a real track record. The rest of the operational AI surface can move much faster precisely because those two areas are handled carefully.
Topics: risk · error handling · blast radius · design
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.