What is the worst thing an AI automation can do if it gets one thing wrong?
Ask that before you build it. Blast radius is how many records, customers or dollars an error touches before someone notices, and it works out roughly as error rate multiplied by throughput multiplied by time to detect. The levers are batch size, rate limits, a cap on actions per hour, a restricted scope, and a stop condition that trips when the automation's own behavior looks abnormal.
Errors scale with throughput
A person applying a wrong rule makes a handful of mistakes before someone says something. An automation applying the same wrong rule makes every mistake available to it, overnight, at machine speed. The error rate may be far lower than the human's and the total damage far higher.
That is the asymmetry to design around. It is not an argument against automating. It is an argument for putting ceilings on how much can go wrong per unit of time.
Four limits worth setting on day one
- A per-run cap. No single execution may touch more than a set number of records. A backfill that wants to touch everything should be a deliberate, separately approved run.
- A rate limit you impose on yourself. Not the vendor's limit. A lower one that keeps a runaway loop slow enough to catch.
- An anomaly stop. If today's action count is far outside the normal range, halt and notify rather than proceed. Most runaway automations are obvious in their volume long before they are obvious in their output.
- Scope restriction. Limit which records are eligible at all, by location, job type or date range, so an unexpected input cannot expand the target set.
Anything that leaves your systems is a different category
A wrong tag is an internal problem. A wrong text message to a customer is a relationship problem, and it cannot be recalled. Automations with outbound reach deserve tighter limits, lower volume caps and a human gate that is real rather than nominal.
The practical rule we apply in integration work: internal, reversible actions can run wide and unattended; customer-facing actions run narrow and reviewed until there is a substantial track record.
Detection time is half the equation
Halving your error rate is hard. Halving your time to detect is usually easy, and it cuts blast radius by exactly as much. Alert on action volume, not only on errors, because the dangerous failures often succeed technically.
Also make sure the alert goes to someone who is awake and empowered to stop it. An alert that lands in a shared inbox at 2am contributes nothing to detection time, which is the number that actually governs how bad an incident gets in an operational system.
Topics: blast radius · risk · guardrails · safety
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.