What does it mean for an automation to fail safely?
It means the failure direction is chosen deliberately rather than inherited from whatever the code did. Customer-facing actions should fail closed — when in doubt, send nothing. Internal alerts should fail open — when in doubt, tell a human. And nothing should ever fail silently: every dropped action needs to land somewhere a person will actually see it, with enough context to finish the job by hand.
Choose the failure direction per action type
Fail closed means the action does not happen. Fail open means it happens anyway, possibly wrongly. Neither is universally correct, and treating them as one policy is where systems go wrong.
- Customer messages: fail closed. A message not sent is a small loss. A wrong message sent to a customer is a real one, and it cannot be recalled.
- Internal alerts: fail open. A duplicate alert is noise. A missed one is an incident. Bias toward telling someone.
- Data writes: fail closed, then queue. Do not write a partial or guessed record. Hold it and retry.
- Scheduling commitments: fail closed with a human path. Never confirm something you could not verify.
The partial-completion problem
Multi-step workflows fail halfway. The customer record was created, the job was not. The message was sent, the CRM note was not written. Now your data says something that is not true, and the next automation reads it.
Two approaches help. Make each step independently idempotent so the whole sequence can simply be re-run. Or write a single record marking the workflow incomplete, so anything reading it knows the state is provisional. What does not work is assuming every step succeeded because step one did. Handling this properly is a large part of what integration engineering actually consists of.
Silent failure is the real enemy
The worst automation failure is not a loud error. It is a workflow that quietly stopped firing in March and nobody noticed until June, because nothing breaks when a thing that should happen does not happen.
Guard against it with a heartbeat: monitor expected volume, not just errors. If a workflow that normally runs a few dozen times a day runs zero times, that is an alert even though nothing threw an exception. Absence of activity is a signal, and almost nobody instruments it.
Every dropped action needs a home
A dead-letter queue is not sophisticated infrastructure; it is a list of things that did not work, with enough context for a person to finish them manually. Give it an owner and a review cadence, or it becomes a landfill.
The test of a well-built automation is not that it never fails. It is that when it fails, a human can tell within a day, understand what did not happen, and complete it by hand. That standard applies whether the workflow lives in the platform or in custom code.
Topics: fail safe · error handling · guardrails · reliability
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.