How should an integration retry when the API returns errors?
Retry only what is safe to retry, with exponential backoff plus randomized jitter, and know when to stop. Retry 429 and 5xx responses; do not retry 400, 401, 403 or 422, because those will repeat forever without changing. Honor the Retry-After header when the vendor sends one. Without jitter, every worker retries at the same instant and you rebuild the exact traffic spike that caused the throttle.
Sort errors before you retry them
- Always retry. 429 rate limited, 502, 503, 504, connection resets and timeouts. These are transient by definition.
- Never retry. 400 malformed request, 401 bad credentials, 403 insufficient permission, 422 validation failure. Retrying these burns your rate limit to receive the same answer.
- It depends. 500 can be transient or a permanent server-side rejection of your specific payload — retry a small number of times, then quarantine the record. 404 on a write may mean the record was deleted, which is information, not an error to retry.
- 409 conflict. Usually means re-read and re-apply rather than blindly retry, because the state you assumed has changed.
Jitter is not optional
Exponential backoff alone synchronizes your workers. If twenty jobs fail at the same moment and all wait exactly two seconds, they all retry at the same moment. You have recreated the spike, get throttled again, and now they all wait four seconds and do it again. The technical name is a thundering herd and it is why some outages last much longer than the underlying fault.
Add randomness to every wait — a random value within the backoff window rather than the window itself. This is a one-line change with an outsized effect.
Retries plus non-idempotent writes equal duplicates
The most damaging retry bug is not a failed retry, it is a successful one. A write request that times out may well have succeeded on the server. Retry it and you have created two jobs, two customers or two invoices from one intent.
Attach an idempotency key to every write the vendor supports one for. Where the vendor does not, check for the record's existence before creating it, or design writes as upserts keyed on your own external identifier. This is a design requirement, not an optimization.
Know when to stop and where the work goes
Cap the total attempts, then move the item to a dead letter queue with the full request, the response and the timestamps. A queue you can inspect and replay turns a data loss incident into an afternoon of cleanup.
Add a circuit breaker for sustained failure. When an endpoint has failed continuously for a while, stop calling it entirely for a cooling period, keep queueing work, and resume when a probe succeeds. Hammering a system that is already down helps nobody, and a well-behaved integration is a much better tenant on shared API infrastructure. These patterns are the boring foundation under any dependable systems integration.
Topics: retries · backoff · rate limits · reliability
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.