Is statistical anomaly detection worth it, or are fixed thresholds fine?
Fixed thresholds are fine when you know the level that demands action: a queue past a certain wait, a location with no jobs dispatched by mid-morning. Statistical detection earns its place when normal varies by weekday, season and location and you are monitoring many series at once. The tradeoff is explainability. A fixed threshold is arguable in a meeting; a model score usually is not.
Decide by asking whether normal is stable
If a metric has a level that is unacceptable regardless of context, use a fixed threshold and stop. Safety-style conditions are like this: nothing dispatched, phones unanswered, spend running past a cap.
If the acceptable level depends on the day, the season, the location and the size of the segment, a fixed number will be wrong most of the time. That is where a baseline plus band beats a constant, and where the extra machinery pays for itself.
The multiple comparisons problem nobody mentions
Monitoring gets deployed across every location and every metric at once. A dozen locations times ten metrics is a hundred and twenty series checked daily. If each rule fires falsely one time in a hundred, you get roughly one false alert per day from a system that is behaving exactly as designed.
So sensitivity must be set with the number of watched series in mind, not per rule in isolation. This is the arithmetic reason monitoring rollouts feel fine in a pilot and collapse at scale, and it is why the answer is usually fewer, tighter rules rather than broader coverage.
A middle path that works in practice
- Robust baseline. Median and median absolute deviation rather than mean and standard deviation, so one outlier does not widen the band and hide the next one.
- Persistence. Two consecutive periods outside the band, which removes a large share of false positives cheaply.
- Absolute floor. A material minimum change in real units, so tiny series stop generating alerts.
- Volume gate. Suppress detection entirely below a minimum count, because a rate on a handful of records has no stable normal.
Whatever you use must be able to explain itself
Every alert should state the observed value, the expected range, the basis for that range and a link to the records behind it. Without those four things people cannot verify the alert, so they either act on faith or ignore it, and ignoring wins.
That requirement rules out a fair amount of black-box detection for operational use. It is also why we build detection into the operational layer with the underlying records attached rather than as a standalone scoring service, and why alerts land in the same reporting environment that can answer the follow-up question.
Topics: anomaly detection · thresholds · false positives · monitoring
Have a version of this question about your own business?
The useful answer usually depends on which systems you run and how they're connected. That's a conversation, not a blog post.