Detector scorecard
Three detectors judge every bucket. One is allowed to raise alerts; the others run in shadow and have their verdicts recorded anyway. Because they see identical data at the same moment, choosing between them is a table rather than an argument — and changing the active one is a configuration change whose effect was measured before it was made.
What these numbers are not
This is not accuracy. Nobody labels these hours by hand. A verdict is scored against a rule applied in hindsight: an hour counts as a genuine surge when its count stands well clear of the rate around it, measured from both sides. A detector could score well by agreeing with a rule that is itself wrong.
What the numbers are good for is comparing detectors against each other on the same data, which is the question being asked. They are shown unfiltered, including where a shadow beats the detector in charge.
A bucket is only scored once enough hours have passed to judge it, so recent activity sits in “awaiting hindsight” rather than being counted early.
ewma
activeExponentially weighted baseline; fires when the hour exceeds it by a multiple.
Precision71%Recall86%Fired17Awaiting hindsight012 of 17 firings held up · 12 of 14 surges caught · 115 buckets judged
true positive 12false positive 5missed 2true negative 96poisson
shadowUpper-tail probability of this count under the rate recent hours imply.
Precision46%Recall86%Fired26Awaiting hindsight012 of 26 firings held up · 12 of 14 surges caught · 115 buckets judged
true positive 12false positive 14missed 2true negative 87zscore
shadowStandard deviations above the trailing mean, with a floor under sigma.
Precision100%Recall57%Fired8Awaiting hindsight08 of 8 firings held up · 8 of 14 surges caught · 115 buckets judged
true positive 8false positive 0missed 6true negative 101
Why this data breaks the textbook detectors
EWMA and rolling z-score were built for a metric series: a continuous signal, sampled regularly, wobbling around a level. Error counts per group are not that. They are non-negative integers, usually small, arriving in bursts — and on this deployment, 97% of group-hour buckets are empty.
Against a baseline of nearly nothing, “several times the usual” and “many sigmas above the mean” are both satisfied by the number two. Every detector here therefore sits behind an absolute floor on the count, which is less a tuning parameter than an admission that ratios carry little information down there.
Above the floor they genuinely differ. A z-score divides by the spread, so a group that is bursty by nature desensitises the detector watching it. Poisson takes the spread from the rate instead, which suits count data — and pays for it when a group is genuinely erratic, where it fires more often than it should. Which failure costs more is not decidable from first principles. That is what the table above is for.