Voidly
All findingsPublished research

A logistic stacker lifts the fused anomaly ensemble from 0.66 to 0.75 AUC

Voidly Atlas fuses four unsupervised anomaly detectors — a DBSCAN per-country shape detector, an STL seasonal-residual detector, a multi-country burst detector, and an HDBSCAN per-domain drift detector — into one composite anomaly score per country per day. The original fusion combined them with a…

Read the full published summary

Voidly Atlas fuses four unsupervised anomaly detectors — a DBSCAN per-country shape detector, an STL seasonal-residual detector, a multi-country burst detector, and an HDBSCAN per-domain drift detector — into one composite anomaly score per country per day. The original fusion combined them with a hand-picked weighted average (0.35/0.25/0.20/0.20), then quietly tried three variants and kept whichever scored the highest AUC on the very labels it then reported against — in-sample model selection, so its published ~0.66-0.68 carried selection leakage. This finding is the rebuild. Fusion v2 changes two things. First, evaluation is rolling-origin forward-temporal cross-validation: each of three folds fits on earlier label dates and scores on a strictly later, held-out block — never a shuffled random split, because a prior Atlas audit showed shuffled splits leak time-autocorrelation between adjacent days and inflate AUC. The reported number is the mean across folds. Second, v2 evaluates five fusion methods, each fit only on the train fold: plain averaging, AUC-weighted averaging, rank-averaging, AUC-weighted rank-averaging, and a small logistic-regression stacker over the four detector scores plus four present/absent indicators. The logistic stacker won decisively — mean held-out composite AUC 0.745 (per-fold 0.753 / 0.769 / 0.715), versus 0.655 for the best averaging variant and 0.584 for plain averaging, which barely clears chance. The stacker beats the best single detector by about ten points, and its worst temporal fold (0.715) is the honest lower bound. So fusion v2 is both higher and more honest than the old number: 0.745 mean on a leak-free split versus 0.663 in-sample-selected. The live /v1/anomaly/fused/* endpoints are unchanged — they just load the updated artifact. Honest caveats baked in: the published 0.745 is the held-out mean while the live per-country composite re-fits the stacker on all labels (standard once a method is selected); two of the four detectors (bursts and HDBSCAN drift) are near-static current-snapshot signals that contribute weakly, so DBSCAN and STL carry most of the discriminative load; and "anomalous" is not "censored" — the fused ensemble is a second-opinion signal, with the supervised v3.3 classifier still the headline censorship predictor.

What this is

Voidly Atlas runs four unsupervised anomaly detectors over the same country-day stream: a DBSCAN per-country shape detector, an STL seasonal-residual detector, a multi-country burst detector, and an HDBSCAN per-domain drift detector rolled up to country level. The fused anomaly ensemble combines all four into one composite anomaly score so a country page can show a single second-opinion number alongside the supervised classifier.

The original fusion (build-fused-anomaly-ensemble.py) combined the four with a hand-picked weighted average (0.35 / 0.25 / 0.20 / 0.20). It then quietly tried three variants — raw, sign-corrected, and drop-below-chance — and kept whichever scored the highest AUC. The catch: it picked that variant on the very labels it then reported the AUC against. That is in-sample model selection. The headline number it published (~0.66–0.68) is therefore optimistic — it includes a small amount of selection leakage.

The fix: honest evaluation + a real stacker

Fusion v2 (build-fused-anomaly-ensemble-v2.py) changes two things. First, evaluation is rolling-origin forward-temporal cross-validation: each fold fits the fusion on earlier label dates and scores it on a strictly later, held-out block of dates. A prior Atlas audit established that shuffled random splits leak time-autocorrelation between adjacent days and inflate AUC — so a shuffled split is deliberately never used here. Three folds run across the 120-day label window, and the reported number is the mean across folds, not a single lucky cut.

Second, v2 evaluates five fusion methods, each fit only on the train fold: plain unweighted averaging, AUC-weighted averaging, rank-average (mean of per-detector rank percentiles), AUC-weighted rank-average, and a small logistic-regression stacker over the four detector scores plus four present/absent indicators. The stacker is the only method that learns how much to trust each detector and how they interact, rather than assuming a fixed blend.

What the held-out folds showed

The logistic stacker won decisively. Mean held-out composite AUC:

  • logistic stacker — 0.745 (per-fold 0.753 / 0.769 / 0.715)
  • AUC-weighted rank-average — 0.655
  • AUC-weighted average — 0.634
  • rank-average — 0.604
  • plain averaging — 0.584

Plain averaging — the obvious baseline — barely clears chance. Every weighting trick on top of averaging helps a little, but none breaks 0.66. Only the stacker, which can down-weight the two weak detectors and learn the interaction between the two strong ones, reaches the mid-0.74 range. It beats the best single detector by about ten points. Critically, the worst of the three temporal folds still scores 0.715, and the mean of 0.745 clears the 0.72 promote floor with margin on a strictly leak-free split.

For comparison, the old in-sample-selected number was 0.663. So fusion v2 is both higher and more honest: a better score on a harder, leak-free test.

Honest caveats

The published 0.745 is the mean across folds; the live per-country composite re-fits the stacker on all labels (standard practice once a method has been selected on held-out data), but the public metric stays the honest held-out figure. Two of the four detectors — multi-country bursts and HDBSCAN domain-drift — are near-static "current snapshot" signals with little historical variation, so they contribute weakly; DBSCAN and STL carry most of the discriminative load. And as always, "anomalous" is not "censored" — a high composite means the day looks unusual on multiple axes, and ground-truthing is still required. The supervised v3.3 classifier remains the headline censorship predictor; the fused ensemble is a second-opinion signal that can surface shape-anomalous days the labels never saw.

The source record.

Original source links and labels. Linked responses may have changed since publication.