Voidly
All findingsPublished research

Concept-drift detector: catching distribution shift before it poisons the models

Voidly Atlas runs two production models — the v3.3 censorship classifier and the v1 7-day forecast — both retrained weekly behind a dual-holdout promotion gate. But there was no principled detector for the question that sits upstream of the gate: has the live data distribution drifted away from…

Read the full published summary

Voidly Atlas runs two production models — the v3.3 censorship classifier and the v1 7-day forecast — both retrained weekly behind a dual-holdout promotion gate. But there was no principled detector for the question that sits upstream of the gate: has the live data distribution drifted away from what the models were trained on? That gap had a concrete cost. In March-April 2026 a labeling bug let raw IODA connectivity-disruption alerts (fiber cuts, BGP leaks, weather outages) flow into the forecast target_7day labels as confirmed censorship; the April positive rate exploded to 79% against a ~5% training baseline, the model learned to call almost everything a shutdown, and the bug festered for weeks before a human noticed. A distribution-shift detector watching the label rate would have flagged it on day 2. The concept-drift detector closes that gap. A one-time baseline builder freezes each model feature's training distribution — mean, std, a 7-point quantile summary, and the ten decile bin edges plus the training proportion per bin (frozen bins are what make the daily comparison honest: live data is binned into the exact same edges). Every day the detector recomputes each model's features on the last 7 days of live data using the same derivation as the training pipeline, and runs two tests per feature: Population Stability Index (PSI) — sum((live% - train%) * ln(live%/train%)) over the ten frozen bins, the primary signal, with standard industry thresholds 0.2 = significant drift and 0.25 = major drift — and a two-sample Kolmogorov-Smirnov statistic as a secondary cross-check. Separately it tracks LABEL drift: the target positive-rate over the trailing 30 days vs the training baseline — the metric that would have caught the IODA disaster. A composite drift_score in [0,1] combines 60% mean feature PSI and 40% label drift, floored at retrain-recommended whenever label drift alone is "major" so the IODA failure mode always triggers a retrain regardless of feature PSI, and maps to a verdict: stable / watch / drifted / retrain-recommended. When a model crosses retrain-recommended the detector writes a flag to the existing drift-trigger queue (retrain-queue.json) that weekly-retrain.sh already reads, so the next retrain runs early — it deliberately does NOT auto-retrain on the spot (retrains are expensive; a 12-hour cooldown serializes drift-driven retrains with the weekly cron). The first run surfaced a built-in trap of any naive PSI monitor: the calendar features month, week_of_year and day_of_week posted PSI of 13 to 17, orders of magnitude past threshold — not drift, but structure, because a 7-day window only ever spans one month and one or two ISO weeks so its decile distribution over a calendar feature can never match a full-year baseline. The detector flags those five cyclical features, still reports their PSI for transparency, but EXCLUDES them from the composite score; before that fix both models read retrain-recommended on calendar noise alone. After it, the verdicts are honest: classifier v3.3 scored drift_score 0.08, stable — its real features (anomaly rate, measurement count, probe signals) all well inside the training distribution, label drift 0.01 (live 30-day censorship positive rate 0.274 vs training baseline 0.263, the IODA label fix holding); forecast v1 scored drift_score 0.71, retrain-recommended — a genuine signal, with the block_rate lag and rolling-mean features (14 of 38) showing PSI ~3.4 because over the identical 21-country set the mean 7-day block rate jumped roughly tenfold (0.023 historical baseline to 0.234 in the trailing week), putting the forecast's recent-history inputs well outside the distribution the current model learned from (forecast label drift a modest 0.10 — not the trigger; the feature drift is). Honest caveats baked into every response: (1) drift is not the same as a broken model — a real censorship surge IS distribution shift and the forecast's 10x block-rate jump may be a genuine escalation the model still handles, so the verdict says "the inputs have moved, a retrain is warranted" not "the model is wrong today"; (2) the PSI thresholds 0.2/0.25 are industry convention, not derived from Voidly's data; (3) KS on a 7-day window is noisy — lean on PSI and the composite score, and KS here is computed against a sample reconstructed from stored training quantiles, a coarse cross-check; (4) the classifier's three contagion features (neighbor_*) are not monitored — they need the offline adjacency + regime-correlation pipeline; (5) the forecast baseline excludes the most-recent 30 days so label drift is baseline-vs-recent. Live at GET /v1/atlas/concept-drift (both models, per-feature PSI/KS, drift_score, verdict, label drift) + GET /v1/atlas/concept-drift/{model} (classifier-v3.3 | forecast-v1) + GET /v1/atlas/concept-drift/info. Refreshed daily 05:20 UTC. Implementation: scripts/build-concept-drift-baseline.py + detect-concept-drift.py + patch-concept-drift-endpoint.py.

What this is

Voidly Atlas runs two production models: the v3.3 censorship classifier (detects censorship in already-collected country-day evidence) and the v1 7-day forecast (predicts shutdown risk). Both are retrained on a weekly cron, and a dual-holdout gate blocks a regressing model from being promoted. But there was no principled detector for the question that sits upstream of the gate: has the live data distribution drifted away from what the models were trained on?

That gap had a concrete cost. In March–April 2026 a labeling bug let raw IODA connectivity-disruption alerts (fiber cuts, BGP leaks, weather outages) flow into the forecast’s target_7day labels as if they were confirmed censorship. The April positive rate exploded to 79% against a training baseline near 5%. The model dutifully learned to call almost everything a shutdown. The bug festered for weeks before a human noticed. A distribution-shift detector watching the label rate would have flagged it on day 2.

The concept-drift detector (scripts/detect-concept-drift.py, daily 05:20 UTC) closes that gap. It is a monitor, not a gate — it does not block anything; it raises a verdict and, when the verdict is severe enough, queues an early retrain.

How it works

A one-time baseline builder (scripts/build-concept-drift-baseline.py, also re-run after every retrain) freezes the training-time distribution of each model’s features: mean, standard deviation, a 7-point quantile summary, and — critically — the ten decile bin edges of the training distribution plus the training proportion in each bin. Freezing the bins is what makes the daily comparison honest: live data is binned into the exact same edges.

Every day the detector recomputes each model’s features on the last 7 days of live data, using the same derivation code path as the training pipeline (the classifier’s country-day aggregation and the forecast’s feature parquet), and runs two tests per feature:

  • Population Stability Index (PSI) — sum((live% − train%) × ln(live% / train%)) over the ten frozen bins. PSI is the primary signal. Standard industry thresholds: > 0.2 = significant drift, > 0.25 = major drift.
  • Kolmogorov–Smirnov (KS) statistic — a two-sample test, computed as a secondary cross-check against a synthetic sample reconstructed from the stored training quantiles.

Separately it tracks label drift: the positive rate of the target over the trailing 30 days versus the training baseline. This is the metric that would have caught the IODA disaster.

A composite drift_score in [0, 1] combines 60% mean feature PSI and 40% label drift, and is floored at retrain-recommended whenever label drift alone is “major” — so the IODA failure mode always triggers a retrain regardless of feature PSI. The score maps to a verdict: stable / watch / drifted / retrain-recommended.

When a model crosses retrain-recommended, the detector writes a flag to the existing drift-trigger queue (/opt/voidly-ai/data/retrain-queue.json), which weekly-retrain.sh already reads — so the next retrain runs early. It deliberately does not auto-retrain on the spot: retrains are expensive, and a 12-hour cooldown serializes drift-driven retrains with the weekly cron.

The calendar-feature trap

A naive feature-by-feature PSI monitor has a built-in bug, and the first run surfaced it loudly. The features month, week_of_year and day_of_week posted PSI values of 13 to 17 — orders of magnitude past any threshold. That is not drift. A 7-day live window only ever spans one calendar month and one or two ISO weeks, so its decile distribution over a calendar feature can never match a baseline drawn from a full year. The PSI is structurally guaranteed to be enormous and says nothing about model health.

The detector flags those five calendar features (month, week_of_year, day_of_week, is_weekend, is_friday) as cyclical, still reports their PSI for transparency, but excludes them from the composite drift_score. Before that fix both models read retrain-recommended on calendar noise alone. After it, the verdicts are honest.

What the first run found

On 21 May 2026 the detector scored both models:

  • Classifier v3.3 — drift_score 0.08, stable. Its real features (anomaly rate, measurement count, probe signals) are all well inside their training distribution. Label drift is 0.01: the live 30-day censorship positive rate (0.274) almost exactly matches the training baseline (0.263). The IODA label fix is holding.
  • Forecast v1 — drift_score 0.71, retrain-recommended. This is a genuine signal. The block_rate lag and rolling-mean features — 14 of the 38 forecast features — show PSI around 3.4. Over the identical 21-country set, the mean 7-day block rate jumped roughly tenfold (0.023 in the historical baseline to 0.234 in the trailing week). The forecast’s recent-history inputs are sitting well outside the distribution the current model learned from. Forecast label drift is a modest 0.10 — not the trigger; the feature drift is.

The detector queued the early retrain via the shared queue (subject to the 12-hour cooldown, which deferred to an already-pending entry on the first run).

Honest caveats

  • Drift is not the same as a broken model. A real real-world censorship surge is distribution shift — and the forecast’s 10× block-rate jump may well be a genuine escalation the model still handles acceptably. The verdict says “the inputs have moved, a retrain is warranted,” not “the model is wrong today.”
  • The PSI thresholds (0.2 / 0.25) are industry convention, not derived from Voidly’s data. They are a reasonable default, not a tuned cut point.
  • KS on a 7-day window is noisy. Treat single-feature KS spikes with suspicion; lean on PSI and the composite score. KS here is computed against a sample reconstructed from stored training quantiles — a coarse cross-check, not the primary test.
  • The classifier’s three contagion features (neighbor_*) are not monitored: they need the offline adjacency and regime-correlation pipeline and cannot be recomputed on a 7-day live window.
  • The forecast baseline excludes the most-recent 30 days so label drift is a baseline-vs-recent comparison, not recent-vs-itself.

Where to look

Live at GET /v1/atlas/concept-drift (both models, per-feature PSI/KS, drift_score, verdict, label drift), GET /v1/atlas/concept-drift/{model} for one model (classifier-v3.3 or forecast-v1; short aliases classifier / forecast accepted), and GET /v1/atlas/concept-drift/info for the full methodology. Refreshed daily at 05:20 UTC. Implementation: scripts/build-concept-drift-baseline.py + scripts/detect-concept-drift.py + scripts/patch-concept-drift-endpoint.py.

The source record.

Original source links and labels. Linked responses may have changed since publication.