What this is

Voidly Atlas runs two production models: the v3.3 censorship classifier (detects censorship in already-collected country-day evidence) and the v1 7-day forecast (predicts shutdown risk). Both are retrained on a weekly cron, and a dual-holdout gate blocks a regressing model from being promoted. But there was no principled detector for the question that sits upstream of the gate: has the live data distribution drifted away from what the models were trained on?

That gap had a concrete cost. In March–April 2026 a labeling bug let raw IODA connectivity-disruption alerts (fiber cuts, BGP leaks, weather outages) flow into the forecast’s target_7day labels as if they were confirmed censorship. The April positive rate exploded to 79% against a training baseline near 5%. The model dutifully learned to call almost everything a shutdown. The bug festered for weeks before a human noticed. A distribution-shift detector watching the label rate would have flagged it on day 2.

The concept-drift detector (scripts/detect-concept-drift.py, daily 05:20 UTC) closes that gap. It is a monitor, not a gate — it does not block anything; it raises a verdict and, when the verdict is severe enough, queues an early retrain.

How it works

A one-time baseline builder (scripts/build-concept-drift-baseline.py, also re-run after every retrain) freezes the training-time distribution of each model’s features: mean, standard deviation, a 7-point quantile summary, and — critically — the ten decile bin edges of the training distribution plus the training proportion in each bin. Freezing the bins is what makes the daily comparison honest: live data is binned into the exact same edges.

Every day the detector recomputes each model’s features on the last 7 days of live data, using the same derivation code path as the training pipeline (the classifier’s country-day aggregation and the forecast’s feature parquet), and runs two tests per feature:

• Population Stability Index (PSI) — sum((live% − train%) × ln(live% / train%)) over the ten frozen bins. PSI is the primary signal. Standard industry thresholds: > 0.2 = significant drift, > 0.25 = major drift.
• Kolmogorov–Smirnov (KS) statistic — a two-sample test, computed as a secondary cross-check against a synthetic sample reconstructed from the stored training quantiles.

Separately it tracks label drift: the positive rate of the target over the trailing 30 days versus the training baseline. This is the metric that would have caught the IODA disaster.

A composite drift_score in [0, 1] combines 60% mean feature PSI and 40% label drift, and is floored at retrain-recommended whenever label drift alone is “major” — so the IODA failure mode always triggers a retrain regardless of feature PSI. The score maps to a verdict: stable / watch / drifted / retrain-recommended.

When a model crosses retrain-recommended, the detector writes a flag to the existing drift-trigger queue (/opt/voidly-ai/data/retrain-queue.json), which weekly-retrain.sh already reads — so the next retrain runs early. It deliberately does not auto-retrain on the spot: retrains are expensive, and a 12-hour cooldown serializes drift-driven retrains with the weekly cron.

The calendar-feature trap

A naive feature-by-feature PSI monitor has a built-in bug, and the first run surfaced it loudly. The features month, week_of_year and day_of_week posted PSI values of 13 to 17 — orders of magnitude past any threshold. That is not drift. A 7-day live window only ever spans one calendar month and one or two ISO weeks, so its decile distribution over a calendar feature can never match a baseline drawn from a full year. The PSI is structurally guaranteed to be enormous and says nothing about model health.

The detector flags those five calendar features (month, week_of_year, day_of_week, is_weekend, is_friday) as cyclical, still reports their PSI for transparency, but excludes them from the composite drift_score. Before that fix both models read retrain-recommended on calendar noise alone. After it, the verdicts are honest.

What the first run found

On 21 May 2026 the detector scored both models:

• Classifier v3.3 — drift_score 0.08, stable. Its real features (anomaly rate, measurement count, probe signals) are all well inside their training distribution. Label drift is 0.01: the live 30-day censorship positive rate (0.274) almost exactly matches the training baseline (0.263). The IODA label fix is holding.
• Forecast v1 — drift_score 0.71, retrain-recommended. This is a genuine signal. The block_rate lag and rolling-mean features — 14 of the 38 forecast features — show PSI around 3.4. Over the identical 21-country set, the mean 7-day block rate jumped roughly tenfold (0.023 in the historical baseline to 0.234 in the trailing week). The forecast’s recent-history inputs are sitting well outside the distribution the current model learned from. Forecast label drift is a modest 0.10 — not the trigger; the feature drift is.

The detector queued the early retrain via the shared queue (subject to the 12-hour cooldown, which deferred to an already-pending entry on the first run).

Honest caveats

• Drift is not the same as a broken model. A real real-world censorship surge is distribution shift — and the forecast’s 10× block-rate jump may well be a genuine escalation the model still handles acceptably. The verdict says “the inputs have moved, a retrain is warranted,” not “the model is wrong today.”
• The PSI thresholds (0.2 / 0.25) are industry convention, not derived from Voidly’s data. They are a reasonable default, not a tuned cut point.
• KS on a 7-day window is noisy. Treat single-feature KS spikes with suspicion; lean on PSI and the composite score. KS here is computed against a sample reconstructed from stored training quantiles — a coarse cross-check, not the primary test.
• The classifier’s three contagion features (neighbor_*) are not monitored: they need the offline adjacency and regime-correlation pipeline and cannot be recomputed on a 7-day live window.
• The forecast baseline excludes the most-recent 30 days so label drift is a baseline-vs-recent comparison, not recent-vs-itself.

Where to look

Live at GET /v1/atlas/concept-drift (both models, per-feature PSI/KS, drift_score, verdict, label drift), GET /v1/atlas/concept-drift/{model} for one model (classifier-v3.3 or forecast-v1; short aliases classifier / forecast accepted), and GET /v1/atlas/concept-drift/info for the full methodology. Refreshed daily at 05:20 UTC. Implementation: scripts/build-concept-drift-baseline.py + scripts/detect-concept-drift.py + scripts/patch-concept-drift-endpoint.py.