A prior audit (forecast-v2-momentum-vs-persistence-2026-05)
proved Voidly's production 7-day forecast has no onset skill.
Its target, target_7day, is a 7-day
sliding window: adjacent country-days share 6 of 7 lookahead
days, so the label is 98.9% autocorrelated. A model — or a trivial
predict-yesterday baseline — scores AUC ~0.95 just reproducing that
near-constant label, but on the rows where a shutdown actually
begins, the forecast scored AUC ~0.33, below chance. The
production forecast measures "is this country currently
censored", not "will a new shutdown start."
This finding attacks the real problem head-on: build a dedicated shutdown-onset predictor on a clean onset label and evaluate it honestly. The headline up front: under a forward-temporal split, onset is not predictable 7 days out from the features Voidly has. Four model families — XGBoost, gradient boosting, and a scaled balanced logistic regression — all land at AUC ~0.45–0.49 on strictly-future onset events. A 3-fold walk-forward averages 0.495. This is a definitive honest negative. Nothing is promoted to production.
For every country-day t, we first decide whether a shutdown is
already active. A confirmed censorship/mixed incident is
treated as active over [first_seen−3d,
last_seen+14d] — the 3-day grace absorbs detection lag, the
14-day cooldown keeps the tail of one shutdown from being mislabelled as
fresh ground. Eligible rows are country-days with no
shutdown active. The label onset_7d = 1
iff a new incident's first_seen
lands in (t, t+7] — evaluated only on
eligible rows. Conditioning on "no active shutdown now" is exactly what
turns this into an onset problem rather than a persistence one.
IODA disruption incidents (fiber cuts,
DDoS, BGP leaks, weather) are excluded — they are real
network observations but not confirmed censorship, and counting them as
onsets would relabel infrastructure failures as intentional shutdowns.
Only incident_type IN ('censorship','mixed')
count. The grid spans every country with ≥2 confirmed incidents in a
2-year window (40 countries) so the rare onset events are not starved.
| Quantity | Value |
|---|---|
| Grid country-days (40 countries, 2024-05-22 → 2026-05-22) | 29,971 |
| Rows with a shutdown active now (dropped) | 8,509 |
| Eligible onset rows | 21,462 |
| Onset events (onset_7d = 1) | 297 |
| Onset base rate | 1.38% |
Everything below uses a forward-temporal split only:
sort by date, train on the past, test on the strictly-future tail. No
shuffling, ever — a random K-fold would scatter rows of the same country
across folds and leak the future. The split cut is 2026-01-21, chosen as
the latest date that still keeps ≥60 onsets in train and ≥30 in
test: 19,226 train rows / 156 onsets vs.
2,236 test rows / 141 onsets. Onset is rare and class-
imbalanced (1.4%), so every model is class-balanced — scale_pos_weight
~122 for the trees, class_weight="balanced"
for the logistic regression — otherwise the model collapses to all-zero.
| Model / baseline | Forward-temporal test AUC | Verdict |
|---|---|---|
| Base-rate constant predictor | 0.500 | by construction |
| block_rate today (1-feature) | 0.456 | no signal |
| block_rate yesterday (predict-yesterday analogue) | 0.447 | no signal |
| XGBoost (16 + momentum features) | 0.456 | at chance |
| Gradient boosting | 0.448 | at chance |
| Scaled balanced logistic regression (best) | 0.490 | at chance |
| 3-fold expanding walk-forward (mean) | 0.495 | at chance, every fold |
The best model scores AUC 0.490 on strictly-future onset events — it does not clear the 0.60 promote floor, and it does not beat a base-rate constant. The walk-forward folds are 0.529 / 0.440 / 0.517: every fold hovers at chance, which is what makes the negative defensible — a tuning artifact would make one fold jump. On the precision side there is no usable operating point: the model never reaches even 25% precision, so the F1-optimal "threshold" collapses to 0.0 (predict every day positive) for recall 0.75 at 7% precision — useless. A per-country breakdown of the test fold averages AUC 0.515 across the 9 countries with ≥5 onsets; the apparent winners (EG 0.97 on 8 events) and losers (IQ 0.11 on 8 events) are the scatter you expect from single-digit event counts, not a real pocket of skill.
An earlier attempt at this build produced the same verdict but for the
wrong reason, and was discarded. Its logistic regression ran on
unscaled features whose standard deviations span 0 to
~500 (gdelt_unrest_30d alone reaches
|v| ~8,000). The lbfgs solver cannot converge on a matrix that
badly conditioned inside 2,000–3,000 iterations; it returned a
near-constant prediction (its F1-optimal threshold was 0.0 — the
fingerprint of an unconverged model), and the run stalled long enough to
be killed by a watchdog. The fix: standard-scale the
features before the logistic regression (a scikit-learn
Pipeline; the tree models are scale-
invariant and need no scaling). After scaling, lbfgs converges in
53 iterations / 1.8 seconds. The result reported
here comes from a converged model — and it agrees with both
scale-free tree models, all three independently at ~0.45–0.49.
The metrics JSON now carries a
logreg_convergence block so this failure
mode can never again be mistaken for a real result.
Shutdown onset is not predictable 7 days out from the features Voidly currently has. The leading indicators in the stack — censorship momentum, acceleration, volatility, election proximity, protest/unrest signals, cross-country contagion — do not separate the country-days that precede a new shutdown from those that don't. This is not a calibration problem or a tuning miss; it holds across four model families, three walk-forward folds, and a per-country breakdown, all at chance.
This is plausible on its own terms. A new internet shutdown is a political decision — an election dispute boiling over, a protest a government decides to suppress, a coup. The measurable precursors to that decision largely are not in network telemetry; they are in situation rooms. With 297 onset events spread across 40 countries — roughly 7 per country over two years — there is also simply too little signal to learn a country-specific onset model even if one exists. Honestly publishing that, rather than shipping a forecast that cannot do what its name implies, is the finding.
Nothing is promoted. The production 7-day forecast
(forecast v1) is unchanged — it remains a
useful current-regime risk-and-calibration signal, just not an onset
predictor, and its endpoint already says so. The onset model is exposed
as a transparency artifact at
GET /v1/forecast/{cc}/onset and
/v1/forecast/onset/info, with
model_promoted: false, the
forward-temporal AUC, and honest caveats inline in every response — a
research signal, never a calibrated forecast. The reproducible pipeline
is scripts/build-onset-features.py +
scripts/train-onset-model.py.