voidly

Shutdown-risk predicts WHICH country, not WHICH day — a forward-validated honest scoping (v9)

Voidly's 7-day shutdown predictor (/v1/shutdown-risk) reported two AUCs: cross-country full-panel ~0.88-0.90 and within-country median ~0.73. This week we shipped v9 (a logit-blend combiner that replaced v5's multiplicative product — the product collapsed when the OONI trajectory was near-zero, e.g. Oman within-AUC 0.498→0.040; alpha=0.60 chosen by leave-one-country-out, all 23 folds agreed; within-country median 0.7291→0.7386, full-panel 0.8848→0.8979). Then we ran the test that actually matters: a strict past→future temporal holdout. The within-country day-ranking AUC is ~0.36 forward — BELOW chance — for both v5 and v9. The published ~0.73 is cross-sectional (isotonic fit on the whole panel, all dates mixed); it measures day-ranking once the model has seen the period, not forward skill, and shutdowns cluster in time so a country quiet in train then active in test inverts the structural signal. BUT the cross-country ranking — the real product — holds forward: full-panel AUC 0.8653 out-of-time (vs 0.8912 in-sample), country-ranking AUC 0.9015 across 60 countries, 7.84x PR-lift over base rate. Verdict: v9 stays live (better at the cross-country ranking that works); the within-country claim was overstated and is now corrected. The live /v1/shutdown-risk/info reports all four numbers (cross-sectional within 0.74, forward within 0.36, forward cross-country 0.87, forward country-ranking 0.90) with a lead caveat; the public page now frames it as a forward-validated country risk ranking, not a within-country calendar oracle. Use it to know WHICH countries to watch — that signal is real and holds going forward — not WHICH day.

#ml#shutdown-risk#forecasting#temporal-cv#honest-scoping#leave-one-country-out#keepiton#accountability#atlas#api

Raw data