Voidly Atlas ships a 7-day internet-shutdown predictor at /v1/shutdown-risk/{cc}, trained and validated against Access Now’s journalist-verified KeepItOn shutdown record. Its public model card has long reported two headline AUCs: a cross-country full-panel AUC ~0.88–0.90 (“which country is at risk”) and a within-country median AUC ~0.73 (“which day inside that country”). This finding is the honest audit of those two numbers — one survives scrutiny and is genuinely strong; the other does not mean what it appears to mean.

Two upgrades, then one hard question

We first improved the model honestly. The live ensemble (v5) combined two base signals — a structural country-risk model (v4, KeepItOn-derived) and an OONI measurement-trajectory model (honest_forecast_v1) — by multiplying them: p = sr × hf. A product requires both signals to be high, so whenever the OONI trajectory was near zero or anti-correlated for a country, the product annihilated an otherwise good structural signal. Oman’s within-country AUC collapsed from 0.498 to 0.040; Sudan, Ethiopia, Indonesia and India lost ground the same way.

v9 replaces the product with a logit-space blend — p = isotonic(sigmoid(0.6·logit(sr) + 0.4·logit(hf))) — which can never let one factor zero out the other. The 0.60 weight was not hand-picked on the evaluation set: it was selected by leave-one-country-out, and all 23 folds independently chose 0.60. The held-out within-country median rose from 0.7291 to 0.7386 (+0.0095, with 18 of 23 countries improving) and the cross-country full-panel AUC rose from 0.8848 to 0.8979. A logistic stacker fit on the two base signals confirmed the imbalance the product had baked in — it put 0.99 weight on structural risk versus 0.056 on the OONI trajectory. v9 is now live.

But a higher in-sample number is not the question that matters to someone deciding whether to trust an alert. The question is: does the model predict the future? So we ran the test the rest of our honesty work demands — a strict past→future temporal holdout. Fit the calibrator only on data up to the 75th-percentile date; score the future; measure skill on the held-out tail.

The within-country “which day” signal does not survive

Forward in time, the within-country day-ranking AUC is ~0.36 — below chance, for both v5 (0.3835) and v9 (0.3555). The published within-country median of ~0.73 is cross-sectional: it is measured with the isotonic calibrator fit on the entire panel, all dates mixed together. It tells you the model can rank the riskier days within a country once it has already seen the whole period. It is not a measure of forward skill, and we should never have let it read like one.

Why does it fall apart forward? Non-stationarity. Internet shutdowns cluster in time — a country that was quiet during the training window and then erupts in the test window inverts the very structural signal the model leans on. Country-level aggregates also miss the roughly half of KeepItOn events that are sub-national (a single region, a single city), which no country-day score can resolve. Predicting the day a shutdown begins inside a given country, from country-level data, going forward, is a problem this model does not solve. We now say so on the model card and the public page.

The cross-country “which country” signal is real — and forward-validated

The good news is the part that matters most. On the same strict past→future holdout, the cross-country ranking holds:

In plain terms: the model reliably tells you which countries are heading into danger, and that skill is validated forward in time, not just in-sample. That is the actual product, and it is strong.

What changed, and what we now claim

v9 stays live — it is the better model at the thing that works (cross-country ranking, full-panel AUC 0.8979 versus v5’s 0.8848). This is not a rollback; it is a correction to how the model is described. The live /v1/shutdown-risk/info now reports all four numbers side by side — cross-sectional within-country (0.74), forward within-country (0.36), forward cross-country (0.87) and forward country-ranking (0.90) — and the lead caveat states plainly that the within-country figure is cross-sectional and that the model is a country risk ranking, not a within-country calendar oracle. The public /shutdown-risk page carries the same framing.

A research organization’s credibility is its product. A model that is strong at what it claims and explicit about what it cannot do is worth more than a bigger number with an asterisk no one reads. Use Voidly shutdown-risk to know which countries to watch — that signal is real, and it holds going forward. Do not use it to bet on which day.