No improvement established: 0 of 12 selected models beat every baseline on onset-event F1. This is a negative result from one frozen, offline v1.3 backtest. The point scores are numerically provisional because the scoring process emitted unresolved RuntimeWarnings. This result does not qualify a model for live warnings.
The question
Can news about protests and violence help forecast a new internet shutdown beyond what a country's past shutdowns already tell us? We tested added news and exam-season signals against history-based models, persistence and an always-alert comparator. We publish the miss as well as the method.
What went in
The source audit counted 92,674,204 qualified GDELT Event1 rows from January 2017 through September 2025. These are coded news-event records, not that many distinct conflicts or protests. We aggregated them into daily protest and violence signals for a 152-country roster, alongside historical shutdown and exam-season features.
Event1 is English-focused. The archive had 3,175 available days and 20 explicitly missing days; geographic mapping is incomplete. Missing days and unknown geography were not converted to zero. The news adapter used archive days through two days before each forecast date and required the archive's Last-Modified time to precede that date. The full source span is not the training period: training and model selection used pre-2024 issue data.
The frozen protocol
We froze the selected models and thresholds, fingerprinted the bundle and made one guarded score attempt after a separate approval. The pretest commitment SHA-256 is 645fa49cb2e7f32eda81cf2abc39655e2482a4dde8c85e4d11c036e94f8cc459.
The score covers forecast issue dates from January 1 through December 31, 2024 only. The models abstained on every 2025 issue date because issue-time 2024 episode history was unavailable. A forecast issued late in December can reach a January 2025 shutdown start. This is not a full 2024–2025 test.
The targets are new shutdown starts within 3-, 7- and 30-day windows. Matching is one-to-one at the onset-event level, with at most two alerts per country-month. Every arm within a cell uses the same eligible exposure panel. F1 combines precision and recall. A selected model had to strictly beat every comparator; a tie did not pass.
The complete result
Six cells selected logistic history itself. The other six selected added-feature models, and none beat every baseline. Across all 12 cells, the strict pass count is 0/12. This does not establish that conflict news has no signal; it establishes no improvement under this protocol.
For all shutdowns over a 30-day window, the selected model scored 0.3215 F1 versus 0.3220 for country base rate. These are point estimates, not a statistically established difference.
In the table, hits / opportunity / span / all means matched starts, starts with an eligible scored 2024 forecast opportunity, documented starts reachable from any 2024 forecast window, and all documented 2024–2025 starts in the frozen roster. The groups overlap; do not sum their counts. All six score columns are event F1 rounded to four decimal places. Exact values are in the frozen result JSON.
| Group | Window (days) | Selected model | Hits / opportunity / span / all | Alerts | Selected F1 | Logistic history | Country base rate | Frozen persistence | Retrospective persistence | Always alert |
|---|---|---|---|---|---|---|---|---|---|---|
| All | 3 | logistic_history_exam | 8 / 99 / 164 / 295 | 98 | 0.0812 | 0.0889 | 0.0736 | 0.0194 | 0.0976 | 0.0096 |
| All | 7 | logistic_history | 16 / 111 / 166 / 295 | 80 | 0.1675 | 0.1675 | 0.1375 | 0.0174 | 0.1467 | 0.0203 |
| All | 30 | gbm_history_exam_reactive | 59 / 158 / 181 / 295 | 209 | 0.3215 | 0.3092 | 0.3220 | 0.0494 | 0.3055 | 0.0601 |
| Scheduled | 3 | logistic_history | 2 / 19 / 25 / 41 | 43 | 0.0645 | 0.0645 | 0.0000 | 0.0870 | 0.0385 | 0.0016 |
| Scheduled | 7 | logistic_history | 2 / 20 / 25 / 41 | 18 | 0.1053 | 0.1053 | 0.1500 | 0.0833 | 0.0574 | 0.0044 |
| Scheduled | 30 | logistic_history | 6 / 26 / 26 / 41 | 16 | 0.2857 | 0.2857 | 0.3182 | 0.0667 | 0.1442 | 0.0137 |
| Reactive | 3 | gbm_history_exam | 7 / 57 / 115 / 188 | 133 | 0.0737 | 0.0947 | 0.0857 | 0.0000 | 0.0735 | 0.0076 |
| Reactive | 7 | logistic_history | 12 / 69 / 117 / 188 | 84 | 0.1569 | 0.1569 | 0.1194 | 0.0000 | 0.1240 | 0.0124 |
| Reactive | 30 | logistic_history | 33 / 102 / 125 / 188 | 86 | 0.3511 | 0.3511 | 0.2927 | 0.0566 | 0.2887 | 0.0391 |
| Platform block | 3 | gbm_history_exam_reactive | 0 / 20 / 28 / 57 | 49 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0096 | 0.0005 |
| Platform block | 7 | gbm_history_exam_reactive | 0 / 22 / 28 / 57 | 12 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0095 | 0.0038 |
| Platform block | 30 | logistic_history_exam | 7 / 33 / 33 / 57 | 203 | 0.0593 | 0.0559 | 0.1053 | 0.0000 | 0.0569 | 0.0148 |
Model names identify their inputs: history uses shutdown history, exam adds exam-season patterns, and the reactive feature suffix adds Event1 signals. Logistic and GBM denote the model families. Frozen persistence uses pre-2024 history. Retrospective persistence also uses earlier gated 2024 episodes with onset upper bounds before the issue date; it has no historical publication-time receipt and is a comparator only, never a candidate feature.
What the history-only model found
For the reactive group over a 30-day window, the selected history-only model matched 33 of 102 eligible starts and issued 86 alerts. Event precision was 0.384 (33/86); recall among eligible starts was 0.324 (33/102). There were 125 documented starts reachable from the forecast windows, so recall over that whole span was 26.4% (33/125). The wider frozen 2024–2025 roster contained 188 reactive starts. These are different denominators, not interchangeable success rates. They describe this provisional backtest, not operational warning accuracy.
Limits that matter
- Retrospective history: annual KeepItOn episode labels were reconstructed in 2026 and lack historical
known_atpublication receipts. The history features are retrospective assumptions. This is an offline backtest, not prospective validation. - Assumed negatives: 70 of the 152 countries have no documented episode in the annual extract and were treated as no-event negatives. They are not verified negatives. The other 82 countries have documented episodes.
- Source gaps: English-focused Event1 has incomplete geographic mapping and missing archive windows. Election calendars were unavailable. This test did not evaluate a full multilingual or first-party network precursor model.
- Selection and earlier exposure: prior v1 aggregate test results were already known. Model and threshold selection used the same pretest validation folds, and the complete selection rule was recorded after some results had been viewed. One frozen v1.3 score does not make this a pristine blind evaluation.
- Uncertainty: the JSON contains per-arm Wilson intervals for event precision and recall, but no paired heldout intervals for F1 differences. Point comparisons are descriptive. Zero observed matches do not prove a true zero rate.
- Numerical warnings remain unresolved: scikit-learn emitted divide-by-zero, overflow and invalid-value warnings during matrix multiplication. The process completed and the JSON is finite, but the cause and any effect on alert ranking remain unverified. Treat all point scores here as numerically provisional.
Evidence and next steps
The complete score receipt has SHA-256 aa756df39b6c0ce849626e767d552c98f8b5e979ade93757c3d1105728a136ee. The source audit has SHA-256 d03b14ff2ce16376f083fff1634e7ecf528e6bbdac9b59d3783eec24cf126b16. These frozen records support the figures above; they are not a complete release of the training data or a claim of independently reproduced results.
Next, we plan to test multilingual news and precursors from our own network probes, while investigating the numerical warnings. Those inputs have not earned a performance claim from this run. Any new evaluation needs its own declared protocol; this exposed result cannot be reused as fresh blind evidence.
Paired: the earlier 7-day forecast audit and its persistence baseline.