Voidly
All findingsPublished research

Shutdown duration is not predictable — an honest audit of the survival model

Voidly Atlas ships a shutdown-duration model — a Random Survival Forest (RSF) that answers "once a shutdown starts, how long will it last?" — served at POST /v1/forecast/duration. The model card reported a concordance index (c-index) of 0.728, above the 0.65 promote floor and flagged…

Read the full published summary

Voidly Atlas ships a shutdown-duration model — a Random Survival Forest (RSF) that answers "once a shutdown starts, how long will it last?" — served at POST /v1/forecast/duration. The model card reported a concordance index (c-index) of 0.728, above the 0.65 promote floor and flagged passed_promote_floor: true. A platform-wide ML audit had caught several Voidly models inflating metrics via shuffled train/test splits; the duration RSF had not been individually audited. This finding is that audit. How the 0.728 was computed: scripts/train-shutdown-duration-rsf.py evaluates the model with a single random 75/25 train_test_split (random_state=42), stratified only on the censoring flag — no temporal ordering, no country grouping. Three problems sink that number. (1) The point estimate is a lucky seed: only 74 of 343 incidents have an observed end (78.4% censored), so a 25% test fold holds ~19 events and ~286 comparable pairs — tiny and noisy. Re-running the identical random split across 20 seeds gives 0.728 ± 0.055, range 0.62–0.82; the published number is just where seed 42 landed. (2) The random split leaks calendar time: the top permutation feature is first_seen_year, and that is leakage not signal — the "event observed" label is status=confirmed, and confirmed status is an artifact of incident AGE, not of the shutdown ending. 2019–2023 incidents are ~100% confirmed; 2024–2026 incidents are 9–56% confirmed. A shuffle scatters old and new years across both folds, so the RSF learns "old year ⇒ resolved row" — a dataset-assembly pattern that cannot generalise to a live shutdown today. (3) The honest split shows no skill: a forward-temporal split (train on shutdowns whose outcome was known before a cutoff, test on strictly later ones, sorted by end date) yields c-index 0.609 / 0.563 / 0.571 / 0.437 across cutoffs 2022–2025 — mean ≈ 0.55, and the most-recent fold (the one matching real use) scores 0.44, below a coin flip. The promote gate requires beating a naive baseline — predict this country's mean past observed duration. On the identical temporal folds the naive baseline scores 0.509; the RSF's honest 0.55 is within noise of it and loses outright on two of four folds. Giving the model its best shot — an enriched feature set (incident confidence, measurement count, anomaly rate, source/domain/service/ASN counts, severity grade, blocking mechanism), dropping the leaky year — scored 0.495 on the forward-temporal split. More features did not help. Verdict: honest negative. No duration model beats a naive country-mean baseline on a leak-free split; nothing is promoted. With Voidly's current data, how long a shutdown lasts is not predictable — too few resolved events (~one per country), an "end" label contaminated by curation lag rather than measurement, and durations quantized to 24-hour ingestion multiples. The artifact and sidecar now carry the honest c-index, passed_promote_floor: false and an honest_caveats block; the endpoint surfaces the audit verdict. Publishing the honest negative — and not shipping a number we cannot stand behind — is the fix.

Voidly Atlas ships a shutdown-duration model: a Random Survival Forest (RSF) that answers “once an internet shutdown or censorship event starts, how long will it last?” The model card reported a concordance index (c-index) of 0.728 — comfortably above the 0.65 promote floor, flagged passed_promote_floor: true, and served live at POST /v1/forecast/duration. This finding is an individual honest audit of that number, prompted by a platform-wide ML review that caught several Voidly models reporting inflated metrics from shuffled train/test splits. The duration RSF had not yet been audited. It has now, and the headline number does not survive.

How the 0.728 was computed

The training script scripts/train-shutdown-duration-rsf.py evaluates the RSF with a single call to scikit-learn’s train_test_split(…, test_size=0.25, random_state=42) — a random 75/25 shuffle, stratified only on the censoring flag. There is no temporal ordering and no country grouping. The c-index 0.728 is the concordance of the RSF’s risk scores on that one random test fold of 86 incidents.

Three problems make that number unreliable, in increasing order of severity.

Problem 1 — the point estimate is a lucky seed

Of 343 incidents, only 74 have an observed end (the rest are right-censored: 78.4% censoring). A 25% test fold therefore contains roughly 19 events, and a c-index is computed from event-vs-other pairs — the reported run had just 209 concordant + 77 discordant pairs. That is a tiny, noisy sample. Re-running the identical random split across 20 different seeds gives a c-index of 0.728 ± 0.055, ranging from 0.62 to 0.82. The published 0.728 is simply where random_state=42 happened to land in a wide band — not a stable estimate of skill.

Problem 2 — the random split leaks calendar time

The single most informative feature, by permutation importance, is first_seen_year (and the runner-up, prior_shutdowns_count_24mo, is also a recency proxy). Here is why that is leakage rather than signal: the “event observed” label is defined as status = 'confirmed', and confirmed status is overwhelmingly an artifact of incident age, not of the shutdown actually ending.

Incident yearIncidentsObserved-end rate
2019–202327~100%
20243315%
20252556%
20262539%

Old incidents have been curated and marked confirmed; recent ones are still active. A random shuffle puts old and new years in both train and test, so the model learns “old year ⇒ this row is one of the resolved ones” — a pattern that exists only because of how the dataset was assembled, and that cannot generalise to a live shutdown happening today.

Problem 3 — the honest split shows no skill

The correct evaluation for a duration model is a forward-temporal split: train on shutdowns whose outcome was known before a cutoff, test on strictly later ones. Sorting incidents by their end date (last_seen) and sweeping the cutoff year:

Train cutoffHonest temporal c-index
≤ 20220.609
≤ 20230.563
≤ 20240.571
≤ 20250.437

The mean honest forward-temporal c-index is about 0.55, and the most recent fold — the one closest to how the model is actually used — scores 0.44, below a coin flip. A leave-country-out cross-validation pooled across all 65 countries lands at 0.71, but LOCO still mixes early and late incidents within each fold, so it inherits the same year leak; it is not the honest number for a model whose job is to predict the future.

Does it beat a naive baseline? No.

The promote gate requires a duration model to beat a naive baseline — predict this country’s mean past observed duration. On the identical forward-temporal folds, that naive baseline scores a mean c-index of 0.509. The RSF’s honest 0.55 is within noise of it, and on two of the four folds the RSF scores below naive. We also gave the model its best fair shot: an enriched feature set adding incident confidence, measurement count, anomaly rate, source count, affected-domain /service/ASN counts, severity grade and blocking mechanism — and dropping the leaky first_seen_year. The enriched RSF scored 0.495 on the forward-temporal split. More features did not help. There is no version of this model that clears the gate.

Evaluationc-indexHonest?
Published (random split, seed 42)0.728no — leaks year
Random split, mean of 20 seeds0.728 ± 0.055no — still leaks year
Leave-country-out, pooled0.712partial — still mixes time
Forward-temporal, base features0.559yes
Forward-temporal, enriched features0.495yes
Naive baseline (country mean duration)0.509yes
Coin flip0.500—

Verdict: honest negative — shutdown duration is not predictable here

No duration model — the original RSF or the enriched rebuild — beats a naive country-mean baseline on a leak-free forward-temporal split. Per the promote gate, nothing is promoted. The honest finding is that, with the data Voidly currently holds, how long a shutdown will last is not predictable from country, region, severity, prior-shutdown history or event context. The reasons are structural and worth stating plainly:

  • Too few resolved events. Only 74 incidents have an observed end, spread across 65 countries — roughly one resolved shutdown per country. There is not enough signal to learn a duration law.
  • The “end” label is a curation artifact. status = 'confirmed' tracks how long ago the incident was filed, not when blocking stopped. Until end dates come from genuine measurement (the day probes stop seeing the block), the survival target is contaminated.
  • Durations are quantized. Observed durations cluster on 24-hour multiples — they are last_seen − first_seen rounded to ingestion cadence, not precise shutdown lengths.

This does not mean the question is unanswerable forever — it means it is not answerable with today’s data, and claiming a 0.728 c-index implied otherwise. The model artifact and its sidecar metrics have been corrected to carry the honest forward-temporal c-index (~0.50–0.56), passed_promote_floor: false, and an explicit honest_caveats block; the /v1/forecast/duration endpoint now surfaces the audit verdict in every response. Publishing the honest negative — and not shipping a number we cannot stand behind — is the fix.

The source record.

Original source links and labels. Linked responses may have changed since publication.