Voidly
All findingsPublished research

How we audited our own shutdown-forecast model and published the embarrassing numbers

The May 2026 audit compared stratified, time-based, and leave-country-out splits and warned against citing the inflated stratified result. Its historical split figures do not establish current rolling accuracy; calibration and interval coverage require a newly eligible outcome cohort.

Historical May 2026 split audit. These split figures are not current rolling accuracy. Empirical calibration and interval coverage remain unvalidated until a fresh eligible outcome cohort is available.

Our shutdown-forecast model, Voidly Sentinel, was previously advertised at 99.8% F1 / 1.000 AUC. Those numbers come from a stratified random holdout — and they're inflated by a known statistical issue called within-country temporal leakage: events from the same country end up in both train and test, so the model effectively memorizes country-specific patterns instead of generalizing.

Our own /v1/sentinel/accuracy endpoint flagged this in a self-published warning months ago: "Stratified AUC overstates real-world performance by 47.9pp vs. time-based split. Do not cite the stratified number as a deployment figure." But the website kept citing 99.8%.

On May 20, 2026 we fixed the contradiction. The /methodology page now publishes all three splits, lead with the honest LOCO median (0.91 AUC, 0.55 F1), and keeps the stratified number visible with a "do not cite" tag.

The three splits, explained

  • Stratified random 15% holdout: AUC 0.98, F1 0.79. The classical ML baseline. Inflated for this problem because it allows within-country temporal leakage. Don't cite.
  • Time-based split (train pre-T, test post-T): AUC 0.50, F1 0.00. The honest deployment baseline. The model is random on novel events because it hasn't seen enough new-pattern data yet. This is the floor.
  • Leave-country-out (LOCO) median: AUC 0.91, F1 0.55. For each country, train on the other 18, test on the held-out one. Take the median across 19 holdouts. This is the honest generalization figure — and what we cite for the model now. Individual countries range from F1 0.19 (Cuba) to 0.84 (Indonesia).

Why this matters

Forecast models in this space get used to brief journalists, threat-intel teams, and on-call infrastructure operators. Citing an inflated F1 leads them to over-trust the model. Citing the honest LOCO number lets them set realistic confidence intervals.

The May calibration page included a rolling snapshot and interval coverage claims. Those values are historical and unvalidated under the current outcome eligibility rules. Do not cite them as present calibration or as proof that a 90% interval is available.

How to cite our model

If you're citing Voidly Sentinel in a paper, news story, or threat-intel report, use one of these depending on your tolerance:

  • For headline academic claims → LOCO median (0.91 AUC, 0.55 F1)
  • For deployment-tolerance claims → time-based (0.50 AUC, 0.00 F1) as the floor
  • For "the model agrees with our existing labels" → stratified 0.98 with the inflation caveat

The historical split methodology and per-country breakdown can be inspected at /v1/sentinel/accuracy; use the current endpoint status before citing any present metric.

The source record.

Original source links and labels. Linked responses may have changed since publication.