Historical May 2026 split audit. These split figures are not current rolling accuracy. Empirical calibration and interval coverage remain unvalidated until a fresh eligible outcome cohort is available.
Our shutdown-forecast model, Voidly Sentinel, was previously advertised at 99.8% F1 / 1.000 AUC. Those numbers come from a stratified random holdout — and they're inflated by a known statistical issue called within-country temporal leakage: events from the same country end up in both train and test, so the model effectively memorizes country-specific patterns instead of generalizing.
Our own /v1/sentinel/accuracy endpoint flagged this in
a self-published warning months ago: "Stratified AUC overstates
real-world performance by 47.9pp vs. time-based split. Do not cite
the stratified number as a deployment figure." But the website
kept citing 99.8%.
On May 20, 2026 we fixed the contradiction. The /methodology page now publishes all three splits, lead with the honest LOCO median (0.91 AUC, 0.55 F1), and keeps the stratified number visible with a "do not cite" tag.
The three splits, explained
- Stratified random 15% holdout: AUC 0.98, F1 0.79. The classical ML baseline. Inflated for this problem because it allows within-country temporal leakage. Don't cite.
- Time-based split (train pre-T, test post-T): AUC 0.50, F1 0.00. The honest deployment baseline. The model is random on novel events because it hasn't seen enough new-pattern data yet. This is the floor.
- Leave-country-out (LOCO) median: AUC 0.91, F1 0.55. For each country, train on the other 18, test on the held-out one. Take the median across 19 holdouts. This is the honest generalization figure — and what we cite for the model now. Individual countries range from F1 0.19 (Cuba) to 0.84 (Indonesia).
Why this matters
Forecast models in this space get used to brief journalists, threat-intel teams, and on-call infrastructure operators. Citing an inflated F1 leads them to over-trust the model. Citing the honest LOCO number lets them set realistic confidence intervals.
The May calibration page included a rolling snapshot and interval coverage claims. Those values are historical and unvalidated under the current outcome eligibility rules. Do not cite them as present calibration or as proof that a 90% interval is available.
How to cite our model
If you're citing Voidly Sentinel in a paper, news story, or threat-intel report, use one of these depending on your tolerance:
- For headline academic claims → LOCO median (0.91 AUC, 0.55 F1)
- For deployment-tolerance claims → time-based (0.50 AUC, 0.00 F1) as the floor
- For "the model agrees with our existing labels" → stratified 0.98 with the inflation caveat
The historical split methodology and per-country breakdown can be
inspected at /v1/sentinel/accuracy; use the current
endpoint status before citing any present metric.