Every model on Voidly Atlas already publishes its training-time test metrics — the AUC and F1 numbers a model earns on the held-out 15% split during training. But journalists asking "is this thing actually accurate in the wild?" deserve a different number: the empirical precision and recall on predictions the model issued in production, joined against observed outcomes.
Shipped today (2026-05-21): GET /v1/atlas/prediction-track-record
— a rolling 30-day production track record across all 12 forecast
models. Sidecar JSON regenerated nightly at 04:00 UTC by joining
daily-logged predictions against observed incidents.
Only the legacy forecast_7day (v1) model has full
production logging today — daily forecasts persisted to
sentinel_forecasts, outcomes resolved into
sentinel_outcomes after the 7-day horizon elapses. On
720 such predictions across the last 30 days, at the operational
threshold of 0.05 literally emitted by the model at request time:
Numbers worth staring at. The model is massively under-predicting in production — it says 5% on average, reality came in at 61%. That gap is the calibration drift the new Adaptive Conformal Inference (ACI) layer is actively chasing, with α already drifted from 0.10 to 0.21 over 840 observations.
Two factors compound the gap: (1) the 2026-05-21 retrain dropped IODA disruption labels from the target — but the outcome joiner in this window still flags an "observed positive" for any non-disruption incident in the 7-day horizon, which is a broader signal than the retrained target distribution; (2) the operational threshold is the conservative 0.05 floor for casting a wide alert net, which trades precision for recall.
Multi-horizon (1d/7d/30d), shutdown duration (RSF), trajectory (1d/7d/30d), contagion-chain (7d), and hourly (6h/12h/24h) all compute predictions on demand — they don't yet persist per-request predictions to a forecasts table. So we honestly cannot publish empirical precision/recall for them yet.
Rather than silently re-show their training-time test metrics and
call them "live", the endpoint marks every such row
kind: "training-time-only", sets
n_predictions: 0, and includes the
training_time_metrics as a separate, clearly-labelled
block. The caveat says it directly: "Production prediction logging
is not yet wired for this model; this row reflects held-out test
metrics, NOT live precision/recall."
The gap between training-time AUC and live prod precision is one of the most consistently mis-cited numbers in ML coverage. The forecast_7day model has training-time ROC AUC 0.954 and a stratified F1 of 0.667. Its real prod precision is 0.69 and prod recall is 0.39 on 720 issued predictions. Both numbers are honest. Citing the first one as "accuracy" without the second one would be misleading.
Live every day at /v1/atlas/prediction-track-record,
refresh metadata at /v1/atlas/prediction-track-record/info.
As production logging lands for the multi-horizon and trajectory
models, those entries will fill in with real empirical numbers
— the schema is forward-compatible.
scripts/build-prediction-track-record.py — joins
sentinel.db.sentinel_forecasts against
sentinel_outcomes in the rolling 30-day window, writes
sidecar JSON at /opt/voidly-ai/ml-deploy/prediction_track_record.json.
scripts/patch-prediction-track-record-endpoint.py —
idempotent patch adds GET /v1/atlas/prediction-track-record
and /info to forecast_api.py.
0 4 * * * on Vultr regenerates the sidecar every
24h. Endpoint reads the file (cheap).