Every model on Voidly Atlas already publishes its training-time test metrics — the AUC and F1 numbers a model earns on the held-out 15% split during training. But journalists asking "is this thing actually accurate in the wild?" deserve a different number: the empirical precision and recall on predictions the model issued in production, joined against observed outcomes.

Shipped today (2026-05-21): GET /v1/atlas/prediction-track-record — a rolling 30-day production track record across all 12 forecast models. Sidecar JSON regenerated nightly at 04:00 UTC by joining daily-logged predictions against observed incidents.

The honest headline (last 30 days)

Only the legacy forecast_7day (v1) model has full production logging today — daily forecasts persisted to sentinel_forecasts, outcomes resolved into sentinel_outcomes after the 7-day horizon elapses. On 720 such predictions across the last 30 days, at the operational threshold of 0.05 literally emitted by the model at request time:

Numbers worth staring at. The model is massively under-predicting in production — it says 5% on average, reality came in at 61%. That gap is the calibration drift the new Adaptive Conformal Inference (ACI) layer is actively chasing, with α already drifted from 0.10 to 0.21 over 840 observations.

Two factors compound the gap: (1) the 2026-05-21 retrain dropped IODA disruption labels from the target — but the outcome joiner in this window still flags an "observed positive" for any non-disruption incident in the 7-day horizon, which is a broader signal than the retrained target distribution; (2) the operational threshold is the conservative 0.05 floor for casting a wide alert net, which trades precision for recall.

What about the other 11 models?

Multi-horizon (1d/7d/30d), shutdown duration (RSF), trajectory (1d/7d/30d), contagion-chain (7d), and hourly (6h/12h/24h) all compute predictions on demand — they don't yet persist per-request predictions to a forecasts table. So we honestly cannot publish empirical precision/recall for them yet.

Rather than silently re-show their training-time test metrics and call them "live", the endpoint marks every such row kind: "training-time-only", sets n_predictions: 0, and includes the training_time_metrics as a separate, clearly-labelled block. The caveat says it directly: "Production prediction logging is not yet wired for this model; this row reflects held-out test metrics, NOT live precision/recall."

Why this matters

The gap between training-time AUC and live prod precision is one of the most consistently mis-cited numbers in ML coverage. The forecast_7day model has training-time ROC AUC 0.954 and a stratified F1 of 0.667. Its real prod precision is 0.69 and prod recall is 0.39 on 720 issued predictions. Both numbers are honest. Citing the first one as "accuracy" without the second one would be misleading.

Live every day at /v1/atlas/prediction-track-record, refresh metadata at /v1/atlas/prediction-track-record/info. As production logging lands for the multi-horizon and trajectory models, those entries will fill in with real empirical numbers — the schema is forward-compatible.

The wiring