The drift

On 2026-05-21 we shipped a new endpoint, /v1/atlas/prediction-track-record, which joins every daily-logged forecast against the observed outcome 7 days later. The first numbers it surfaced for forecast_7day were bad:

• mean predicted probability: 4.9 percent
• empirical positive rate: 61.4 percent
• calibration drift: +56.45 percentage points
• precision 0.69, recall 0.39, Brier 0.556

A model that says "5 percent chance" on average while the world delivers a 61 percent positive rate is not a model anyone should be acting on.

Two bugs, not one

The drift was the visible symptom of two distinct upstream problems.

Bug 1 — stale calibrator. The deployed isotonic calibrator on top of the XGBoost model was fit when forecast labels still counted IODA disruption rows as confirmed censorship. After we cleaned the label rule on 2026-05-21 (see the disruption-fix finding), the model's raw probabilities became approximately correct, but the old isotonic mapping kept squashing them back down.

Bug 2 — disruption-inflated outcome labels. The sentinel-outcome-joiner cron marks observed=1 if ANY incident lands in the 7-day horizon — including IODA disruption rows (fiber cuts, BGP, weather, DDoS). Most "true positives" in the 30-day window were disruptions, not censorship. Both the calibrator's training target and the track-record's evaluation labels were inflated.

The fix

The new script scripts/refit-forecast-calibration.py does the following on the last 30 days of sentinel_outcomes:

1. Load (raw probability, observed) pairs.
2. Rebuild the observed label by joining voidly_data.incidents filtered to incident_type IN ('censorship','mixed'). Disruption rows are kept in the incidents table for the country pages but excluded as positives here, matching the forecast model's post-2026-05-21 labelling rule.
3. Fit sklearn.isotonic.IsotonicRegression with out_of_bounds="clip", y_min=0, y_max=1.
4. Compute Brier and ECE on the same pairs before and after.
5. Promote only if both gates pass: new Brier no worse by more than 0.05 AND new ECE strictly tighter.
6. Backup the old calibrator with a timestamped .bak suffix; write the new one to both /opt/voidly-ai/models/ and /opt/voidly-ai/ml-deploy/.
7. Materialize a 0.01-step lookup table sidecar (.lookup.json) as a defensive fallback in case the sklearn artifact can't be loaded.

Numbers, honestly

On 720 (raw probability, censorship-only observed) pairs covering 2026-04-21 through 2026-05-14:

The honest read: the raw XGBoost model was already close to well-calibrated against clean labels (9pp drift, Brier 0.13). The OLD isotonic mapping actively made things worse against the same labels (-50pp drift, Brier 0.38) because it had been trained to chase a disruption-inflated target. The NEW mapping drives drift to zero on the fit window, with a Brier of 0.12 — better than the uncalibrated baseline.

Track record after refit

We also updated build-prediction-track-record.py to apply the live calibrator to the historical raw probabilities and to use censorship-only labels for the headline metric. This makes the published track record reflect what production WOULD emit, not the legacy uncalibrated logs. The raw_uncalibrated_raw_labels_view is still surfaced in the JSON for full transparency, with a note that it conflates the two bugs.

Headline calibration drift dropped from +56.45pp to 0.00pp, well under the <15pp success threshold.

Honest caveats

The 30-day refit window is narrow. A single shock during the window can dominate the fit; broader windows would dilute recent regime changes. We chose narrow on purpose — the goal was to react to the post-disruption-fix regime, not average over the broken-label regime.

We did not touch the underlying XGBoost ranker. If the ranker itself drifts (feature distributions shift, ASN topology changes), this refit will not catch that — a full retrain script is in scripts/train-forecast.py for that case.

On the fit window the new calibrator is close to a flat constant (raw probs in [0.019, 0.197] all map to ~0.14 — the empirical base rate). This is mathematically the optimal isotonic fit when the raw model output has very low resolution in the observed positive region. It also makes precision_at_threshold degenerate (everything above 0 is above the operational threshold). When the underlying model is retrained on more discriminative features we expect the calibrator's mapping to recover its slope.

The previous calibrator is preserved at forecast_calibrator_v2_isotonic_prod.pkl.bak.20260521_154014 for rollback. The OLD model bundle (XGBoost + scaler) is unchanged.