Model honesty, public
Every Sentinel shutdown forecast ships with a 90% conformal interval. This page tracks how often the real outcome lands inside that interval — the closer to 90%, the more honest the model. Data lives at /v1/sentinel/calibration/history and updates every 24h.
/v1/forecast/{cc}/7day response under aci_alpha + aci.* fields. Full ACI methodology →Live forecast accuracy (prod_rolling, 30-day window)
Brier < 0.10 is good, > 0.30 is concerning. Calibration MAE < 0.05 means predicted-probabilities track observed-rates closely. See /sentinel/backtest for the actual reliability diagram (predicted-mean vs observed-rate scatter) and /methodology#validation for the full evaluation methodology + 3-split honest baselines.
Empirical coverage — 90-day rolling
The blue line is the actual fraction of forecasts where the real outcome landed inside the 90% conformal interval. The green dashed line is the nominal target (0.90). If the blue stays close to the green, the model is well calibrated.
Blue: empirical coverage · Dashed green: nominal 0.90 target · Orange circles: drift alerts
Last 14 days
| Date | Coverage | q90 | n holdout | Drift? |
|---|---|---|---|---|
| Aug 19 | 91.5% | 0.171 | 2,047 | — |
| Aug 18 | 91.5% | 0.171 | 2,047 | ⚠️ |
| Aug 17 | 91.5% | 0.171 | 2,047 | ⚠️ |
| Aug 16 | 91.5% | 0.171 | 2,047 | ⚠️ |
| Aug 15 | 90.7% | 0.130 | 2,025 | — |
| Aug 14 | 90.7% | 0.130 | 2,025 | — |
| Aug 13 | 90.7% | 0.130 | 2,025 | — |
| Aug 12 | 90.7% | 0.130 | 2,025 | — |
| Aug 11 | 90.7% | 0.130 | 2,025 | — |
| Aug 10 | 90.7% | 0.130 | 2,025 | — |
| Aug 9 | 90.7% | 0.130 | 2,025 | — |
| Aug 8 | 91.8% | 0.125 | 2,003 | — |
| Aug 7 | 91.8% | 0.125 | 2,003 | — |
| Aug 6 | 91.8% | 0.125 | 2,003 | — |
What features the model actually uses
Sklearn feature_importances_ on the underlying XGBoost. 39 features total. Top-3 sum: 0.329 · Top-5: 0.416 · Top-10: 0.583. Healthy distribution — no single feature dominates the model.
- 1.block_rate_roll30_mean17.2%
- 2.recent_shutdown10.3%
- 3.month5.4%
- 4.week_of_year4.5%
- 5.critical_incident_7d4.2%
- 6.high_importance_event4.0%
- 7.election_in_7days3.9%
- 8.high_urgency_signals_7d3.4%
- 9.block_rate_roll14_mean3.0%
- 10.gdelt_protests_7d2.4%
- 11.gdelt_conflict_tone_7d2.4%
- 12.gdelt_unrest_30d2.4%
Interpretation: The forecast model's top feature is gdelt_unrest_30d (0.25) — protest + conflict signals from the GDELT 1.0 global news feed. recent_shutdown, block_rate rolling means, and incident counts follow. risk_tier — the leaky country-level encoding that dominated our older classifier at 85% — contributes only ~2% here. Healthy distribution; no single feature dominates.
Raw JSON: /v1/sentinel/feature-importance
Related
- /methodology#validation — 3-split honest baselines (LOCO 0.91 AUC vs stratified 0.98)
- /v1/sentinel/accuracy — full evaluation JSON, updated nightly
- /atlas/elections — see the forecast in action (90-day upcoming elections)