GradientBoostingClassifier
794 of 1,116 positive training labels (71.1%) are country-days whose only incident is an IODA `disruption` row — network outages, not confirmed censorship. The same class was excluded from forecast labels in 2026-05 as ~94% noise. So this model is substantially trained to detect DISRUPTION, and its F1 should be read as such. Corpus is frozen at 2026-05-21; relabelling is a pending decision, not an oversight.
- Model trained
- 2026-05-21T03:01:46.793987+00:00
- Record generated
- Not supplied
- Request checked
Cross-country F1 is not a per-prediction probability or a future-time accuracy estimate. Loaded introspection does not establish which model every prediction route serves.
Evaluation population, distribution & source caveats
- Source headline
- LOCO mean F1 across all countries
- Training rows
- 4,237
- Positive training rows
- 1,116
- Training countries
- 131
- LOCO median F1 · distribution caveat
- 0.870
- Countries with perfect F1
- 46
- Stratified F1 · different split
- 0.729
- Stratified AUC · different metric
- 0.899
The headline LOCO MEDIAN F1 (0.870) is dominated by many small-sample countries scoring a perfect 1.0 on a handful of days; the MEAN F1 (0.711) is the honest single number. Censorship-heavy, high-volume countries score materially lower (CN ~0.29 on n=95, BY ~0.21 on n=84, AZ ~0.11 on n=65). Cite the mean — or the specific per-country number — for hard countries, not the median. The model is CLEAN (no label leakage); this is a distribution caveat, not an accuracy retraction.
HONESTY: forward-temporal holdout (train past, test future) gives AUC 0.669 / F1 0.474 vs the random-split AUC 0.895 / F1 0.725. v3.3 generalizes across COUNTRIES (LOCO F1 0.87) but DEGRADES across TIME (delta AUC -0.226) — which is why it is retrained weekly. Do not read the random-split number as forward-deployment accuracy. See /atlas/findings/classifier-v3.3-temporal-generalization-2026-05.
confidence is
stratified cross-validation F1, NOT a calibrated per-prediction probability
correct use
Cross-country censorship-risk classification. Per-country accuracy varies widely — 16 MENA / former-Soviet countries (OM, UZ, TN, LY, YE, JO, MA, …) regress 5-29pp on sparse neighbor-pair overlap. See evaluation.honest_evaluation + evaluation.per_country for the full distribution.
headline metric
LOCO median F1 0.87 (cross-country); honest MEAN F1 0.71; well-sampled (n>=30 countries) mean ~0.63
model status
CLEAN — no label leakage (ML_LEAKAGE_AUDIT.md); this caveat is about the score distribution, not leakage.
training label composition
794 of 1,116 positive training labels (71.1%) are country-days whose only incident is an IODA `disruption` row — network outages, not confirmed censorship. The same class was excluded from forecast labels in 2026-05 as ~94% noise. So this model is substantially trained to detect DISRUPTION, and its F1 should be read as such. Corpus is frozen at 2026-05-21; relabelling is a pending decision, not an oversight.
Different historical random-split runs report different values. Keep the source version and split with each number; do not merge them into one evaluation.
Temporal-generalization study ↗