It is easy to claim "F1 0.87" for a censorship-forecast model. It is much harder to answer the next question honestly: is the model adding value, or is it just memorizing the prior? An F1 of 0.87 on a target with an 80% positive rate is barely beating "always predict positive". On a target with a 5% positive rate, that same F1 is genuine signal. Without a baseline, the headline number is uninterpretable.

This finding ships the synthetic-baseline benchmark suite: every Voidly Atlas ML model evaluated against six trivial baselines on the same data, with the lift surfaced inline as a single number per row. Live at GET /v1/atlas/baseline-benchmark; weekly cron Thursday 04:45 UTC.

The six baselines

• always_zero — predict 0 for everything. F1 = 0 by definition.
• always_one — predict 1 for everything. Recall = 1, precision = base rate, F1 = 2·p/(1+p).
• base_rate_constant — predict every row's probability as the training-set positive rate. Well-calibrated by definition, but provides no resolution.
• predict_yesterday — lag-1 of the label within each country group. For autocorrelated targets like "will country X be in shutdown next week", this is a brutal baseline: if the country was in shutdown yesterday, it almost certainly will be tomorrow.
• country_base_rate — predict each country's historical positive rate. No temporal signal at all.
• random_with_base_rate — Bernoulli sample with p = base rate. The sanity floor.

Lift is reported in percentage points of F1 (or AUC, for AUC-only models) against the predict_yesterday baseline — the toughest one of the six.

The honest headline numbers

• 23 models benchmarked. Forecast (7-day), forecast multi-horizon (1/7/30d), classifier v3.3, trajectory v1 (1/7/30d), contagion-chain 7d, duration RSF, 5 sampled per-platform, 5 sampled per-domain, 4 per-method.
• 7 models flagged barely_beats_baseline=true — lift of less than 5pp vs predict_yesterday. That list includes: forecast_7day (F1 lift +2.29pp), forecast_1d/7d/30d (multi-horizon, all F1-negative), classifier_v3.3 (F1 lift −21.04pp), and trajectory_d7 / d30 (AUC lift −21 to −23pp).
• Best by AUC lift: per_domain_pornhub.com at +50.74pp. Other per-domain and per-platform models cluster near 1.000 AUC. This is a code smell, not a triumph: AUC at 0.999+ usually means the model is reconstructing the labeling rule from a leaked feature. Every such row gets an honest_caveats entry inline.
• Worst by AUC lift: trajectory_d30 at −22.78pp. Trajectory models lose to predict_yesterday on long horizons because, once a country is in shutdown today, it usually stays that way for the next 30 days. Persistence beats trajectory at that horizon.

Why forecast_7day only beats predict_yesterday by 2.29pp F1

This is the point of the whole exercise. forecast_7day has a headline F1 of 0.937 on the temporal holdout. Looks great. But predict_yesterday — literally just "the country's label yesterday" — scores F1 0.914 on the same holdout. The model adds 2.29 percentage points of F1 over a one-line baseline. On AUC the lift is 4.51pp, which is more meaningful (AUC is threshold-free).

This does not mean the model is useless. The model still adds: per-country calibration, 90% conformal intervals, SHAP attributions, and it correctly forecasts events 7 days out for the country-day pairs that weren't in shutdown the day before. But the headline F1 alone overstates its lift. That's what this endpoint exists to expose.

Why classifier_v3.3 shows a negative F1 lift

The classifier reports an F1 of 0.742 from 5-fold stratified CV. The predict_yesterday baseline on the rolling holdout scores 0.953. That's an F1 lift of −21pp.

Honest reading: the classifier is solving a different task. It scores country-days that already crossed an evidence threshold — mostly positive examples — whereas predict_yesterday is trivially perfect on day-to-day continuity within a country. Comparing the two as F1 is approximate, not apples-to-apples. The per-row source field documents this; readers who want a fair comparison should look at the classifier's stratified CV against a CV baseline computed on the same folds. The benchmark surfaces the ugly-but-honest number rather than hiding it.

What this is NOT

• Not a full LOCO re-evaluation of every model. Some rows pull training-time sidecar metrics and compute the baselines on a separate last-60d temporal holdout. Lift is approximate in those cases — check the per-row source field.
• Not a verdict on the models. barely_beats_baseline=true is a yellow flag, not red. The point of this endpoint is to make every shipped model's value claim auditable in one query.
• Not a static benchmark. The cron regenerates the sidecar every Thursday at 04:45 UTC; the numbers move as we ship new model versions and as the holdout window slides forward.

How to read the endpoint

Every row in the response has the same shape:

• model_id, family, horizon
• model_metrics: F1, AUC, Brier, n, pos_rate on this evaluation
• baseline_metrics: the six baselines' F1/AUC/Brier on the same data
• lift_vs_predict_yesterday: signed pp delta for F1, AUC, Brier (Brier is sign-flipped so positive = better)
• barely_beats_baseline: boolean flag
• barely_beats_metric: which metric the flag is on (F1 or AUC)
• source: live re-eval vs. training-time sidecar
• honest_caveats: model-specific gotchas, including the "AUC at 0.999+ is a code smell" flag

Query filters: family=<name>, barely=true|false, min_lift_pp=<float>. The /info endpoint exposes the methodology and the per-row caveat policy without hitting the full payload.

Methodology footnote: why predict_yesterday is the right toughest baseline

Internet shutdowns are extremely autocorrelated: if a country was in shutdown yesterday, the prior on shutdown today is over 95%. On those targets, “persistence” is the hardest one-line baseline to beat. We pick it deliberately. If a model can't beat predict_yesterday at some metric by at least 5pp, we want it on a list the user can see — not buried.

This pattern is borrowed from the time-series forecasting literature (Hyndman & Athanasopoulos, Forecasting: Principles and Practice) and from the GluonTS benchmarking conventions. It is also consistent with how Cloudflare and NetBlocks report their own shutdown detection — usually as deltas from a 7-day baseline rather than as absolute counts.