Voidly
All findingsPublished research

Data-driven country risk tiers: re-deriving a hand-set model input from objective signals

Every country in Voidly Atlas carries a risk tier — an integer 1-5 in country_geography.risk_tier, tier 1 = highest censorship risk, tier 5 = lowest. That tier is not a measurement: an analyst hand-set it. And it is not inert — the v3.3 censorship classifier consumes it as an input feature, the…

Read the full published summary

Every country in Voidly Atlas carries a risk tier — an integer 1-5 in country_geography.risk_tier, tier 1 = highest censorship risk, tier 5 = lowest. That tier is not a measurement: an analyst hand-set it. And it is not inert — the v3.3 censorship classifier consumes it as an input feature, the 7-day shutdown forecast consumes it, and several downstream surfaces lean on it. A stale or wrong hand-set tier is silently baked into every model that reads it. This finding ships a data-driven alternative: a tier derived from six objective signals over the trailing 365 days — confirmed censorship incident count (disruption rows excluded, same rule the forecast labeler uses), mean block rate, distinct domains blocked, distinct blocking methods, mean 7-day forecast risk, and DBSCAN anomaly frequency. Each feature is rank-transformed then z-scored; the composite is their equal-weight mean; countries are clustered with KMeans(k=5) on the composite (highest-mean cluster relabeled tier 1), with an equal-frequency quantile binning reported alongside as a cross-check. The first cut produced nonsense — the US scored 60% blocked, Sweden 100% — and three data-quality fixes each corrected a real bias: IODA alerts excluded (they are ASN connectivity outages, not censorship), only signal_level=critical counts as blocked (elevated/warning are soft early-warning levels that community probes self-report — one mis-networked UK probe had flagged DuckDuckGo and WhatsApp as blocked), and the block rate is empirical-Bayes-shrunk toward the global pooled rate (0.328) so a 29-row country like Sweden is pulled to the prior while China, with thousands of rows, barely moves. Result: of 148 countries with a hand-set tier, 107 land in a different data-derived tier and only 41 agree — the data says higher risk for 41 and lower risk for 66, with 50 large (>=2 tier) disagreements. The hand-set column parks 87 countries in a tier-3 catch-all; the data does not. The five biggest disagreements are exactly the cases that prove why this stays a PROPOSAL: Eritrea, North Korea and Turkmenistan are hand-set tier 1 (the most repressive information environments on earth) but the data drops them to tier 4-5 — not because the data is right but because it is blind, there are almost no OONI probes inside those countries so total censorship and zero measurement look identical to a feature pipeline. The UK and Canada move the other way (hand-set tier 5, data tier 2-3) because they are heavily probed and accumulate DBSCAN anomaly frequency plus minor genuine blocking. PROPOSAL ONLY — this never writes the country_geography table; the hand-set tier stays authoritative until a human reviews the diff. The artifact makes a hand-set number auditable and hands a reviewer a ranked list of exactly which country tiers to re-examine first. Live at GET /v1/atlas/risk-tiers + /v1/atlas/risk-tiers/{cc}. Built by scripts/build-data-driven-risk-tiers.py.

Every country in Voidly Atlas carries a risk tier, an integer 1–5 stored in country_geography.risk_tier (tier 1 = highest censorship risk, tier 5 = lowest). That tier is not a measurement. It was hand-set — an analyst typed a number. And it is not inert: the v3.3 censorship classifier consumes it as an input feature, the 7-day shutdown forecast consumes it, and several downstream surfaces lean on it. If the hand-set tier is stale or simply wrong, that error is silently baked into every model that reads it.

This finding ships a data-driven alternative: a risk tier derived from objective signals over the trailing 365 days, computed so a human can see exactly where the hand-set tier and the data disagree. It is a proposal surface only. It does not touch the country_geography table. The hand-set tier stays authoritative until a human reviews the diff.

The six features

For each country, over the last 365 days:

FeatureWhat it measures
confirmed_censorship_incident_countIncidents typed censorship or mixed. Pure disruption rows excluded — same rule the forecast labeler uses.
mean_block_rateEmpirical-Bayes-shrunk fraction of non-IODA evidence at signal_level=critical.
n_distinct_domains_blockedDistinct domains observed blocked at critical level.
n_blocking_methods_observedDistinct blocking mechanisms (DNS poison, TCP reset, blockpage…).
forecast_mean_riskMean 7-day shutdown risk from the live forecast model.
dbscan_anomaly_frequencyFraction of evidence days a CenDTect-style per-country DBSCAN flags as shape-anomalous.

Each feature is rank-transformed to [0,1] then z-scored, and the composite is the equal-weight mean of the six z-scores. Countries are then clustered with KMeans(k=5) on the composite — clusters relabeled so the highest-mean cluster is tier 1. K-means is the headline binning; an equal-frequency quantile split is reported alongside as a cross-check.

Three data-quality fixes that mattered

The first cut produced nonsense — the United States scored “60% blocked,” Sweden “100% blocked.” Three corrections, each one a real bias:

  • IODA excluded. IODA alerts are ASN-level connectivity outages, not censorship. Their critical rows were inflating the block rate of well-monitored countries. (This is the same noise the forecast labeler strips.)
  • Only critical counts as blocked. elevated / warning are soft early-warning levels. Community probes self-report elevated, and one mis-networked probe in the UK flagged DuckDuckGo, WhatsApp and the Washington Post as “blocked.” critical comes only from the rigorous automated sources (OONI, CensoredPlanet).
  • Block rate is empirical-Bayes-shrunk. Sweden had 29 non-IODA rows, all 29 flagged — a raw rate of 1.0 off a handful of probes. The rate is now shrunk toward the global pooled rate (0.328) with a 200-pseudo-count, so a sparse country is pulled to the prior while China, with thousands of rows, barely moves.

Result: 107 of 148 countries disagree

Of the 148 countries with a hand-set tier, 107 land in a different data-derived tier and only 41 agree. The data says higher risk than the hand-set tier for 41 countries and lower risk for 66. Fifty disagreements are large (≥2 tiers).

TierHand-set countData-derived (k-means) count
1 (highest risk)611
21726
38729
41643
5 (lowest risk)22108

The hand-set column parks 87 countries in a tier-3 catch-all. The data does not: most of those countries have little observed censorship and the data pushes them down to tier 4–5.

The five biggest disagreements

CountryHand-setData-derivedΔ
ER Eritrea15+4
KP North Korea15+4
TM Turkmenistan14+3
GB United Kingdom52−3
CA Canada53−2

These five are exactly the cases that prove why this stays a proposal, not an auto-apply.

Eritrea, North Korea, Turkmenistan are hand-set tier 1 — the most repressive information environments on earth. The data drops them to tier 4–5. That is not the data being right. It is the data being blind: there are almost no OONI probes inside those countries, so the trailing 365-day window shows near-zero blocked domains, near-zero incidents, a flat block rate. Total censorship and zero measurement look identical to a feature pipeline. The hand-set tier encodes political reality the data cannot see — this is the headline reason the hand-set tier stays authoritative.

The UK and Canada move the other way: hand-set tier 5, data-derived tier 2–3. They are heavily probed, so they accumulate a non-trivial DBSCAN anomaly frequency and a handful of genuine but minor blocking events. The data is not claiming the UK is China; it is saying that, on recent measured signal alone, a well-instrumented democracy with any blocking at all out-ranks an unmeasured country. Whether that is a tier-2 country or a tier-5 country with measurement noise is precisely the call a human should make.

Honest caveats

  • Proposal only. The hand-set country_geography.risk_tier is not modified. Nothing in v3.3 or the forecast changes. The data-derived tier is advisory.
  • Recent vs. historical. The data tier reflects the trailing 365 days. The hand-set tier may encode a decade of shutdowns the recent window does not show — the ER / KP / TM cases above.
  • Boundaries are arbitrary. A country one cluster-edge either side of a boundary moves a whole tier. K-means and quantile tiers are both reported so single-tier disagreements read as soft.
  • Probe coverage is uneven. China / Russia / Iran have thousands of non-IODA rows; most countries have under 100. DBSCAN frequency and domain counts are noisier for sparse countries even after block-rate shrinkage.
  • forecast_mean_risk is partial. The live forecast model only carries recent feature rows for ~21 watched countries; the rest are mean-imputed, so that feature is weaker elsewhere.
  • Correlated features. Block rate, blocked domains and incident count move together, so the equal-weight composite is not an information-theoretic average.

Live at

GET /v1/atlas/risk-tiers — full comparison table
GET /v1/atlas/risk-tiers?disagreeing_only=1&min_magnitude=2 — just the big disagreements
GET /v1/atlas/risk-tiers/CN — per-country detail with features + z-scores

Built by scripts/build-data-driven-risk-tiers.py. Sidecar at /opt/voidly-ai/ml-deploy/data_driven_risk_tiers_v1.json. This is a transparency artifact: it makes a hand-set number auditable, and it gives a human a ranked list of exactly which 148 country tiers to re-examine first.

The source record.

Original source links and labels. Linked responses may have changed since publication.