Cross-source agreement

Methodology

The four sources

• OONI — active web-connectivity, Signal, WhatsApp, Telegram, Tor, and middlebox-detection probes run by volunteers worldwide.
• IODA — BGP route observations + Trinocular active probing aggregated by Georgia Tech, surfacing AS-level outages.
• CensoredPlanet — University of Michigan's Satellite (DNS) and Hyperquack (HTTP/HTTPS) global scans from outside the target country.
• Voidly probes — our 40-node active probe network (Vultr + Fly.io + community probes) running 62-domain reachability checks every 5 minutes.

The unit: country-day

For each (country, date), we ask: did source s emit at least one row with signal_level in (elevated, warning, critical)? If yes, source s “fired.” This gives a 4-bit indicator per country-day: (ooni, ioda, censoredplanet, voidly_probes).

Cohen's kappa

For each pair of sources (a, b), kappa measures chance-adjusted agreement on the binary fire indicator:

Po = observed agreement   = (n_11 + n_00) / N
Pe = chance agreement     = P(a=1)*P(b=1) + P(a=0)*P(b=0)
κ  = (Po - Pe) / (1 - Pe)

Kappa is computed over country-days where at least one source fired — days where no source observed anything anomalous are excluded so that trivial “both quiet” agreement across 200+ countries doesn't artificially inflate the metric.

The ≥2-sources rate

For the journalist-facing “is this corroborated?” answer, we compute the simpler metric: of all anomalous country-days, what fraction had ≥2 sources firing? When restricted to country-days with a confirmed censorship incident (type in ('censorship', 'mixed'); IODA-only disruptions excluded), this rises to 50.7%.

Honest caveats

• Source-presence is binary at the country-day level (one row -> present), so a single elevated OONI test counts the same as a dozen IODA outage rows. This avoids letting one noisy source dominate the agreement rate.
• Voidly and Voidly-Community are merged into a single voidly_probes bucket -- their coverage overlaps geographically.
• Cohen's kappa is computed over country-days WHERE AT LEAST ONE SOURCE FIRED. Days where no source fired are excluded so kappa isn't inflated by trivial 'both quiet' agreement across 200+ countries.
• Most pairwise kappas are LOW (0.05-0.20). This is the honest finding -- the four sources observe DIFFERENT layers of the internet (active probing vs BGP vs DNS scans vs reachability checks) and rarely happen to fire on the same country-day. Low kappa here doesn't mean the sources are wrong; it means they're complementary, which is exactly why multi-source corroboration is informative when it DOES happen.
• IODA fires heavily on connectivity disruptions that are not censorship (fiber cuts, BGP misconfigs). This is why the forecast pipeline excludes IODA from confirmed-censorship labels -- but we still report IODA presence here, because the journalist's question 'did multiple sources see this?' is answered by what they observed, not by our label rules.