Voidly
All findingsPublished research

Government statement scraper: pairing ministry press releases with Voidly shutdown incidents

Voidly Atlas previously saw shutdowns only from the network side (OONI/IODA/Voidly probes). This v1 ships a curated government-statement scraper over 7 ministries (Russia Roskomnadzor, Iran MICT, Pakistan PTA, Bangladesh BTRC, Egypt MCIT, Turkey BTK, China MIIT), an NLP entity extractor (domain…

Read the full published summary

Voidly Atlas previously saw shutdowns only from the network side (OONI/IODA/Voidly probes). This v1 ships a curated government-statement scraper over 7 ministries (Russia Roskomnadzor, Iran MICT, Pakistan PTA, Bangladesh BTRC, Egypt MCIT, Turkey BTK, China MIIT), an NLP entity extractor (domain mentions, ASN refs, language-aware keyword vocabularies in en/ru/fa/ar/tr/bn/zh), and a correlation pass that pairs each statement with Voidly incidents in the same country within ±72h. First run ingested 13 statements (BD 8 + TR 5) across the reachable sources and surfaced 13 (statement, incident) pairs; the top pair (confidence 0.545) is a Turkey BTK regulatory draft published ~48h after a critical IODA-confirmed connectivity disruption on 2026-04-28 — a retroactive timing flagged correctly. 2 sources (RU, IR) are unreachable from our Vultr egress; 2 (PK, CN) reached but JS-shell-only — all documented up-front rather than papered over. Cron every 6h. Endpoints: GET /v1/atlas/government-statements, /v1/atlas/government-statements/info, /v1/atlas/government-statements/correlations, /v1/atlas/government-statements/{stmt_id}.

Voidly Atlas observes shutdowns from the network side: an IODA ASN flip, an OONI test-list anomaly, a 99% telemetry drop on a probe pair. That view is precise, but it’s also one-sided. Governments often publish — or fail to publish — a parallel narrative on their own ministry websites: “scheduled maintenance,” “regulatory action,” “security threat,” “spam attack.” This finding ships v1 of a curated government-statement scraper that turns that parallel narrative into structured, time-stamped, cross-referenceable evidence.

Why this matters

A statement near a confirmed shutdown is journalist-grade signal that a standalone statement (or a standalone shutdown) is not. If the Ministry of ICT tweets “routine maintenance” four hours before an IODA-confirmed outage, that’s a public warning the ministry doesn’t want to call a warning. If a regulator publishes a retroactive “spam attack” press release twenty-four hours after the network restores, that’s a cover story we should record alongside the network truth. Either way the pairing — not the standalone item — is what reporters cite.

Sources configured (v1)

We curate seven sources covering the highest-impact censorship regulators in the world. Each entry below documents the URL we poll plus an honest note on what we expect to retrieve:

  • Russia — Roskomnadzor (https://rkn.gov.ru/news/rsoc/, language ru). Frequently unreachable from our Vultr egress; documented as a geo-restricted source we still keep configured.
  • Iran — Ministry of ICT (https://www.ict.gov.ir/fa/news, language fa). DNS resolution and TLS handshakes often fail from outside Iran; coverage expected sparse.
  • Pakistan — PTA (https://www.pta.gov.pk/en/media-center/news, language en). PTA’s media center is a JavaScript-rendered single-page application; without a headless browser we retrieve the shell only. Source kept so coverage lights up if they ever ship a non-SPA listing.
  • Bangladesh — BTRC (http://www.btrc.gov.bd/site/view/notices, language bn). HTTP-only with a broken TLS chain; we drop verify deliberately. Live coverage: 8 statements ingested on the first fetch.
  • Egypt — MCIT (https://mcit.gov.eg/en, language en). The /Latest_News path is currently 500'ing; we fall back to the home page, which lists news links but no items matched keywords on the first run.
  • Turkey — BTK (https://www.btk.gov.tr/duyurular, language tr). Modern Tailwind site; dates are embedded in anchor text as DDMMMYYYY prefixes. Live coverage: 5 statements ingested.
  • China — MIIT (https://www.miit.gov.cn/xwdt/gxdt/sjdt/index.html, language zh). MIIT serves a ~5KB JavaScript shell on this page; near-zero coverage is the truth and we say so up-front.

Methodology (one tick of the cron)

  1. Fetch — one requests.get per source, 20s timeout, real Chrome User-Agent, TLS verify on by default (off only when we’ve documented a broken chain). We do not use a headless browser; we do not bypass logins, paywalls, or robots.txt.
  2. Parse — BeautifulSoup on the response. Each source declares a curated CSS selector for items, with an href-substring fallback for the cases where the CSS selector misses (Tailwind / hashed class names). For Turkey-BTK we additionally strip the embedded 21MAY2026-style date prefix from the anchor text and store it as the canonical published_at.
  3. Keyword filter — per-language vocabulary (en, ru, fa, ar, tr, bn, zh) of shutdown / block / limit / filter terms plus broader regulator language (“elektronik haberleşme,” “tebliğ,” “yönetmelik,” etc.). Short ambiguous English tokens (ban, VPN, DNS) use word-boundary regexes so “ban” does not match “Bangladesh.” Items that match zero keywords are dropped (we already caught a false-positive class with this filter and document the fix).
  4. Entity extraction — one regex sweep for bare domains and ASNs, plus the language-aware keyword hit list. A small block-list rejects analytics chrome (schema.org, w3.org).
  5. Persist — INSERT OR IGNORE into government_statements keyed on sha1(country | ministry | url | title) so re-fetching is safe. Statement text is stored in the original language; we explicitly do not translate.

Correlation pass

A second script, correlate-statements-to-shutdowns.py, runs 30 min after each scrape. For every statement it queries incidents in the same country whose first_seen is within ±72h of the statement’s published_at (falling back to fetched_at when the source did not expose a date). Each (statement, incident) pair gets a transparent confidence score:

  • entity_overlap — 0.3 per shared domain between the statement’s extracted entities and the incident’s affected_domains, capped at 1.5.
  • keyword — 0.12 per matched keyword in the statement, capped at 0.6 (5+ keyword hits = strong signal that the statement is really about a shutdown).
  • severity — 0.5 for critical / 0.3 warning / 0.1 info, plus 0.5x the incident’s upstream confidence.
  • type_weight — 1.0 censorship / 0.9 mixed / 0.5 disruption. Multiplied, so disruption-only pairs cannot dominate the leaderboard.

Delta-hours is signed: “warning” for >+3h (statement before incident), “retroactive” for <−3h (statement after), “simultaneous” otherwise. Output is a single JSON sidecar at /opt/voidly-ai/ml-deploy/government_statement_correlations_v1.json that the API serves verbatim.

First-run results (2026-05-21)

  • 7 ministries / URLs configured.
  • 58 items seen across reachable sources; 13 statements stored after keyword filtering (BD × 8, TR × 5).
  • 2 sources unreachable from our infra (RU Roskomnadzor TCP timeout, IR MICT DNS resolution failure) — documented honestly rather than papered over.
  • 2 sources reached but yielded 0 items (PK PTA — SPA shell; CN MIIT — JS shell). Both documented up-front; coverage expected near-zero by design.
  • 13 (statement, incident) pairs within ±72h on the first correlation pass, all in Turkey against IODA connectivity-disruption incidents. 0 pairs cleared the 1.0 confidence floor because all matching incidents are typed disruption and get a 0.5x weight — that’s the system working as designed, not a bug.

Best-example correlation

Highest-confidence pair on first run: BTK Turkey, 30 NIS 2026 — a draft regulation about electronic device identity registration (“Elektronik Kimlik Bilgisini Haiz Cihazların Kayıt Altına Alınmasına Dair Yönetmelik”) was published ≈48h after a critical IODA-confirmed Internet connectivity disruption in Turkey on 2026-04-28. Confidence score 0.545, sign “retroactive,” type_weight 0.5 because the incident is classed disruption, not confirmed censorship. The pair is real, the timing is real, and the type-weight discount keeps the score out of the high-confidence tier — exactly what we want from a transparent, journalist-defensible index.

Endpoints (live, public)

  • GET /v1/atlas/government-statements/info — sources configured + per-country counts + last fetch time.
  • GET /v1/atlas/government-statements?country=TR&limit=20 — list with filters (country, ministry, since, with_text).
  • GET /v1/atlas/government-statements/{stmt_id} — full record incl. statement_text.
  • GET /v1/atlas/government-statements/correlations?min_conf=0.5 — (statement, nearby incident) pairs within ±72h.

Honest caveats

  • JS-rendered ministry sites under-cover by design in the requests-only mode (PK PTA, CN MIIT). We do not run a headless browser; this is a deliberate operational choice.
  • Some governments are geo-restricted from our infra (RU Roskomnadzor, IR MICT). Coverage will appear thin in our data even when those sites are publishing aggressively.
  • Statements may be retroactive cover stories, not real warnings. We surface delta_sign so the reader can tell.
  • Translation is not in scope. Statement text is stored in the original language; the keyword filter is the cross-language index.
  • Correlation is not causation. A pair only means “same country, ±72h.” The confidence_score is a transparent sum, not a calibrated probability.
  • Some incidents that pair against statements are typed disruption, which our forecast labeling treats as noise. The pair-weight 0.5x discount keeps disruption-only pairs out of the high-confidence tier.
  • If a ministry redesigns their site, our selectors will miss new items until they’re updated. That’s the cost of not using a generic AI-extraction layer; the upside is the pipeline is auditable and cheap.

The source record.

Original source links and labels. Linked responses may have changed since publication.