Voidly Atlas observes shutdowns from the network side: an IODA ASN flip, an OONI test-list anomaly, a 99% telemetry drop on a probe pair. That view is precise, but it’s also one-sided. Governments often publish — or fail to publish — a parallel narrative on their own ministry websites: “scheduled maintenance,” “regulatory action,” “security threat,” “spam attack.” This finding ships v1 of a curated government-statement scraper that turns that parallel narrative into structured, time-stamped, cross-referenceable evidence.
Why this matters
A statement near a confirmed shutdown is journalist-grade signal that a standalone statement (or a standalone shutdown) is not. If the Ministry of ICT tweets “routine maintenance” four hours before an IODA-confirmed outage, that’s a public warning the ministry doesn’t want to call a warning. If a regulator publishes a retroactive “spam attack” press release twenty-four hours after the network restores, that’s a cover story we should record alongside the network truth. Either way the pairing — not the standalone item — is what reporters cite.
Sources configured (v1)
We curate seven sources covering the highest-impact censorship regulators in the world. Each entry below documents the URL we poll plus an honest note on what we expect to retrieve:
- Russia — Roskomnadzor
(
https://rkn.gov.ru/news/rsoc/, language ru). Frequently unreachable from our Vultr egress; documented as a geo-restricted source we still keep configured. - Iran — Ministry of ICT
(
https://www.ict.gov.ir/fa/news, language fa). DNS resolution and TLS handshakes often fail from outside Iran; coverage expected sparse. - Pakistan — PTA
(
https://www.pta.gov.pk/en/media-center/news, language en). PTA’s media center is a JavaScript-rendered single-page application; without a headless browser we retrieve the shell only. Source kept so coverage lights up if they ever ship a non-SPA listing. - Bangladesh — BTRC
(
http://www.btrc.gov.bd/site/view/notices, language bn). HTTP-only with a broken TLS chain; we drop verify deliberately. Live coverage: 8 statements ingested on the first fetch. - Egypt — MCIT
(
https://mcit.gov.eg/en, language en). The/Latest_Newspath is currently 500'ing; we fall back to the home page, which lists news links but no items matched keywords on the first run. - Turkey — BTK
(
https://www.btk.gov.tr/duyurular, language tr). Modern Tailwind site; dates are embedded in anchor text asDDMMMYYYYprefixes. Live coverage: 5 statements ingested. - China — MIIT
(
https://www.miit.gov.cn/xwdt/gxdt/sjdt/index.html, language zh). MIIT serves a ~5KB JavaScript shell on this page; near-zero coverage is the truth and we say so up-front.
Methodology (one tick of the cron)
- Fetch — one
requests.getper source, 20s timeout, real Chrome User-Agent, TLS verify on by default (off only when we’ve documented a broken chain). We do not use a headless browser; we do not bypass logins, paywalls, orrobots.txt. - Parse — BeautifulSoup on the response.
Each source declares a curated CSS selector for items, with an
href-substring fallback for the cases where the CSS selector
misses (Tailwind / hashed class names). For Turkey-BTK we
additionally strip the embedded
21MAY2026-style date prefix from the anchor text and store it as the canonicalpublished_at. - Keyword filter — per-language vocabulary (en, ru, fa, ar, tr, bn, zh) of shutdown / block / limit / filter terms plus broader regulator language (“elektronik haberleşme,” “tebliğ,” “yönetmelik,” etc.). Short ambiguous English tokens (ban, VPN, DNS) use word-boundary regexes so “ban” does not match “Bangladesh.” Items that match zero keywords are dropped (we already caught a false-positive class with this filter and document the fix).
- Entity extraction — one regex sweep for
bare domains and ASNs, plus the language-aware keyword hit list.
A small block-list rejects analytics chrome
(
schema.org,w3.org). - Persist —
INSERT OR IGNOREintogovernment_statementskeyed onsha1(country | ministry | url | title)so re-fetching is safe. Statement text is stored in the original language; we explicitly do not translate.
Correlation pass
A second script,
correlate-statements-to-shutdowns.py, runs 30 min after
each scrape. For every statement it queries
incidents in the same country whose
first_seen is within ±72h of the statement’s
published_at (falling back to fetched_at
when the source did not expose a date). Each (statement, incident)
pair gets a transparent confidence score:
- entity_overlap — 0.3 per shared domain
between the statement’s extracted entities and the incident’s
affected_domains, capped at 1.5. - keyword — 0.12 per matched keyword in the statement, capped at 0.6 (5+ keyword hits = strong signal that the statement is really about a shutdown).
- severity — 0.5 for critical / 0.3 warning / 0.1 info, plus 0.5x the incident’s upstream confidence.
- type_weight — 1.0 censorship / 0.9 mixed / 0.5 disruption. Multiplied, so disruption-only pairs cannot dominate the leaderboard.
Delta-hours is signed: “warning” for >+3h (statement
before incident), “retroactive” for <−3h
(statement after), “simultaneous” otherwise. Output is a
single JSON sidecar at
/opt/voidly-ai/ml-deploy/government_statement_correlations_v1.json
that the API serves verbatim.
First-run results (2026-05-21)
- 7 ministries / URLs configured.
- 58 items seen across reachable sources; 13 statements stored after keyword filtering (BD × 8, TR × 5).
- 2 sources unreachable from our infra (RU Roskomnadzor TCP timeout, IR MICT DNS resolution failure) — documented honestly rather than papered over.
- 2 sources reached but yielded 0 items (PK PTA — SPA shell; CN MIIT — JS shell). Both documented up-front; coverage expected near-zero by design.
- 13 (statement, incident) pairs within ±72h
on the first correlation pass, all in Turkey against IODA
connectivity-disruption incidents. 0 pairs cleared the 1.0
confidence floor because all matching incidents are typed
disruptionand get a 0.5x weight — that’s the system working as designed, not a bug.
Best-example correlation
Highest-confidence pair on first run: BTK Turkey, 30 NIS 2026 — a draft regulation about electronic device identity registration (“Elektronik Kimlik Bilgisini Haiz Cihazların Kayıt Altına Alınmasına Dair Yönetmelik”) was published ≈48h after a critical IODA-confirmed Internet connectivity disruption in Turkey on 2026-04-28. Confidence score 0.545, sign “retroactive,” type_weight 0.5 because the incident is classed disruption, not confirmed censorship. The pair is real, the timing is real, and the type-weight discount keeps the score out of the high-confidence tier — exactly what we want from a transparent, journalist-defensible index.
Endpoints (live, public)
GET /v1/atlas/government-statements/info— sources configured + per-country counts + last fetch time.GET /v1/atlas/government-statements?country=TR&limit=20— list with filters (country, ministry, since, with_text).GET /v1/atlas/government-statements/{stmt_id}— full record incl. statement_text.GET /v1/atlas/government-statements/correlations?min_conf=0.5— (statement, nearby incident) pairs within ±72h.
Honest caveats
- JS-rendered ministry sites under-cover by design in the requests-only mode (PK PTA, CN MIIT). We do not run a headless browser; this is a deliberate operational choice.
- Some governments are geo-restricted from our infra (RU Roskomnadzor, IR MICT). Coverage will appear thin in our data even when those sites are publishing aggressively.
- Statements may be retroactive cover stories, not real warnings. We surface delta_sign so the reader can tell.
- Translation is not in scope. Statement text is stored in the original language; the keyword filter is the cross-language index.
- Correlation is not causation. A pair only means “same country, ±72h.” The confidence_score is a transparent sum, not a calibrated probability.
- Some incidents that pair against statements are typed
disruption, which our forecast labeling treats as noise. The pair-weight 0.5x discount keeps disruption-only pairs out of the high-confidence tier. - If a ministry redesigns their site, our selectors will miss new items until they’re updated. That’s the cost of not using a generic AI-extraction layer; the upside is the pipeline is auditable and cheap.