Voidly Atlas observes shutdowns from the network side: an IODA ASN flip, an OONI test-list anomaly, a 99% telemetry drop on a probe pair. That view is precise, but it’s also one-sided. Governments often publish — or fail to publish — a parallel narrative on their own ministry websites: “scheduled maintenance,” “regulatory action,” “security threat,” “spam attack.” This finding ships v1 of a curated government-statement scraper that turns that parallel narrative into structured, time-stamped, cross-referenceable evidence.

Why this matters

A statement near a confirmed shutdown is journalist-grade signal that a standalone statement (or a standalone shutdown) is not. If the Ministry of ICT tweets “routine maintenance” four hours before an IODA-confirmed outage, that’s a public warning the ministry doesn’t want to call a warning. If a regulator publishes a retroactive “spam attack” press release twenty-four hours after the network restores, that’s a cover story we should record alongside the network truth. Either way the pairing — not the standalone item — is what reporters cite.

Sources configured (v1)

We curate seven sources covering the highest-impact censorship regulators in the world. Each entry below documents the URL we poll plus an honest note on what we expect to retrieve:

• Russia — Roskomnadzor (https://rkn.gov.ru/news/rsoc/, language ru). Frequently unreachable from our Vultr egress; documented as a geo-restricted source we still keep configured.
• Iran — Ministry of ICT (https://www.ict.gov.ir/fa/news, language fa). DNS resolution and TLS handshakes often fail from outside Iran; coverage expected sparse.
• Pakistan — PTA (https://www.pta.gov.pk/en/media-center/news, language en). PTA’s media center is a JavaScript-rendered single-page application; without a headless browser we retrieve the shell only. Source kept so coverage lights up if they ever ship a non-SPA listing.
• Bangladesh — BTRC (http://www.btrc.gov.bd/site/view/notices, language bn). HTTP-only with a broken TLS chain; we drop verify deliberately. Live coverage: 8 statements ingested on the first fetch.
• Egypt — MCIT (https://mcit.gov.eg/en, language en). The /Latest_News path is currently 500'ing; we fall back to the home page, which lists news links but no items matched keywords on the first run.
• Turkey — BTK (https://www.btk.gov.tr/duyurular, language tr). Modern Tailwind site; dates are embedded in anchor text as DDMMMYYYY prefixes. Live coverage: 5 statements ingested.
• China — MIIT (https://www.miit.gov.cn/xwdt/gxdt/sjdt/index.html, language zh). MIIT serves a ~5KB JavaScript shell on this page; near-zero coverage is the truth and we say so up-front.

Methodology (one tick of the cron)

1. Fetch — one requests.get per source, 20s timeout, real Chrome User-Agent, TLS verify on by default (off only when we’ve documented a broken chain). We do not use a headless browser; we do not bypass logins, paywalls, or robots.txt.
2. Parse — BeautifulSoup on the response. Each source declares a curated CSS selector for items, with an href-substring fallback for the cases where the CSS selector misses (Tailwind / hashed class names). For Turkey-BTK we additionally strip the embedded 21MAY2026-style date prefix from the anchor text and store it as the canonical published_at.
3. Keyword filter — per-language vocabulary (en, ru, fa, ar, tr, bn, zh) of shutdown / block / limit / filter terms plus broader regulator language (“elektronik haberleşme,” “tebliğ,” “yönetmelik,” etc.). Short ambiguous English tokens (ban, VPN, DNS) use word-boundary regexes so “ban” does not match “Bangladesh.” Items that match zero keywords are dropped (we already caught a false-positive class with this filter and document the fix).
4. Entity extraction — one regex sweep for bare domains and ASNs, plus the language-aware keyword hit list. A small block-list rejects analytics chrome (schema.org, w3.org).
5. Persist — INSERT OR IGNORE into government_statements keyed on sha1(country | ministry | url | title) so re-fetching is safe. Statement text is stored in the original language; we explicitly do not translate.

Correlation pass

A second script, correlate-statements-to-shutdowns.py, runs 30 min after each scrape. For every statement it queries incidents in the same country whose first_seen is within ±72h of the statement’s published_at (falling back to fetched_at when the source did not expose a date). Each (statement, incident) pair gets a transparent confidence score:

• entity_overlap — 0.3 per shared domain between the statement’s extracted entities and the incident’s affected_domains, capped at 1.5.
• keyword — 0.12 per matched keyword in the statement, capped at 0.6 (5+ keyword hits = strong signal that the statement is really about a shutdown).
• severity — 0.5 for critical / 0.3 warning / 0.1 info, plus 0.5x the incident’s upstream confidence.
• type_weight — 1.0 censorship / 0.9 mixed / 0.5 disruption. Multiplied, so disruption-only pairs cannot dominate the leaderboard.

Delta-hours is signed: “warning” for >+3h (statement before incident), “retroactive” for <−3h (statement after), “simultaneous” otherwise. Output is a single JSON sidecar at /opt/voidly-ai/ml-deploy/government_statement_correlations_v1.json that the API serves verbatim.

First-run results (2026-05-21)

• 7 ministries / URLs configured.
• 58 items seen across reachable sources; 13 statements stored after keyword filtering (BD × 8, TR × 5).
• 2 sources unreachable from our infra (RU Roskomnadzor TCP timeout, IR MICT DNS resolution failure) — documented honestly rather than papered over.
• 2 sources reached but yielded 0 items (PK PTA — SPA shell; CN MIIT — JS shell). Both documented up-front; coverage expected near-zero by design.
• 13 (statement, incident) pairs within ±72h on the first correlation pass, all in Turkey against IODA connectivity-disruption incidents. 0 pairs cleared the 1.0 confidence floor because all matching incidents are typed disruption and get a 0.5x weight — that’s the system working as designed, not a bug.

Best-example correlation

Highest-confidence pair on first run: BTK Turkey, 30 NIS 2026 — a draft regulation about electronic device identity registration (“Elektronik Kimlik Bilgisini Haiz Cihazların Kayıt Altına Alınmasına Dair Yönetmelik”) was published ≈48h after a critical IODA-confirmed Internet connectivity disruption in Turkey on 2026-04-28. Confidence score 0.545, sign “retroactive,” type_weight 0.5 because the incident is classed disruption, not confirmed censorship. The pair is real, the timing is real, and the type-weight discount keeps the score out of the high-confidence tier — exactly what we want from a transparent, journalist-defensible index.

Endpoints (live, public)

• GET /v1/atlas/government-statements/info — sources configured + per-country counts + last fetch time.
• GET /v1/atlas/government-statements?country=TR&limit=20 — list with filters (country, ministry, since, with_text).
• GET /v1/atlas/government-statements/{stmt_id} — full record incl. statement_text.
• GET /v1/atlas/government-statements/correlations?min_conf=0.5 — (statement, nearby incident) pairs within ±72h.

Honest caveats

• JS-rendered ministry sites under-cover by design in the requests-only mode (PK PTA, CN MIIT). We do not run a headless browser; this is a deliberate operational choice.
• Some governments are geo-restricted from our infra (RU Roskomnadzor, IR MICT). Coverage will appear thin in our data even when those sites are publishing aggressively.
• Statements may be retroactive cover stories, not real warnings. We surface delta_sign so the reader can tell.
• Translation is not in scope. Statement text is stored in the original language; the keyword filter is the cross-language index.
• Correlation is not causation. A pair only means “same country, ±72h.” The confidence_score is a transparent sum, not a calibrated probability.
• Some incidents that pair against statements are typed disruption, which our forecast labeling treats as noise. The pair-weight 0.5x discount keeps disruption-only pairs out of the high-confidence tier.
• If a ministry redesigns their site, our selectors will miss new items until they’re updated. That’s the cost of not using a generic AI-extraction layer; the upside is the pipeline is auditable and cheap.