Voidly
All findingsPublished research

The AS-topology GNN does beat chance — once you give it a real label and a real test

Voidly Atlas runs a 2-layer GraphSAGE GNN over the CAIDA AS-AS peering graph (7,060 nodes, 841K edges) to score per-ASN shutdown risk. It shipped 2026-05-21 with passed_promote_floor=false and an honest caveat: leave-one-out CV ran across only 6 ASNs, so although it reported AUC 0.80, a permutation…

Read the full published summary

Voidly Atlas runs a 2-layer GraphSAGE GNN over the CAIDA AS-AS peering graph (7,060 nodes, 841K edges) to score per-ASN shutdown risk. It shipped 2026-05-21 with passed_promote_floor=false and an honest caveat: leave-one-out CV ran across only 6 ASNs, so although it reported AUC 0.80, a permutation test gave p=0.32 — n=6 cannot reject the null. This finding is the better-powered re-evaluation, and it surfaced a second flaw the n=6 caveat had hidden. The old label, had_shutdown_next_7d, was defined as "did any next-7-day measurement hit block_rate >= 0.5" — but every ASN-tagged evidence row in the database is a CensoredPlanet block, so block_rate is always 1.0 and the label collapsed to "does this ASN have >= 5 measurements on a post-cutoff day" — a measurement-density flag, not a censorship signal. Proof: the 40 old positives had a median of 24 post-cutoff rows, the 18 old negatives a median of 2. And the GNN node features included n_evidence_30d/180d, n_unique_dates and has_evidence, so the model could read its own label off its inputs — the AUC 0.80 was partly density predicting density. The fix builds a genuine censorship label. CensoredPlanet rows carry signal_value, a continuous block-intensity score (low = the ASN let the probe through, high = it blocked it). Every ASN with >= 20 measurements in the trailing 180 days is relabeled by the fraction of its measurements showing strong blocking (signal_value >= 0.5): >= 60% blocked is positive (censors, 62 ASNs), <= 25% is negative (clean, 35 ASNs), the 25-60% middle band is dropped. That is the directive definition exactly — positive = ASN with confirmed censorship evidence, negative = ASN with clean evidence — and it grows the labeled set from 58 to 97 ASNs across 30 countries. Because signal_value and signal_level are tightly coupled, all seven signal-derived features (block_rate_30d/180d and five pct_* buckets) are dropped to kill leakage; the GNN is retrained on density + topology features only (evidence counts, distinct days, domains, CAIDA degree, country risk tier). Evaluation is leave-AS-out only, never shuffled: leave-one-AS-out across all 97 ASNs, and leave-one-COUNTRY-out (30 country folds, every ASN of a held country removed together — the leakage-safe gate, since same-country ASNs share country features and are topological neighbors). Skill is pooled out-of-fold AUC plus a 5,000-permutation test that shuffles only the label vector. The honest verdict is SIGNIFICANT: leave-one-COUNTRY-out gives AUC 0.7751, permutation p=0.0002; leave-one-AS-out gives AUC 0.7645, p=0.0002 — the two protocols agree, so the result is not same-country leakage. The permutation null averaged AUC 0.4998 (exactly chance, 95th percentile 0.60); the observed 0.775 is far in the tail. Confusion at threshold 0.5 under country-out: precision 0.86, recall 0.68 (42 TP, 20 FN, 28 TN, 7 FP), mean score 0.68 for censoring ASNs vs 0.39 for clean ones. Both the 0.65 AUC floor and the p<0.05 bar are cleared by a wide margin under the leakage-safe protocol, so passed_promote_floor is flipped to true. The measured framing: AUC 0.775 is clearly-better-than-chance, not operationally decisive — recall 0.68 still misses a third of censoring ASNs, the 97-ASN label set is small, and the genuine label says whether an ASN censors, not when. But the GraphSAGE-over-AS-topology approach is validated: AS topology plus measurement density carry real, statistically significant signal for whether an ASN censors. The old AUC 0.80 / n=6 / p=0.32 headline is superseded; the endpoint now ships a defensible significance claim instead of a thin one. Both outcomes were acceptable under the directive — this one happens to be a real positive.

Voidly Atlas runs a GraphSAGE graph neural network over the CAIDA AS-AS peering graph (7,060 nodes, 841K edges) to forecast per-ASN 7-day shutdown probability. It shipped on 2026-05-21 with an honest red flag baked into every API response: passed_promote_floor: false. The reason given was statistical thinness — leave-one-out cross- validation ran across only 6 ASNs, reported AUC 0.80, but a permutation test gave p = 0.32. With n = 6 you cannot reject the null hypothesis at any reasonable level. This finding is the better-powered re-evaluation: did the GNN actually work, or was it riding noise?

Two flaws, not one

Auditing the original setup surfaced a second problem the n = 6 caveat had hidden. The GNN's label was had_shutdown_next_7d — defined as “did any day in the next 7 days hit block_rate ≥ 0.5 with ≥ 5 measurements?” But every single ASN-tagged evidence row in the database is a CensoredPlanet block — there are no clean ASN-tagged rows at all. So block_rate is always 1.0, and the label collapses to a far dumber question: does this ASN have ≥ 5 measurements on a post-cutoff day?

It was a measurement-density flag, not a censorship signal. The proof: the 40 “positive” ASNs had a median of 24 post-cutoff evidence rows; the 18 “negatives” had a median of 2. And the GNN's node features included n_evidence_30d, n_evidence_180d, n_unique_dates and has_evidence — so the model could read its own label straight off its inputs. The original AUC 0.80 was, in part, density predicting density.

The fix — a genuine censorship label

CensoredPlanet rows carry signal_value, a continuous block-intensity score: low (~0) means the ASN let the measurement probe through, high (~1) means the ASN blocked it. That is the real signal. We relabeled each ASN with ≥ 20 CensoredPlanet measurements in the trailing 180 days by the fraction of its measurements that show strong blocking (signal_value ≥ 0.5):

  • Positive (censors): ≥ 60% of measurements blocked → 62 ASNs
  • Negative (clean): ≤ 25% of measurements blocked → 35 ASNs
  • Ambiguous middle band (25–60%): 8 ASNs, dropped — not labeled

That is the directive's definition exactly: positive = an ASN with confirmed censorship evidence, negative = an ASN with clean evidence. The labeled set goes from 58 → 97 ASNs (62 pos / 35 neg, balanced) spanning 30 countries — up from the old 6-ASN evaluation set.

Closing the leakage door

signal_value and the categorical signal_level are tightly coupled (a critical row has block-fraction 1.0, a warning row 0.0), so any feature derived from them would near-perfectly leak the new label. The re-evaluation therefore dropped all seven signal-derived features — block_rate_30d/180d and the five pct_* signal-composition buckets. The GNN was retrained and re-scored on a leakage-audited feature set of density and topology only: evidence counts, distinct measurement days, domains tested, CAIDA node degree (total / peer / provider), and country risk tier. This makes the test an honest one: can AS topology plus how-much-we-measure predict whether an individual ASN censors?

Properly-powered evaluation — leave-AS-out only

Two cross-validation protocols, both leave-AS-out, never a shuffled split (a prior Atlas audit showed shuffled splits leak structure):

  • Leave-one-AS-out — each of the 97 labeled ASNs held out in turn, model trained on the other 96.
  • Leave-one-COUNTRY-out — every ASN of the held country removed together (30 country folds). Same-country ASNs share country_risk_tier and are topological neighbors, so plain leave-one-AS-out can leak through siblings. Country-out is the leakage-safe generalization test and is used as the promote gate.

Skill is measured by pooled out-of-fold AUC plus a label-permutation p-value: 5,000 times the label vector is shuffled (and only the label vector — never the features, graph, or fold assignment) and the AUC recomputed, giving the distribution of AUC under “the model has no skill.”

The honest verdict — significant

  • Leave-one-COUNTRY-out (the gate): AUC 0.7751, permutation p = 0.0002 — 97 held-out ASNs across 30 country folds.
  • Leave-one-AS-out: AUC 0.7645, permutation p = 0.0002 — 97 folds. The two protocols agree, which means the result is not an artifact of same-country leakage.
  • The permutation null AUC averaged 0.4998 (exactly chance) with a 95th percentile of 0.60. The observed 0.775 sits far out in the tail — only ~1 in 5,000 shuffles reached it.
  • Confusion at threshold 0.5 (country-out): 42 true positives, 20 false negatives, 28 true negatives, 7 false positives → precision 0.86, recall 0.68. Mean GNN score is 0.68 for ASNs that censor vs 0.39 for clean ASNs — the right ordering, with a wide +0.29 gap.

Both the floor (AUC ≥ 0.65) and the significance bar (p < 0.05) are cleared, by a wide margin, under the leakage-safe protocol. passed_promote_floor has been flipped to true. The GNN genuinely beats chance: AS topology and measurement-density signal carry real, statistically significant information about whether an individual ASN censors — message-passing lets a data-rich ASN's neighborhood inform the prediction for a data-poor one.

What this does and does not claim

This is a real, publishable positive result — but a measured one. AUC 0.775 is “clearly better than chance,” not “operationally decisive.” The recall at 0.5 is 0.68: the GNN still misses about a third of censoring ASNs. The labels remain CensoredPlanet-only and sparse (97 ASNs is small); a larger labeled set would tighten the estimate. And “this ASN censors” is a cross-sectional property, not a dated 7-day forecast — the genuine label says whether, not when. The honest framing: the GraphSAGE-over-AS-topology approach is validated, the old AUC 0.80 / n = 6 / p = 0.32 headline is superseded, and the endpoint now ships a defensible significance claim instead of a thin one.

What shipped

  • Genuine signal_value-based censorship label (97 ASNs, 62 / 35) replacing the density-artifact label.
  • Leakage-audited density + topology feature set (7 signal-derived features dropped).
  • Leave-one-AS-out and leave-one-COUNTRY-out CV with a 5,000-permutation test, replacing the underpowered 6-fold tier-1-only CV.
  • Live sidecar updated — passed_promote_floor: true, AUC 0.7751, p = 0.0002 — surfaced inline on every /v1/forecast/asn-gnn/* response with the full re-eval block and honest caveats.
  • Build script: scripts/rebuild-gnn-asn-genuine-labels.py. Full re-eval report: gnn_asn_reeval_v2.json.

Live at

GET /v1/forecast/asn-gnn/info — full sidecar with the reeval_v2 block (both CV protocols, per-fold predictions)
GET /v1/forecast/asn-gnn/{asn} — per-ASN score + raw inputs + honest caveats
GET /v1/forecast/asn-gnn/coverage — every scoreable ASN ranked by predicted risk

The source record.

Original source links and labels. Linked responses may have changed since publication.