Voidly Atlas runs a GraphSAGE graph neural network over the CAIDA AS-AS peering graph (7,060 nodes, 841K edges) to forecast per-ASN 7-day shutdown probability. It shipped on 2026-05-21 with an honest red flag baked into every API response: passed_promote_floor: false. The reason given was statistical thinness — leave-one-out cross- validation ran across only 6 ASNs, reported AUC 0.80, but a permutation test gave p = 0.32. With n = 6 you cannot reject the null hypothesis at any reasonable level. This finding is the better-powered re-evaluation: did the GNN actually work, or was it riding noise?

Two flaws, not one

Auditing the original setup surfaced a second problem the n = 6 caveat had hidden. The GNN's label was had_shutdown_next_7d — defined as “did any day in the next 7 days hit block_rate ≥ 0.5 with ≥ 5 measurements?” But every single ASN-tagged evidence row in the database is a CensoredPlanet block — there are no clean ASN-tagged rows at all. So block_rate is always 1.0, and the label collapses to a far dumber question: does this ASN have ≥ 5 measurements on a post-cutoff day?

It was a measurement-density flag, not a censorship signal. The proof: the 40 “positive” ASNs had a median of 24 post-cutoff evidence rows; the 18 “negatives” had a median of 2. And the GNN's node features included n_evidence_30d, n_evidence_180d, n_unique_dates and has_evidence — so the model could read its own label straight off its inputs. The original AUC 0.80 was, in part, density predicting density.

The fix — a genuine censorship label

CensoredPlanet rows carry signal_value, a continuous block-intensity score: low (~0) means the ASN let the measurement probe through, high (~1) means the ASN blocked it. That is the real signal. We relabeled each ASN with ≥ 20 CensoredPlanet measurements in the trailing 180 days by the fraction of its measurements that show strong blocking (signal_value ≥ 0.5):

• Positive (censors): ≥ 60% of measurements blocked → 62 ASNs
• Negative (clean): ≤ 25% of measurements blocked → 35 ASNs
• Ambiguous middle band (25–60%): 8 ASNs, dropped — not labeled

That is the directive's definition exactly: positive = an ASN with confirmed censorship evidence, negative = an ASN with clean evidence. The labeled set goes from 58 → 97 ASNs (62 pos / 35 neg, balanced) spanning 30 countries — up from the old 6-ASN evaluation set.

Closing the leakage door

signal_value and the categorical signal_level are tightly coupled (a critical row has block-fraction 1.0, a warning row 0.0), so any feature derived from them would near-perfectly leak the new label. The re-evaluation therefore dropped all seven signal-derived features — block_rate_30d/180d and the five pct_* signal-composition buckets. The GNN was retrained and re-scored on a leakage-audited feature set of density and topology only: evidence counts, distinct measurement days, domains tested, CAIDA node degree (total / peer / provider), and country risk tier. This makes the test an honest one: can AS topology plus how-much-we-measure predict whether an individual ASN censors?

Properly-powered evaluation — leave-AS-out only

Two cross-validation protocols, both leave-AS-out, never a shuffled split (a prior Atlas audit showed shuffled splits leak structure):

• Leave-one-AS-out — each of the 97 labeled ASNs held out in turn, model trained on the other 96.
• Leave-one-COUNTRY-out — every ASN of the held country removed together (30 country folds). Same-country ASNs share country_risk_tier and are topological neighbors, so plain leave-one-AS-out can leak through siblings. Country-out is the leakage-safe generalization test and is used as the promote gate.

Skill is measured by pooled out-of-fold AUC plus a label-permutation p-value: 5,000 times the label vector is shuffled (and only the label vector — never the features, graph, or fold assignment) and the AUC recomputed, giving the distribution of AUC under “the model has no skill.”

The honest verdict — significant

• Leave-one-COUNTRY-out (the gate): AUC 0.7751, permutation p = 0.0002 — 97 held-out ASNs across 30 country folds.
• Leave-one-AS-out: AUC 0.7645, permutation p = 0.0002 — 97 folds. The two protocols agree, which means the result is not an artifact of same-country leakage.
• The permutation null AUC averaged 0.4998 (exactly chance) with a 95th percentile of 0.60. The observed 0.775 sits far out in the tail — only ~1 in 5,000 shuffles reached it.
• Confusion at threshold 0.5 (country-out): 42 true positives, 20 false negatives, 28 true negatives, 7 false positives → precision 0.86, recall 0.68. Mean GNN score is 0.68 for ASNs that censor vs 0.39 for clean ASNs — the right ordering, with a wide +0.29 gap.

Both the floor (AUC ≥ 0.65) and the significance bar (p < 0.05) are cleared, by a wide margin, under the leakage-safe protocol. passed_promote_floor has been flipped to true. The GNN genuinely beats chance: AS topology and measurement-density signal carry real, statistically significant information about whether an individual ASN censors — message-passing lets a data-rich ASN's neighborhood inform the prediction for a data-poor one.

What this does and does not claim

This is a real, publishable positive result — but a measured one. AUC 0.775 is “clearly better than chance,” not “operationally decisive.” The recall at 0.5 is 0.68: the GNN still misses about a third of censoring ASNs. The labels remain CensoredPlanet-only and sparse (97 ASNs is small); a larger labeled set would tighten the estimate. And “this ASN censors” is a cross-sectional property, not a dated 7-day forecast — the genuine label says whether, not when. The honest framing: the GraphSAGE-over-AS-topology approach is validated, the old AUC 0.80 / n = 6 / p = 0.32 headline is superseded, and the endpoint now ships a defensible significance claim instead of a thin one.

What shipped

• Genuine signal_value-based censorship label (97 ASNs, 62 / 35) replacing the density-artifact label.
• Leakage-audited density + topology feature set (7 signal-derived features dropped).
• Leave-one-AS-out and leave-one-COUNTRY-out CV with a 5,000-permutation test, replacing the underpowered 6-fold tier-1-only CV.
• Live sidecar updated — passed_promote_floor: true, AUC 0.7751, p = 0.0002 — surfaced inline on every /v1/forecast/asn-gnn/* response with the full re-eval block and honest caveats.
• Build script: scripts/rebuild-gnn-asn-genuine-labels.py. Full re-eval report: gnn_asn_reeval_v2.json.

Live at

GET /v1/forecast/asn-gnn/info — full sidecar with the reeval_v2 block (both CV protocols, per-fold predictions)
GET /v1/forecast/asn-gnn/{asn} — per-ASN score + raw inputs + honest caveats
GET /v1/forecast/asn-gnn/coverage — every scoreable ASN ranked by predicted risk