What this is

Voidly Atlas has an active-learning queue: every day it ranks the unlabeled country-days the v3.3 censorship classifier is least sure about — uncertainty × measurement-volume × DBSCAN drift — and surfaces them for human review at /atlas/active-learning. The point of an active-learning loop is that those human labels flow back into the model and make the next version better. Until this week, that loop was open. The chain ran:

queue  →  submit-label  →  aggregate (≥3-vote consensus)  →  active_learning_labels.json  →  ∅

The consensus file was a dead end. Nothing read it. A reviewer could label a hundred uncertain Iran days and the v3.3 classifier would never see one of them, because v3.3 trains from a separate file (labeled_incidents_v3.3.json) that the aggregator never touched. This finding is the fix: the loop is now closed end to end.

What closing the loop means, concretely

Three pieces ship together.

1. A promotion step. promote-al-labels-to-training.py reads the consensus file and, for every candidate that reached a binary censorship/no_censorship consensus, rebuilds the full 16-feature v3.3 training row from the same live evidence database the v3.3 corpus builder uses — anomaly rate, measurement count, spike magnitude, the three probe features, the three neighbor-contagion features — and appends it to labeled_incidents_v3.3.json with _source='active-learning' provenance. It de-duplicates on (country, date): re-running never double-counts, and a country-day whose human consensus later flips is replaced, not duplicated. A country-day that cannot be reconstructed from evidence (no data that day) is skipped and reported, never fabricated. Crucially, it inherits a prior Atlas data fix: IODA connectivity-disruption incidents (fiber cuts, BGP leaks, weather) are not counted as confirmed censorship in the neighbor-incident feature — disruption is a real network observation but it is not citable censorship.

2. A retrain trigger. Once RETRAIN_MIN_NEW_LABELS (10) genuinely-new human-consensus country-days have landed in the corpus, the promotion script writes /opt/voidly-ai/data/retrain-queue.json with trigger='active-learning'. That is the same queue file the weekly retrain cron (weekly-retrain.sh, Sunday 02:00 UTC) already services, and the same file the concept-drift detector writes for drift-triggered retrains. The trigger respects a 12-hour cooldown shared with the drift trigger: if a retrain is already queued, the active-learning provenance is merged into the existing entry (trigger becomes drift+active-learning) rather than overwriting it. No thrash; the two retrain sources serialize cleanly.

3. A public status endpoint. GET /v1/sentinel/active-learning-loop-status reports every stage of the loop in one call: queue depth, labels submitted, consensus reached, rows promoted into the corpus, the retrain-queue state, the timestamp of the last loop run, and the honest simulated F1 lift below. When zero humans have labelled anything it returns loop_state: "plumbed_idle" and says so plainly.

The honest result: does feeding the loop actually help?

A closed loop is only worth shipping if the labels it feeds back measurably improve the model. We measured it instead of assuming it. simulate-al-loop-lift.py pulls the current AL queue's top-50 candidates, assigns each a proxy label (the incident table within ±3 days if one exists, else negative), reconstructs the v3.3 feature rows, and trains two classifiers with the identical v3.3 recipe — baseline corpus, and corpus + the proxy-labelled AL rows — then evaluates both with the same leave-one-country-out (LOCO) protocol the real v3.3 promote gate uses.

The result, on the live queue: of 50 ranked candidates, only 13–19 were reconstructable from evidence on any given run (the rest are recent days the evidence DB has not filled in yet). Adding those rows to the 4,237-row corpus moved LOCO median F1 between −0.0124 and −0.0362 and LOCO mean F1 between +0.0002 and −0.0055 across two runs hours apart. In plain terms: the lift is negligible — statistically indistinguishable from zero, and slightly negative on the median. We are not going to claim a lift that is not there.

Why so flat? Two honest reasons. First, 13–19 rows over a 4,200-row corpus is a rounding error — the fact that two runs hours apart disagreed (median −0.0124 vs −0.0362) is itself the proof: the swing is run-to-run sampling noise, not a stable effect. Second, and more interesting: the per-country breakdown shows the augmented model improving on exactly the hard, sparse-data countries the queue is built to surface — Belize, Oman, Morocco, Uzbekistan, China all gained double-digit F1 points on the larger run — while it slips elsewhere, netting out near zero. That is the queue doing its job: it is surfacing learnable rows for the countries the classifier is weakest on. The flat headline is a volume problem, not a signal problem. The loop will pay off when reviewers label a few hundred candidates, not a dozen.

Caveats we are keeping honest about

• The loop is plumbed but idle until humans label. The status endpoint says plumbed_idle today because zero labels have been submitted. The value of an active-learning loop is entirely contingent on humans actually reviewing the queue. We built the machine; the labels are not ours to fabricate.
• The simulated lift uses proxy labels, not human labels. Proxy positives come from the incident table within ±3 days; everything else is labelled negative. This is optimistic — the AL queue surfaces uncertain days precisely where the incident table is silent or ambiguous, so real human labels would be both noisier and more informative than the proxy. Treat the near-zero number as “is the queue surfacing rows the model could learn from at all,” not as a promotion signal.
• Promotion is not the promote gate. Landing labels in the corpus does not bypass anything. A retrain still goes through the v3.3 LOCO evaluation and the dual-holdout gate; a regressing model is still rejected. The loop feeds the training set; it does not feed the production model directly.
• Neighbor features on promoted rows use a simpler contagion. The full v3.3 builder re-weights neighbor contagion by 180-day regime similarity; the promotion script uses an unweighted geographic mean because the regime weighting cannot be recomputed cheaply at promotion time. This under-weights contagion on promoted rows; it never fabricates it. Documented in the corpus metadata.

Why publish a near-zero result

Because the honest answer to “did closing the loop lift F1” is “not yet, and here is exactly why.” The loop had to be closed regardless — an open active-learning loop is a bug, not a feature — but shipping it with a fabricated lift number would be the dishonest move. The infrastructure is real and verified end to end; the payoff is gated on label volume, and the status endpoint makes that state inspectable by anyone. This finding is filed alongside the eight other honest negatives in the Atlas record.