Probe scheduling optimizer: a Thompson-sampling priority list for where to point the probe network next

The Voidly probe network has a hard capacity ceiling: roughly 40 nodes, 62 domains, a 5-minute cadence. Today every node runs the same fixed domain list everywhere. That spends probe attention uniformly — and uniform is wasteful. A (country, domain) pair we have measured 400 times, all of them showing the same block, has nothing left to teach us on probe 401. A pair we have seen four times, or one whose block state flipped last week, is exactly where a fresh probe adds knowledge.

The probe scheduling optimizer is a recommendation engine for that problem. It produces a ranked priority list — which (country, domain) pairs to probe next — so a scheduler, or a human running the network, can concentrate scarce probe cycles where the model is most uncertain.

The method: Thompson sampling per (country, domain)

Each (country, domain) pair is modelled as a bandit arm carrying a Beta(alpha, beta) posterior over the latent parameter p = P(this pair is blocked). We start from a weakly informative Jeffreys prior Beta(0.5, 0.5).

1. Accumulate evidence. alpha gains recency-weighted block observations, beta gains recency-weighted unblock observations. A block is any evidence row whose signal_type is in the standard Atlas block set (dns-blocking, http-blocking-tcp-reset, block, interference, tor-blocking, and similar); an unblock is an ok row. Each observation is weighted by exp(-age_days / 45), so a block state that flipped a month ago is not drowned by a year of stale agreement.
2. Compute posterior variance. The variance of Beta(a, b) is a·b / ((a+b)² · (a+b+1)). It is large when a pair is under-sampled or genuinely 50/50, and small when many consistent observations have piled up. That variance is the information-gain proxy — probe where the posterior is widest.
3. Draw a Thompson sample. One sample from each Beta. This is the bandit exploration step: it occasionally surfaces a pair even when its mean variance is modest, so the ranking never fully starves an arm.
4. Combine and rank. priority = posterior_variance × recency_weight × flip_weight × (1 + thompson_weight · (1 - |sample-0.5|·2)). The recency weight favours pairs not observed in a long time; the flip weight (1.5×) favours pairs whose block state changed in the trailing 30 days; the Thompson term is bounded small so the deterministic variance-and-recency signal leads and the sample only re-orders near-ties.

First run

The first build (21 May 2026) scored 1,054 (country, domain) pairs from a 365-day evidence window, skipping 57 cold pairs with fewer than two observations. 269 pairs showed a block-state flip in the trailing 30 days. The five highest-priority pairs to probe next were Uzbekistan / twitter.com, Iran / binance.com, Algeria / tiktok.com, Iran / tumblr.com, and Azerbaijan / telegram.org — all sparse-but-contested pairs where the posterior is wide and the last observation is months old.

Why this is hard to do honestly

Four honest caveats are baked into every API response:

1. This is a recommendation, not a control loop. It is not wired into probe_module.py or the live probe scheduler. Nothing changes the probe cadence yet. The pipeline produces a priority list; consuming it is a separate, out-of-scope step.
2. Thompson sampling assumes independent arms. Each (country, domain) pair is treated as an independent bandit. In reality DPI policy correlates domains within a country — a censor flipping one news site often flips many at once. A truly correct model would be hierarchical or contextual. Treat the ranking as a heuristic, not an optimum.
3. High variance is not always censorship uncertainty. A wide posterior can mean a pair is simply intermittently reachable — a flaky DNS resolver, a congested transit link — rather than genuinely contested. The recent-flip term partly separates the two, but not perfectly.
4. Cold pairs are out of scope. Pairs with fewer than two observations cannot be scored from evidence and are not ranked here. The existing fixed-list scheduler already covers them, so unseen pairs still get probed.

Endpoints

• GET /v1/atlas/probe-priority — the top-K (country, domain) pairs to probe next, ranked by priority score. Supports ?limit, ?country, ?domain, ?recent_flip=true, and ?min_observations filters.
• GET /v1/atlas/probe-priority/info — full methodology, the Beta prior, the priority formula, generation metadata, and the honest caveats. No row data.

Schema and provenance

Pipeline script: scripts/build-probe-scheduling-priority.py. Endpoint patch: scripts/patch-probe-priority-endpoint.py. Both shipped to Vultr at /opt/voidly-ai/scripts/. Sidecar written to /opt/voidly-ai/ml-deploy/probe_scheduling_priority_v1.json. Cron entry: 50 4 * * * (daily, 04:50 UTC). Schema: voidly-atlas-probe-scheduling-priority/v1. The Thompson draws use a fixed random seed so the daily cron is reproducible.