Why uncertainty alone is not enough

The active-learning queue at /v1/sentinel/active-learning-queue used to rank candidates by a single number: |p - 0.5|, the distance from the decision threshold. This is the textbook uncertainty sampling heuristic from Lewis & Gale 1994 — ask the labeler for the case the model is least sure about.

It is a fine baseline. But it ignores one thing every labeling budget cares about: not every label is worth the same. A correct label on an Iran day cascades through 762 sibling measurements via shared-feature generalization. A correct label on a Mali day with 1 measurement in the last 30 days is mostly a curiosity. The model already shrugs both the same way.

The 3-factor impact score

The new default ranking is the heuristic Estimated Error Reduction (EER) from Settles 2009 (UW-Madison TR-1648, Active Learning Literature Survey), factored into three pieces we can cheaply compute:

impact_score = uncertainty  ×  volume_factor  ×  drift_factor

• uncertainty = 1 - |p - 0.5| × 2  — peaks at p = 0.5, drops to 0 at p ∈ {0, 1}.
• volume_factor = log(1 + n_evidence_30d) / log(1 + global_max)  — 0..1, high-volume countries close to 1.0. Sourced from the live evidence table.
• drift_factor = clamp01(country_recent_block_rate / (global_mean × 2))  — 0..1, anomalous-period countries close to 1.0. Sourced from the DBSCAN rolling 14-day window.

What changed in the top of the queue

On the smoke-test run (2026-05-21, 13 candidates), the impact ranking moves Venezuela 2026-05-20 to the top (impact 0.67, uncertainty 0.67, volume 1.00, drift 1.00) and pushes Philippines and Brazil candidates with similar uncertainty (0.59 / 0.60) to the bottom (impact 0.022 each) because both countries have near-zero drift in the current window. The labeler's next click is now spent on a region the model is genuinely off-balance in.

How to read the response

Every candidate now carries the four scores inline:

{
  "candidate_id": "VE-2026-05-20-12340",
  "country": "VE",
  "current_probability": 0.335,
  "impact_score": 0.67,
  "uncertainty_score": 0.67,
  "volume_factor": 1.0,
  "drift_factor": 1.0,
  "rank_by": "impact"
}

You can fall back to the old ordering at any time by passing ?rank_by=uncertainty. Both modes return the same fields; only the sort key changes.

Honest caveat — this is a heuristic, not Bayesian EER

A real Estimated Error Reduction implementation (Roy & McCallum 2001) would train one delta-loss model per candidate: for each unlabeled point, simulate the two possible labels, retrain the classifier, and measure the expected drop in held-out loss. That is the correct number. It is also O(n * retrain_cost) — tractable for 20-row toy problems, untenable for a 4,237-sample / 131-country classifier running on a single 5-vCPU box.

The 3-factor product above is a rough analytic stand-in. It captures the two strongest signals (model uncertainty, sample density per country) plus a drift modifier that approximates “the per-country evidence base has shifted, so a new label will move the decision boundary in this region.” In Settles' survey this falls under the “density-weighted methods” family (§ 3.4, eq. 12 generalized). We are not claiming this matches Bayesian EER — we are claiming it is meaningfully better than pure uncertainty on a tight labeling budget, and we can compute it in under 50ms per request from cached factor tables.

References

• Settles, B. (2009). Active Learning Literature Survey. University of Wisconsin–Madison TR-1648. PDF
• Lewis, D. & Gale, W. (1994). A Sequential Algorithm for Training Text Classifiers. SIGIR '94.
• Roy, N. & McCallum, A. (2001). Toward Optimal Active Learning through Sampling Estimation of Error Reduction. ICML '01.