What we tested

Hypothesis: a self-supervised encoder pretrained on a superset of unlabeled country-day features could lift the v3.3 GradientBoosting classifier's 0.729 stratified F1. The unlabeled superset was 9,722 country-days extracted from the same evidence table v3.3 labels its 4,237 country-days from — 5,485 of those rows are UNLABELED (most are thin-data days with fewer than 5 measurements that the v3.1 expansion filtered out). We implemented a tabular masked-autoencoder (MAE) in PyTorch: encoder 13 → 32 → 16, decoder 16 → 32 → 13, mask 15% of features per row, MSE loss on masked positions only, 50 epochs, batch 256.

Fine-tune: load the pretrained encoder, freeze the first layer (l1 + bn1), attach a linear classifier head 16 → 1, train with BCE-with-logits at the v3.3 class-weight balanced ratio, 80 epochs, batch 64, lr 5e-4. Evaluated with the same stratified 5-fold and leave-country-out methodology v3.3 was evaluated under.

Honest scope note: we used 13 features for both SSL and a freshly rebuilt 13-feature v3.3-style GradientBoosting baseline (i.e., the production v3.3 16-feature variant minus the three contagion features). The contagion features can't be cleanly pretrained on unlabeled rows because they require adjacency-graph context that's tightly bound to the labeled-window time series, so keeping them out of both SSL and the local baseline gives an apples-to-apples comparison. We also report the published v3.3 production number (16 features) for context.

The result

Stratified 5-fold F1: 0.573 ± 0.019 (AUC 0.802). The local 13-feature GradientBoosting baseline ran the same split and scored F1 0.673 ± 0.026 (AUC 0.868). The production v3.3 16-feature model sits at F1 0.729. SSL is 10pp BELOW the like-for-like baseline and 16pp below production. Promote floor for v3.6 was 0.76 stratified — failed by 19pp.

Leave-country-out F1 median: 0.645 (n=127 countries with at least one positive). The local 13-feature baseline LOCO median is 0.818; production v3.3 16-feature is 0.870. SSL is 17pp below the like-for-like baseline and 23pp below production. Promote floor for LOCO was 0.85 — failed by 21pp.

Per-country LOCO had no clear pattern. SSL did improve a handful of tail countries (MT +1.0, EE +0.5, YE +0.47) but collapsed on well-served Western/OECD countries the baseline trivially solves (TG −1.0, AU −0.9, JP −0.83, PT −0.81, ES −0.78). Net regression wins by a wide margin.

Why SSL underperformed here

Tabular MAE / SCARF / VIME pretraining helps most when (a) the unlabeled set is at least 10× the labeled set and (b) features are noisy enough that reconstruction learns a real denoising prior. Our setup violates both:

• The unlabeled superset (9,722 country-days) is only 2.3× the labeled set (4,237). The published SCARF / Bahri results need 5–50× more unlabeled rows than labels to show consistent gains over GBM on small tabular tasks.
• Our 13 features are already heavily engineered (rate-spike interactions, probe-agreement composites, weekday cyclicals) and have low noise. Reconstruction has little to learn that the GBM doesn't already extract from raw splits.
• The MLP encoder lost the GBM's native handling of interactions and missing-value semantics. A linear head over a 16-dim latent simply doesn't carve the same decision boundary that a depth-4 GradientBoosting ensemble carves over the raw 13 features.
• The contagion features (which add roughly +6pp to stratified and +16pp to LOCO median for v3.3) live OUTSIDE the SSL pipeline. Even a successful SSL encoder would have to recover equivalent regional signal from per-country-day features alone — and they don't carry it.

Decision

v3.6 is NOT promoted. v3.3 GradientBoosting (16 features, regime-weighted contagion, F1 0.729 stratified / 0.870 LOCO median) stays in production unchanged. The experimental bundle is archived on Vultr at /opt/voidly-ai/models/experimental/censorship_classifier_v3.6_ssl_20260521_121900.pkl for reproducibility. This is the second SSL-style negative result on our classifier (TabPFN was the first); the pattern is now documented twice.

What we shipped

• Feature build: scripts/build-ssl-pretrain-features.py (9,722 country-days x 13 features, all-time evidence window).
• Pretrain: scripts/pretrain-tabular-ssl.py (PyTorch tabular MAE, 50 epochs, 4.4 s wall-clock).
• Fine-tune + eval: scripts/finetune-ssl-classifier.py (freeze layer 1, BCE-with-logits, stratified 5-fold + LOCO, 626 s wall-clock).
• Archived bundle + metrics JSON for downstream replication. v3.3 PKL on Vultr is bit-identical to the pre-experiment file.

Configuration notes for replication

Same Vultr CPU box that hosts production classifier (no GPU). Pinned torch==2.11.0+cu130 in .venv-ml; CUDA driver too old for GPU acceleration so all training ran on CPU. Seed 42 throughout. Pretrain wall-clock is dominated by data loading, not gradient steps — the MSE has converged well before epoch 50, so a 20-epoch run would be indistinguishable. LOCO eval dominates wall-clock (~10 minutes for 127 country splits at 60 epochs each); this could be parallelized in a future round.