Voidly Atlas's semantic search at /v1/atlas/search
used all-MiniLM-L6-v2, an
English-only sentence-transformer. That model was trained on
the same general-purpose web corpora as the rest of the
sentence-transformers family, but it embeds non-English
queries into a near-random region of the 384-d space — so a
journalist in Tehran typing اینترنت ایران or in
Moscow typing блокировка интернета got mostly noise.
We added a parallel multilingual index using
paraphrase-multilingual-MiniLM-L12-v2 —
same dimensionality (384), same sentence-transformers
library, but trained on the LaBSE parallel corpus across
50+ languages. We re-embedded all 2,696
incidents into a separate SQLite table
(incident_embeddings_multilingual)
so the legacy English-only endpoint stays byte-identical.
New endpoint
GET|POST /v1/atlas/search/multilingual?q=... —
same response shape as the English-only endpoint, plus a
detected_language field from a
cheap script-based heuristic (Unicode block prefixes for
Arabic / Persian / Cyrillic / CJK / Hangul / Hiragana /
Hebrew / Devanagari / Thai / Burmese / Ethiopic). The
heuristic distinguishes scripts but won't separate
Spanish vs French vs Italian inside Latin.
Test queries
- AR إيران الإنترنت → top-3 Iran incidents, similarity 0.71–0.73
- FA اینترنت ایران → top-3 Iran incidents, similarity 0.69–0.70
- RU блокировка интернета Россия → top-3 Russia incidents, similarity 0.76–0.79
- CN 中国互联网封锁 → top-3 China disruption rows, similarity ≈ 0.74
- ES censura internet → Mauritius / South Sudan / Turkey censorship rows, similarity 0.74–0.77
- EN baseline Iran internet blocking → top-3 Iran, similarity 0.72–0.73
Latency
- English-only baseline: ~40 ms (cached query encode + dot product over 2,637 rows)
- Multilingual: 73–89 ms (+33-49 ms overhead — L12 encoder vs L6, ~3x the parameters)
- Both well under the 200 ms p95 target documented in
/v1/atlas/search/info
Honest caveats
- Incident descriptions are mostly English — the corpus itself is monolingual. Cross-language matching relies entirely on the encoder mapping non-English queries into the same shared embedding space; we are not retrieving Arabic-language documents because none exist.
-
Multilingual sentence-transformers are typically 5-10pp
lower precision than per-language or English-only
equivalents on monolingual benchmarks. If you know your query is
English, use
/v1/atlas/searchfor best precision. -
Script-based language detection is a heuristic. Persian-specific
letters (ژپچگیک) flip an Arabic match to
fa, but otherwise we report the dominant script. Pure-ASCII Latin queries always report asen; the multilingual model still embeds them correctly, the detection field is just a hint. -
Templated IODA disruption descriptions (e.g. "Internet
connectivity disruption in Iran") cluster very tightly in
embedding space. Country/type filters
(
?country=IR&incident_type=censorship) remain strongly recommended.
How it's rebuilt
- Weekly cron
5 5 * * 0runs/opt/voidly-ai/scripts/rebuild-incident-embeddings-multilingual.pyvia the ML venv (sentence-transformers + torch-cpu). - The encoder run takes ~30 s on CPU for the full 2,696-incident corpus (batch=32).
- Service auto-reloads embeddings every 5 min via
_maybe_reload_multilingual()— no restart needed.
This is an accessibility ship, not a precision claim. Journalists and researchers outside the West can now ask Voidly Atlas questions in their own language and get back the same permalink-backed incident IDs we hand to English speakers.