Voidly
All findingsPublished research

Multilingual semantic search: 50+ languages over the incident corpus

Voidly Atlas semantic search at /v1/atlas/search ran on all-MiniLM-L6-v2, an English-only sentence-transformer. Journalists querying in Arabic, Persian, Russian, Chinese, Hindi, etc. got noise. Added a parallel multilingual index (paraphrase-multilingual-MiniLM-L12-v2, same 384-d shape, 50+…

Read the full published summary

Voidly Atlas semantic search at /v1/atlas/search ran on all-MiniLM-L6-v2, an English-only sentence-transformer. Journalists querying in Arabic, Persian, Russian, Chinese, Hindi, etc. got noise. Added a parallel multilingual index (paraphrase-multilingual-MiniLM-L12-v2, same 384-d shape, 50+ languages) over the same 2,696 incidents, stored in a separate table (incident_embeddings_multilingual) so the English-only endpoint stays byte-identical. New endpoint GET|POST /v1/atlas/search/multilingual?q=... returns the same shape plus `detected_language` from a script-based heuristic (Arabic/Persian/Cyrillic/CJK/Hebrew/Devanagari/Thai/Burmese/Ethiopic blocks). Test queries: AR "إيران الإنترنت" → top-3 Iran incidents (sim 0.71-0.73), FA "اینترنت ایران" → top-3 Iran (0.69-0.70), RU "блокировка интернета Россия" → top-3 Russia (0.76-0.79), CN "中国互联网封锁" → top-3 China (0.74), ES "censura internet" → Mauritius/South Sudan/Turkey censorship rows (0.74-0.77). Latency overhead +33-49ms vs English-only (40ms → 73-89ms) — the L12 encoder is ~3x larger but still well under the 200ms p95 target. Honest caveats inline in every response: (1) incident descriptions are mostly English so cross-language matching relies on the encoder mapping queries into a shared space, (2) multilingual models score 5-10pp lower precision than per-language, (3) script-based language detection cannot distinguish ES vs FR vs IT inside Latin. Weekly cron `5 5 * * 0` rebuilds the multilingual index 5 minutes after the English one.

Voidly Atlas's semantic search at /v1/atlas/search used all-MiniLM-L6-v2, an English-only sentence-transformer. That model was trained on the same general-purpose web corpora as the rest of the sentence-transformers family, but it embeds non-English queries into a near-random region of the 384-d space — so a journalist in Tehran typing اینترنت ایران or in Moscow typing блокировка интернета got mostly noise.

We added a parallel multilingual index using paraphrase-multilingual-MiniLM-L12-v2 — same dimensionality (384), same sentence-transformers library, but trained on the LaBSE parallel corpus across 50+ languages. We re-embedded all 2,696 incidents into a separate SQLite table (incident_embeddings_multilingual) so the legacy English-only endpoint stays byte-identical.

New endpoint

GET|POST /v1/atlas/search/multilingual?q=... — same response shape as the English-only endpoint, plus a detected_language field from a cheap script-based heuristic (Unicode block prefixes for Arabic / Persian / Cyrillic / CJK / Hangul / Hiragana / Hebrew / Devanagari / Thai / Burmese / Ethiopic). The heuristic distinguishes scripts but won't separate Spanish vs French vs Italian inside Latin.

Test queries

  • AR إيران الإنترنت → top-3 Iran incidents, similarity 0.71–0.73
  • FA اینترنت ایران → top-3 Iran incidents, similarity 0.69–0.70
  • RU блокировка интернета Россия → top-3 Russia incidents, similarity 0.76–0.79
  • CN 中国互联网封锁 → top-3 China disruption rows, similarity ≈ 0.74
  • ES censura internet → Mauritius / South Sudan / Turkey censorship rows, similarity 0.74–0.77
  • EN baseline Iran internet blocking → top-3 Iran, similarity 0.72–0.73

Latency

  • English-only baseline: ~40 ms (cached query encode + dot product over 2,637 rows)
  • Multilingual: 73–89 ms (+33-49 ms overhead — L12 encoder vs L6, ~3x the parameters)
  • Both well under the 200 ms p95 target documented in /v1/atlas/search/info

Honest caveats

  • Incident descriptions are mostly English — the corpus itself is monolingual. Cross-language matching relies entirely on the encoder mapping non-English queries into the same shared embedding space; we are not retrieving Arabic-language documents because none exist.
  • Multilingual sentence-transformers are typically 5-10pp lower precision than per-language or English-only equivalents on monolingual benchmarks. If you know your query is English, use /v1/atlas/search for best precision.
  • Script-based language detection is a heuristic. Persian-specific letters (ژپچگیک) flip an Arabic match to fa, but otherwise we report the dominant script. Pure-ASCII Latin queries always report as en; the multilingual model still embeds them correctly, the detection field is just a hint.
  • Templated IODA disruption descriptions (e.g. "Internet connectivity disruption in Iran") cluster very tightly in embedding space. Country/type filters (?country=IR&incident_type=censorship) remain strongly recommended.

How it's rebuilt

  • Weekly cron 5 5 * * 0 runs /opt/voidly-ai/scripts/rebuild-incident-embeddings-multilingual.py via the ML venv (sentence-transformers + torch-cpu).
  • The encoder run takes ~30 s on CPU for the full 2,696-incident corpus (batch=32).
  • Service auto-reloads embeddings every 5 min via _maybe_reload_multilingual() — no restart needed.

This is an accessibility ship, not a precision claim. Journalists and researchers outside the West can now ask Voidly Atlas questions in their own language and get back the same permalink-backed incident IDs we hand to English speakers.

The source record.

Original source links and labels. Linked responses may have changed since publication.