Arabic Arena
A speech-to-text leaderboard scored on many test sets at once: MSA, Gulf, Egyptian, Levantine, Maghrebi, code-switching, noisy speech, and simulated 8 kHz telephone and call-center channels. One test set can mislead; the average of many is harder to game. Lower is better.
Click a column to sort; the best value in each column is highlighted. Small grey numbers are 95 % bootstrap intervals (WER) and, under the rank, the rank range. Values in parentheses are the number of sets finished so far. Closed APIs and fine-tunes with gated weights are unranked references, scored on the re-hosted sets only. † marks models trained on the train split of one of the benchmark corpora, so their score there is not zero-shot (hover for details). Halluc. % is the share of noise-only clips for which the model wrote any words; it is not part of the Arena score.
Per-set results
Charts
Arena WER
Lower is better.
Accuracy vs speed
RTFx on one NVIDIA A10 (log scale). Down and to the right is better.
Phone-channel robustness
WER on the same source sets: clean audio, then the telephony channel, then the call-center condition.
Where models break
Hallucination on noise-only audio
Share of 300 clips with no target speech (line silence, hold hiss, mains hum, distant babble, office, street and transport noise; half sent through a phone codec) for which the model still wrote words (v2: three or more words, so a short filler does not count). Lower is better. Not part of the Arena score.
How scores are computed
- Per set: corpus WER and CER, meaning total errors over total reference words or characters, after the same normalization is applied to references and hypotheses (Arabic: diacritics, alef/hamza/ta-marbuta spellings and punctuation are unified and digits are written out as words).
- Per track: the unweighted mean over that track's sets.
- Arena: .
- Scorer v2: per set, dialect-lenient normalization (fillers, elongation, same-word spelling variants such as برضو/برضه), space-insensitive alignment (ما فيش = مفيش), several references where a set has them (MGB-3's four annotators, Mixat's Arabic-script English) and every utterance's errors capped at its reference length. The Arena is the weighted geometric mean of the set scores; intervals come from a bootstrap that resamples whole speakers, videos or recordings, and the rank is a simultaneous 95 % confidence set. See the methodology page for the measurements behind each choice.
- Uncertainty: 95% intervals from a paired bootstrap over utterances (1000 resamples per set, the same resample for every model). The rank range is how far a model's Arena rank moves across resamples; models whose ranges overlap are not reliably separated.
- Telephony: v1 board: 8 kHz, a 300–3400 Hz band, a G.711, GSM, AMR-NB, G.726 or Opus codec, and packet loss. v2 board: chain v2, an A-law-first MENA codec mix with AMR-WB, Opus and G.722 paths, tandem coding to A-law and losses on the encoded frames. Call-center adds noise and a room at the caller before that channel (v2: a handset room and unintelligible babble).
- Decoding: greedy, language forced to the suite language, batch inference on one NVIDIA A10 (24 GB).
- Models: open-weight models are ranked, all run locally; closed APIs and fine-tunes with gated weights are listed as unranked references.