Is AI as Good in Arabic as in English? What the Latest Benchmarks Show
Table of Contents
Arabic AI has nearly caught up with English in formal Arabic. On one hard reasoning test, the top model lost only 2 points moving from English to Arabic. It still falls short on dialects, cultural fit, and reliable evaluation. So Arabic AI is not bad, but it is uneven, and the scarce resource is no longer more Arabic text. It is trusted human judgment of Arabic quality. This guide covers what the benchmarks show, why dialects remain hard, how quality is measured, which Arabic AI models lead, and why the leading builders still rely on human experts.
How good is AI in Arabic?
AI is strong in Modern Standard Arabic (MSA), the formal Arabic of news, education, and government. The gap with English is small for top models and much larger for older ones.
MMLU-ProX, presented at EMNLP 2025, asks the same 11,829 hard questions in 29 languages, which allows a direct comparison:
| Model | English score | Arabic score | Points lost in Arabic |
|---|---|---|---|
| Qwen3 | 80.7% | 78.7% | 2.0 |
| GPT-4.1 | 79.8% | 74.1% | 5.7 |
| Llama 3.3 70B (an older model) | 65.7% | 51.0% | 14.7 |
This test measures reasoning in formal Arabic. It says little about dialects or cultural fit, and that is where the gap opens up.
Four Numbers on Formal Arabic, Dialects, Data, and Testing
The gap is small in formal Arabic and larger in dialects. Clean data and reliable tests remain the bottlenecks.
2 points
Lost in formal Arabic
Top model on MMLU-ProX, 78.7% against 80.7% in English
47.7%
Average score in dialects
19 open models, against 51.9% in MSA and 62.8% in English
0.6%
Of websites are in Arabic
Against 49.5% in English
81 to 88%
Arabic coding prompts rewritten
Before TII would use them in its QIMMA leaderboard
Source: MMLU-ProX, EMNLP 2025; DialectalArabicMMLU, arXiv; W3Techs; TII, QIMMA (April 2026).
Why does AI struggle with Arabic dialects?
AI struggles with Arabic dialects because they are underrepresented in the benchmarks used to test models, and because models understand a dialect better than they write it. Dialects are where the performance gaps persist.
DialectalArabicMMLU tested models on about 15,000 questions in five dialects: Syrian, Egyptian, Emirati, Saudi, and Moroccan. It found persistent gaps against standard Arabic and English:
| Models tested | English | MSA | Dialect average | Lowest dialect (Moroccan) |
|---|---|---|---|---|
| Average of 19 open models | 62.8% | 51.9% | 47.7% | 45.0% |
| Jais-2-70B, the highest dialect average in the table | 76.2% | 70.7% | 67.2% | 65.0% |
Writing in dialect is harder still. AL-QASIDA, from Johns Hopkins University and Cohere, found that models do not produce dialect as well as they understand it, and that this comes more from reluctance than from poor fluency. One of its authors needed 14 turns of conversation before ChatGPT answered in Egyptian Arabic.
Why is there so little good Arabic data?
There is little Arabic online, and much of it is noisy. W3Techs estimates that 0.6% of websites use Arabic, against 49.5% for English.
When LightOn built ArabicWeb24 from an Arabic web crawl, cleaning cut about 284 billion tokens to 28 billion. News made up 76% of its top 150 URLs.
More text is not the answer on its own, because native data beats translated data. In a 2025 study on Lebanese translation, about 3,000 native Lebanese sentences beat 140,000 non-native ones on a Lebanese test set, scoring 74.4 against 73.2 on the xCOMET quality measure.
How is Arabic AI quality measured?
Arabic AI quality is measured with benchmarks, standard sets of questions and tasks that are scored automatically. The problem is that Arabic AI benchmarks are often unreliable, because many were translated from English. The Arabic NLP community now says so openly.
- Translated tests introduce errors. The team behind the Open Arabic LLM Leaderboard v2 wrote that many tasks were direct translations from English, which introduced “linguistic and contextual mismatches.”
- Even respected tests contain faults. When the Technology Innovation Institute (TII) built its QIMMA leaderboard, it discarded 3.1% of ArabicMMLU questions as faulty and had to rewrite 81% to 88% of Arabic coding problems.
- Quality checks take real human time. MMLU-ProX needed over 400 hours of expert review, with native professional translators checking each of the 15 languages they reviewed.
Which are the best Arabic LLMs?
There is no single best Arabic large language model (LLM), because the ranking depends on the test. Rankings also change fast, so treat these results as a snapshot.
| Leaderboard | Who leads | Worth knowing |
|---|---|---|
| HELM Arabic (Stanford, checked November 2026) | Arabic.AI’s LLM-X ranks first (0.919), followed by incept-arabic-1 (0.883). Claude 4.7 Opus (0.841) and GPT-5.4 (0.825) also place in the top 10 | Produced in collaboration with Arabic.AI |
| QIMMA (TII, April 2026) | Qwen3.5 leads, ahead of Karnak (Egypt) and Jais-2-70B (UAE) | Validates its benchmarks before scoring models |
HELM shows how quickly this moves. When Stanford first published the leaderboard in December 2025, LLM-X was already first and multilingual open-weights models took 4 of the top 10 places. None of those four models is in the top 10 today. Large multilingual models still compete strongly with models built for Arabic, while Arabic-specific models lead in some areas. On QIMMA, Jais-2-70B ranks highest on ArabicMMLU and Arabic culture.
Why does Arabic AI still need human experts?
Arabic AI still needs human experts because the remaining weak spots are the ones that more text does not fix. Leading model builders rely on people:
- HUMAIN refined ALLaM 34B with input from over 600 domain experts and 250 evaluators.
- TII trained Falcon Arabic on 100% native Arabic data, with no machine translation.
- The Jais 2 team used large batches of human preference comparisons.
The performance problems are mostly in dialects and in certain domains of expertise, such as business and financial Arabic. Our Language Service team’s native linguists deliver annotation, tagging, instructions, preference tuning, evaluation, and red teaming for AI-generated output in Arabic and its dialects. For more on the business side, see our guide to Arabic localization and our look at what is holding back AI value in the Gulf.
Contact our experts to understand how we solve for it →
Frequently Asked Questions
How good is AI in Arabic?
In formal Arabic, it is close to English. On the MMLU-ProX test, top models lost 2 to 6 points compared with English, although an older model lost about 15. It still falls short on dialects, cultural fit, and reliable evaluation.
Why does AI struggle with Arabic dialects?
Dialects are underrepresented in the benchmarks used to test models, and tests show persistent gaps against standard Arabic and English. Models also understand dialect better than they write it, and they are often reluctant to answer in it.
Which are the best Arabic LLMs?
It depends on the test. Arabic.AI’s LLM-X ranks first on Stanford’s HELM Arabic, as checked in November 2026. On TII’s QIMMA in April 2026, Qwen3.5 led, ahead of Karnak from Egypt and Jais-2-70B from the UAE.
How is Arabic AI quality measured?
Through benchmarks and leaderboards such as ArabicMMLU, HELM Arabic, and QIMMA. Many benchmarks were translated from English and carry mismatches, so newer leaderboards validate their questions or favor native Arabic data.
Why does Arabic AI still need human experts?
Because the remaining gaps are in dialects, cultural fit, and specialist domains. Leading builders use domain experts, evaluators, and human preference comparisons to refine their models. HUMAIN, for example, used over 600 domain experts and 250 evaluators for ALLaM 34B.
ARABIC AI
See How Our AI Experts and Arabic Language Experts Solve for This Problem
Our AI engineers are LLM specialists building agents, RAG pipelines, fine-tuning models, and deploying production AI systems. Our native linguists deliver annotation, tagging, instructions, preference tuning, evaluation, and red teaming for AI-generated output in Arabic and its dialects.
