MMLU (Massive Multitask Language Understanding)

MMLU (Massive Multitask Language Understanding) is a multiple-choice examination designed to measure how broadly a language model understands the world. Introduced by Dan Hendrycks and colleagues in 2020 and published at ICLR 2021, it became one of the most frequently quoted figures in model releases and research papers, and for several years served as the de facto standard test of general knowledge in large language models.^[1]

History

The benchmark was released on arXiv in September 2020 as Measuring Massive Multitask Language Understanding. At the time, the strongest published system, GPT-3, scored 43.9 per cent in the few-shot setting against a random baseline of 25 per cent — a result the authors presented as evidence that even large models lacked broad knowledge. The simplicity of a single accuracy number made MMLU easy to compare across papers, and adoption spread quickly through the field.

As models improved, the benchmark motivated a family of extensions. MMLU-Pro (2024) raised the difficulty by expanding to ten answer options and adding reasoning-heavy questions, while Global-MMLU (2024) translated and professionally post-edited the questions into 42 languages, including Malay, to correct the cultural and linguistic bias of the original English data set.^[3]

Design

MMLU contains 15,908 multiple-choice questions drawn from freely available sources such as professional and university entrance examinations, including GRE and USMLE practice material, advanced placement questions and textbook exercises. The questions were collected and verified by graduate and undergraduate students and are grouped into 57 subjects spanning four broad areas — humanities, social sciences, STEM and professional or applied topics. Subjects range from elementary mathematics and high school geography to professional law, medicine, econometrics, machine learning and moral scenarios.

Each question has four options, and scoring is plain classification accuracy. The data set is split into a small development set of five questions per subject for few-shot prompting, a validation set of 1,540 questions and a test set of 14,079 questions. Models are evaluated zero-shot or few-shot without task-specific fine-tuning, using a standard prompt template that identifies the subject, so that the score reflects knowledge acquired during pretraining rather than adaptation to the test.

Performance and criticism

MMLU scores rose from roughly 44 per cent in 2020 to the high 80s and low 90s for leading models by 2024, and the benchmark has attracted several recurring criticisms:

• Contamination, because the questions and answers circulate publicly and may appear in training corpora (see data contamination).
• Saturation, since small score differences at the top no longer correspond to meaningful differences in capability.
• Format effects, because four-option guessing rewards calibration as much as knowledge.
• Cultural bias, because the subjects were assembled from Western, largely American sources. Global-MMLU found that 28 per cent of questions require culturally specific knowledge and that roughly 85 per cent of geography questions focus on North America or Europe.

Despite this, MMLU remains a common reporting line in model cards and a useful baseline for comparing generations of models on a fixed scale.

>See Also

• AI Benchmarking
• LLM Evaluation
• Large Language Models
• Data Contamination
• SEA-LION