Humanity's Last Exam

Humanity's Last Exam (HLE) is a benchmark of expert-level academic questions designed to test how close artificial intelligence systems are to the frontier of human knowledge. It was released on 23 January 2025 by the Center for AI Safety (CAIS) and Scale AI, and published in Nature in January 2026 as "A benchmark of expert-level academic questions to assess AI capabilities".^[1][2]

History

HLE was conceived by Dan Hendrycks, director of the Center for AI Safety, who stated that he was inspired to create it after a conversation with Elon Musk arguing that existing language-model benchmarks such as MMLU were too easy. Hendrycks worked with Scale AI to compile the questions.^[3] Leading models had reached over 90% accuracy on popular benchmarks — a condition known as benchmark saturation — which limited informed measurement of state-of-the-art capabilities.^[1]

Questions were crowdsourced from subject-matter experts at institutions worldwide. An initial pool of roughly 70,000 candidate questions was first filtered by frontier AI models: questions the models answered, or answered no better than random guessing on multiple-choice items, advanced to two rounds of human expert review. Submitters of the top-rated questions shared a prize pool of US$500,000 — US$5,000 for each of the top 50 questions and US$500 for the next 500. After launch, a "community feedback bug bounty" was opened to remove errors.^[3][4]

Key Concepts and Technology

The public set contains 2,500 questions across more than 100 subjects, with 500 additional private holdout questions guarding against overfitting. Roughly 24% of questions are multiple choice and 76% are short-answer, exact-match items; about 10–14% are multimodal, requiring a model to interpret an image or diagram alongside text.^[3][4] Subject coverage skews hard: mathematics 41%, biology and medicine 11%, computer science and AI 10%, humanities and social science 9%, physics 9%, chemistry 7% and engineering 4%.^[4]

Every question has a single unambiguous answer that a domain expert can verify but that cannot be quickly resolved by internet retrieval. Automated grading scores exact matches and uses an automated judge model for verification, with the evaluation framework released open-source.^[1][3]

Results track a wide capability gap. In January 2025 GPT-4o scored about 2.7% on the full benchmark. By late 2026 leading models exceeded 60% on the text-only subset — Claude Opus 5.5 at 61.4%, Gemini 4 Argon at 57.1% — while expert human performance sits near 90%, so HLE remains unsaturated.^[5] The benchmark has also surfaced quality problems of its own: a July 2025 investigation by FutureHouse suggested around 30% of HLE's text-only chemistry and biology answers could be incorrect, prompting the team to launch HLE-Rolling, a continuously revised version.^[3] Spin-offs include Humanity's Last Code Exam (HLCE), built from ICPC World Finals and Olympiad in Informatics problems.^[6]

Applications and Impact

HLE is used by laboratories to report frontier model capability, by researchers to study calibration and reasoning, and by policymakers as a reference point for how close AI is to expert-level performance. Its design — expert-authored, retrieval-resistant and privately held out — is widely cited as the response to the saturation that made earlier exams uninformative, and it has been cited in Stanford HAI's AI Index as one of the "more challenging benchmarks" of its era.^[3][4]

>See Also

• AI Benchmarking
• LLM Evaluation
• MMLU
• Reasoning Models
• AI Safety