LLM Evaluation

LLM evaluation (often shortened to evals) is the systematic measurement of a large language model's capabilities, reliability, safety and cost — typically by running a fixed set of prompts through a standardised harness and scoring the outputs against reference answers or rubrics. Its purpose is comparability: the same underlying model can differ by 5–10 percentage points on MMLU depending on prompt format, few-shot settings and whether chain-of-thought is elicited, so without a harness that fixes these choices, cross-model rankings are close to meaningless.^[4]

Background

Early NLP evaluation revolved on single datasets — SQuAD, GLUE, SuperGLUE — but foundation models in the 2020s pushed evaluation toward suites. Stanford's Holistic Evaluation of Language Models (HELM), introduced in 2022, was a watershed: it measured seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) across 16 core scenarios and 7 targeted evaluations, covering 42 scenarios in total for 30 prominent models. Before HELM, models were evaluated on average only 17.9% of the core scenarios, and some prominent models shared no common scenario at all; afterwards all 30 were densely benchmarked under standardised conditions.^[1] EleutherAI's lm-evaluation-harness became the de facto open-source runner, supporting hundreds of benchmarks against local Hugging Face models and OpenAI, Anthropic and other APIs, and powering Hugging Face's Open LLM Leaderboard.^[4]

Key Concepts

• Harness — the layer that handles dataset loading, prompt templating, inference, output parsing and metric computation so every model is asked the same question the same way.^[4]
• Static benchmarks — fixed exam-style sets such as MMLU, GPQA Diamond, GSM8K and HumanEval, scored by accuracy or pass@k; prone to saturation and data contamination when they leak into training corpora.
• Agentic and real-work benchmarks — SWE-bench Verified runs models against real GitHub issues with pass/fail CI, and ARC-AGI tests few-shot generalisation rather than recall.^[4]
• Human preference evaluation — pairwise voting on arena platforms such as LMArena produces Elo-style rankings, while LLM-as-a-judge setups use a strong model to grade outputs at scale, introducing their own biases (length, self-preference).
• Beyond accuracy — HELM's multi-metric design exists because accuracy alone hides calibration, fairness and toxicity trade-offs; production teams extend the same idea with task-specific regression suites and golden datasets of real user prompts.^[1]

Known failure modes include benchmark gaming, test-set contamination, leaderboard overfitting and the misleading practice of quoting a single number without naming the harness, prompting and sampling settings used.^[3]

Applications and Impact

Model providers publish benchmark tables for release marketing; enterprises run internal evals before shipping an assistant to customers; and regulators increasingly expect documented evaluation evidence. The EU AI Act and standards such as ISO/IEC 42001 push organisations toward structured testing, documentation and post-deployment monitoring — turning evaluation from a research nicety into a compliance artefact.^[2]

>See Also

• AI Benchmarking
• Data Contamination
• LLM-as-a-Judge
• SWE-Bench
• Model Cards
• EU AI Act