AIWiki
Malaysia
Back to all articles
AI Foundationsswe-benchbenchmarkscoding agents

SWE-bench

4 min readUpdated September 2026
SWE-bench
Type
Software engineering evaluation benchmark
Developed by
Princeton University and collaborators
Released
2023; ICLR 2024 oral paper
Dataset
2,294 real GitHub issue tasks from 12 Python repositories
Variants
SWE-bench Lite, Verified, Multimodal, Pro
Related
SWE-agent, Devin, Claude Code, Code Generation
SWE-bench is a benchmark for evaluating large language models and AI agents on real-world software engineering tasks. Introduced in 2023 by researchers at Princeton University and collaborators, and presented as an oral paper at ICLR 2024, it pairs code repositories with actual GitHub issue descriptions and measures whether an AI system can produce a patch that resolves the issue — success being defined by the project's own tests passing.[1][2]

History

SWE-bench was created by Carlos Jimenez, John Yang, Shunyu Yao and colleagues to address a gap in AI evaluation: earlier coding benchmarks tested isolated programming puzzles rather than realistic maintenance work. Each task is built from a resolved GitHub issue and its accompanying pull request, including the tests that failed before the fix and passed after it. At publication the best models resolved under 2 per cent of tasks, but progress was rapid. In April 2024 the SWE-agent scaffold reached 12.47 per cent, and a containerised Docker harness in June 2024 made evaluation reproducible.[2] In August 2024 OpenAI and the benchmark's authors released SWE-bench Verified, a 500-task subset screened by 93 professional software developers for well-specified issues and fair tests; GPT-4o resolved 33.2 per cent of that set at release.[3] Subsequent versions extended the format to visual applications (SWE-bench Multimodal, ICLR 2025) and to enterprise-grade, long-horizon problems (SWE-bench Pro, released by Scale AI in 2025, where top systems initially solved around 23 per cent of its public set). By July 2025, a lightweight agent called mini-SWE-agent showed that a hundred lines of Python could reach 65 per cent on Verified, and by early 2026 the leading leaderboard results approached 76 to 79 per cent.[4][5][6]

Key Concepts and Technology

A SWE-bench sample contains a problem statement (the issue text), a snapshot of the repository, and hidden test specifications: FAIL_TO_PASS tests that the patch must make pass, and PASS_TO_PASS tests that must keep passing to prove nothing else broke. Agents never see the tests; evaluation runs in containers against the project's real test suite. Because graders execute code rather than compare text, scoring is objective — a task counts as "resolved" only when the tests agree. The public leaderboard standardises agents and reports cost per task in US dollars alongside resolve rates. Known limitations include contamination risk, since the repositories are public and may appear in training data, and the Python-only, English-language scope of the original dataset; Verified and Pro were created partly in response.[1][3][6][7]

Applications and Impact

SWE-bench became the de facto yardstick for the coding-agent era, cited in model launches by major AI laboratories and used to justify enterprise deployment of agents such as Devin, Claude Code and OpenAI Codex. Its trajectory — from under 2 per cent in 2023 to roughly three-quarters of Verified tasks by 2026 — is often cited as evidence of how quickly agentic AI improved on long-horizon work. The benchmark also informs safety policies, because autonomous software engineering is treated as a risk-relevant capability in some frontier model evaluations.[3][5]

>See Also

🇲🇾Malaysian Context

For Malaysian software teams, SWE-bench provides a vendor-neutral way to compare coding assistants before procurement, at a time when local banks, telcos and government digital programmes are piloting AI-assisted development. The datasets are public and free to download, so universities in the Klang Valley and Penang tech ecosystems can use them for research and teaching without licensing fees. The benchmark's Python focus means it underrepresents the legacy Java and .NET systems common in Malaysian enterprises, so teams should treat scores as one signal among several. The same public datasets also let Malaysian AI companies benchmark locally built models against international results, supporting the skills and transparency goals promoted by MDEC and the National AI Office.[2][7]

References

  1. ↑Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. https://openreview.net/forum?id=VTF8yNQM66
  2. ↑SWE-bench. (2026). Overview. https://www.swebench.com/SWE-bench/
  3. ↑OpenAI. (2024). Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/
  4. ↑Yang, J., et al. (2025). SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? ICLR 2025. https://openreview.net/forum?id=riTiq3i21b
  5. ↑SWE-bench. (2026). SWE-bench Leaderboards. https://www.swebench.com/
  6. ↑Deng, X., et al. (2025). SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? https://arxiv.org/abs/2509.16941
  7. ↑Scale AI. (2025). SWE-Bench Pro (Public Dataset). https://labs.scale.com/leaderboard/swe_bench_pro_public
Back to all articles