SWE-bench-Live
Evaluating your AI system on latest software engineering tasks.
Paper
GitHub
HuggingFace
About
SWE-bench-Live is the first automatically-updating, multi-language and multi-os SWE task set designed for agentic benchmarking and training. Thanks to our automated dataset curation pipeline, we plan to update SWE-bench-Live on a regular basis to provide the community with up-to-date task instances and support rigorous and contamination-free evaluation.
Note: If you think your repository is not suitable to be included in our benchmark, please contact us to remove it.
News
Leaderboard Update
Added results contrubited by MIT-IBM Agent (BOAD) and Brokk Agent. Added GPT-5.5 and Deepseek-v4-Pro results, evaluated on each language subset after filtering out invalid instances by running regression tests three times.
Multi Language and OS update
We upgraded RepoLaunch Agent to support building repos on all mainstram languages (C C++ C# Python Java Go JS/TS Rust) and on both Linux&Windows platforms. The SWE-bench-Live/MultiLang and SWE-bench-Live/Windows benchmark has been released on HuggingFace. SWE-bench-Live/Windows is aimed at test agents in taking actions in Windows powershell and making Windows-specific code implementation. Through experiments we find none of SWE-agent, OpenHands and ClaudeCode can run on Windows containers, so we implement a minimal Windows-compatible agent with the same tool calls as SWE-agent and OpenHands, named as Win-agent, for benchmarking of LLMs on Windows tasks. On the leaderboard below, Lite, Full and Verified splits are still for Python tasks only.
Dataset update (through Aug 2025)
We will continue to add new llm-verified issues to the dataset. The lite and verified splits will remain frozen, ensuring fair leaderboard comparisons and keeping evaluation costs manageable. To access the latest issues, please refer to the test split!
Leaderboard
| Rank | Method | Resolved ↓ | num_valid_inst ↕ | Date ↕ |
|---|
Loading leaderboard data...
Submit your results
Trajectories of the agents shown above have been archived at SWE-bench-Live/submissions. If you want to submit your success rate with your model-agent combination, please refer to the submission instructions in SWE-bench-Live/submissions and submit a Pull Request to this repository.
Correspondence
Corresponding to SWE-bench-Live@microsoft.com
GitHub Copilot Team, Microsoft US is actively hiring FTE/Interns
DKI Group, Microsoft Shanghai is actively hiring Interns
We welcome external part-time open-source collaborators to join us to update our dataset tasks each month.
Acknowledgement
SWE-bench-Live is built upon the foundation of SWE-bench. We extend our gratitude to the original SWE-bench team for their pioneering work in software engineering evaluation benchmarks.
Citation
If you refer to the SWE task creation pipeline of SWE-bench-Live, or SWE-bench-Live/SWE-bench-Live (Python only tasks), please cite:
@article{zhang2025swebenchgoeslive,
title={SWE-bench Goes Live!},
author={Linghao Zhang and Shilin He and Chaoyun Zhang and Yu Kang and Bowen Li and Chengxing Xie and Junhao Wang and Maoquan Wang and Yufan Huang and Shengyu Fu and Elsie Nallipogu and Qingwei Lin and Yingnong Dang and Saravan Rajmohan and Dongmei Zhang},
journal={arXiv preprint arXiv:2505.23419},
year={2025}
}
If you refer to the automated build and test tool RepoLaunch, SWE benchmarking/training/RFT/RL environment build, SWE-bench-Live/Multi-Language or SWE-bench-Live/Windows, please cite:
@article{li2026repolaunch,
title={RepoLaunch: Automating Build and Management of Code Repositories across Languages and Platforms},
author={Li, Kenan and Li, Rongzhi and Zhang, Linghao and Jin, Qirui and Zhu, Liao and Huang, Xiaosong and Zhang, Geng and Zhang, Yikai and He, Shilin and Xie, Chengxing and others},
journal={arXiv preprint arXiv:2603.05026},
year={2026}
}