Supported benchmarks

SWE-bench Verified¶

Software engineering: fix a real GitHub issue so the failing tests pass.

Key	`swe-bench-verified`
Category	Coding
Origin	Official
Runs on	Nullius, Harbor
Tasks	500
Paper	SWE-bench: Can Language Models Resolve Real-world Github Issues?, ICLR · 2024
Needs, ports proven	`agent.workspace/v1`

Datasets¶

Dataset	Key	Tasks	From	Verifier
SWE-bench Verified	`swe-bench-verified`	500	Hugging Face SWE-bench/SWE-bench_Verified Harbor registry `swebench-verified@1.0`	The failing tests must pass and the passing ones keep passing.

On the standard¶

Control	Written by
`nop`	Nullius
`oracle`	the authors

Versions¶

Version	Setups as published, whose model	Options
`swe-bench-verified`	none	none

As published¶

Harbor agent, among the builder's	Model	What it is
`swe-agent`	any	SWE-agent, the SWE-bench authors' own agent and the paper's agent baseline, which Harbor registers; a model of your choice runs it.

Upstream limitations¶

django__django-10097 passes with no change: the tests it grades pass without the fix, and the URL tests its fix is for are not among them.

Nullius limitations¶

On a shared API key, rate limits cause most failures: run few trials at once.
gpt-5.x models need the Responses API to reason while using tools.
On Nullius, the agent works in the instance's own container and the tests run there after it. Test files are reset first, so editing them does not help.
On Nullius, the agent can reach only its model. The tests get internet access after the agent ends.
The prompt is the issue's title and body, without the hints, as on the leaderboard.
Each instance's image is pinned. An image the authors rebuild is not used until the pin is refreshed.
SWE-agent, the authors' own agent, runs only on Harbor.

Cite¶

BibTeX

@inproceedings{
    jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}