Supported benchmarks

Supported benchmarks

Every benchmark in the catalogue: where it runs, what it needs, the ways it runs.

Read the controls before any agent score

A passing nop or a failing oracle means the dataset's verifier cannot be trusted; see Read results.

Available benchmarks

Benchmark	What it is	Category	Runs on		Tasks	Origin
			Nullius	Harbor
GSM8K gsm8k	Grade-school math word problems that take two to eight steps of basic arithmetic, each with a single numeric answer.	Math	✓	–	1319	Official
HumanEval humaneval	Python functions written from their signature and docstring, each checked by its unit tests.	Coding	✓	–	164	Official
LoCoMo locomo	Very long-term conversational memory over conversations of 19 to 32 sessions.	Memory	✓	–	10	Official
LongMemEval longmemeval	Long-term chat memory: questions over timestamped chat histories.	Memory	✓	–	1500	Official
SWE-bench Verified swe-bench-verified	Software engineering: fix a real GitHub issue so the failing tests pass.	Coding	✓	✓	500	Official
τ³-bench tau3-bench	Customer service with domain tools and a simulated user: airline, retail, telecom, banking.	Customer service	✓	✓	375	Official
Terminal-Bench 2.0 terminal-bench	Real terminal tasks in a container: build, configure, debug.	Terminal	✓	✓	89	Official

How each runs

Benchmark	Needs, ports proven	Controls, written by		Runs with, any model
		nop	oracle	your agent or capability	as published	as published, your model
GSM8K	agent.task/v1	Nullius	the authors	✓	✓	✓
HumanEval	agent.workspace/v1	Nullius	the authors	✓	✓	✓
LoCoMo	cap.memory/v1	Nullius	Nullius	✓	✓	✓
LongMemEval	cap.memory/v1	Nullius	Nullius	✓	✓	✓
SWE-bench Verified	agent.workspace/v1	Nullius	the authors	✓	✓	✓
τ³-bench	agent.tools/v1, agent.conversation/v1	Nullius	Nullius	✓	✓	✓
Terminal-Bench 2.0	agent.workspace/v1	Nullius	the authors	✓	✓	✓

• Nullius wrote a control where the upstream published none.
• As published: the upstream's own answering program (on Harbor, its own agent) with the models its paper fixes; your model, where the paper's are not served.

• Add a benchmark manually

  A Harbor entry from the form, or a standard definition and its driver.

• Add a benchmark with an assistant

  One prompt, the /import-benchmark skill.

• Controls and proofs

  What cli prove checks and records.