Skip to content
CLI

Run a benchmark

One benchmark and one agent, capability, control or setup, into a job.

Run
nullius standard run <benchmark>[@version] <other> [--model M] [--dataset D] [--limit N] [-j N] \
    [--unsandboxed | --backend docker] [--option NAME=VALUE]... [--bench-option NAME=VALUE]... \
    [--repeat K] [--remove-images] [--judge-model M] [--simulator-model M] [--name NAME] [--jobs DIR] [--provider P]
Examples
nullius standard run humaneval reference --model gpt-5-mini --limit 2 --unsandboxed
nullius standard run humaneval oracle --limit 6
nullius standard run locomo original --model gpt-3.5-turbo-4k
nullius standard run locomo model --model gpt-4o-mini
Argument Meaning
<benchmark> KEY or KEY@VERSION; a fork is a version
<other> KEY or KEY@VERSION; nop or oracle; original or model
--model None for a control; original defaults to the paper's first
--provider anthropic, custom, openai or openrouter
--dataset, --limit, -j One dataset, the first N units, N at a time
--unsandboxed An agent runs as you; else --backend docker
--backend local (default) or docker, each unit in a container with its relay
--option NAME=VALUE An agent's or capability's option (repeatable)
--bench-option NAME=VALUE A benchmark's option (repeatable), read as NULLIUS_OPTION_<NAME>
--repeat K Each unit K times; result.json carries pass_hat_k
--remove-images Remove each image the run pulls or builds when its last unit ends (never with --repeat); by default they stay for the next run, listed and removed in Settings, Sandbox
--judge-model Instead of the official judge
--simulator-model Instead of the official simulated user
--name The job's name
--jobs Where jobs are written (default jobs_standard/ relative to the current directory)
--from-run Started by Nullius: OPENAI_API_KEY and OPENAI_BASE_URL only, saved settings unread
--shard I/N, --launch One shard of a launch: Shard and merge
Rule Detail
The pair must fit Else refused, naming the missing port
Keys Environment, then user settings.json; the agent never gets one
A control No --model, no --unsandboxed; only a judge is called
Machine programs An option needing them is refused without them
Progress stderr, one line per stage ([nullius] stage=install ..., stage=units n=10, stage=done, stage=refused), also progress.jsonl
The job Opens in the app; events.jsonl feeds the live board