Nullius

Bring anything. Run it. Prove it.

Bring benchmarks, agents and capabilities into one standard. Run them, and trace every score back to its evidence.

• 7 benchmarks
• 6 agents
• 5 conformance kits

Bring anything

Add it with one prompt.

Connect each component once. Your coding assistant pins the upstream code and records how to install, invoke and check it. Reuse the entry in later recipes.

• Benchmarks → /import-benchmark
• Agents → /import-agent
• Features → /import-feature

Prove it fits

Only pairs that fit are offered.

Kits record what an agent or feature can do. Nullius checks those results, versions and required skills against the benchmark’s requirements.

• OpenCode · agent.workspace/v1 · proven 2/2
• HumanEval needs agent.workspace/v1

Check the benchmark

Benchmarks get checked too.

Check repeatable task generation, exposed reference answers, and the benchmark’s null and reference controls. Each control has its own expected result.

• openai/grade-school-math @ 3101c7d
• nop 0/1319
• oracle 1319/1319
• repeatable task IDs
• reference-string check passed

Run pinned benchmark code with the model or compatible participant you choose. This alone does not reproduce the paper’s experiment.

Run it anywhere

One machine, a host, or many.

Run locally or over SSH. Split work across hosts and merge the results. Standard runs record configured model calls through a per-unit relay.

• --shard 1/3
• docker · one container per unit
• the key stays in the relay

Read it, then prove the change

See what failed, and why.

Open recorded conversations, model calls and verifier evidence. Separate infrastructure failures, then compare the same tasks across runs.

• Illustrative run
• JiuwenSwarm × HumanEval

Read existing Harbor jobs, ATIF trajectories and OpenTelemetry exports.

Judge it

Ask the trials a question.

Ask a judge about selected trials. Its answers sit beside the evidence and never change benchmark scores.

• task_completed
• failure_cause
• 7 behaviour flags

Use your configured judge provider and model.