Metrics

Metric field guide¶

Read a result, inspect its evidence, and compare runs on the same terms.

Read a result¶

Did it run?¶

Check the execution status and run log first. A failed launch is not a benchmark score. A stopped or failed run may contain only some planned trials.

What was measured?¶

Use solve rates for pass/fail benchmarks and mean score for graded ones. No verdict means no solve rate, not 0%.

Why that result?¶

Open a trial for the verifier, answer, conversation and recorded model calls. Tool activity alone does not prove correctness.

Controls and port proofs¶

Benchmarks measure performance. Kits check that an agent or capability implements a port.

Proven: 2/2. Passed basic checks out of required basic checks. All must pass to prove the port. Skills are additional checks, required only by benchmarks that ask for them. Open the kit to inspect its checks and shared source.
Original procedure, chosen participant and model. The builder lists reference agents first, or No agent · Original for direct model evaluation. Memory benchmarks offer Original memory alongside compatible capabilities. Like the paper filters documented models and keeps the official user simulator and judge; Like the leaderboard, where a benchmark declares one, runs its simulator, judge, options and attempts; Any of your choice opens the provider catalogue.
Native and combined setups. A capability's proof does not prove an agent uses it. Native, augmented and replacement setups need separate evidence. A replacement must disable or redirect the native path; an extra tool alone is not a swap. General agent-plus-capability recipes are not supported yet.
Null and reference controls. The null control leaves the task unanswered; the reference uses the benchmark's declared answer or solution. Inspect both before trusting a new benchmark's scores. Control runs stay out of rankings.
Isolation matters. Local agents run as your user and can read accessible scorer files. Docker conformance keeps the driver and oracle outside the agent's mounts. Public kit checks remain compatibility checks, not evidence of resistance to cheating.

Repeated attempts¶

Standard runs with repeated, scored attempts record both measures in pass_hat_k. They answer different questions.

pass@k¶

Chance that at least one of k sampled attempts passes: 1 − C(n−c,k) / C(n,k).

pass^k¶

Chance that all k sampled attempts pass: C(c,k) / C(n,k).

Here n is the attempt count and c the passed count per unit; results are averaged across eligible units. Units with an exception or missing verdict are excluded. The writer uses the smallest available attempt count across eligible units and emits no series below two attempts. This is separate from the trial-level adjusted solve rate.

Compare like with like¶

Whole runs include all their trials. Matched tasks use the intersection of recorded dataset and task identities, without guessing aliases. Every attempt on a matched task contributes to rates, tokens and costs; its comparison row shows the last indexed attempt.

Check dataset, model, provider, agent version, limits and execution backend before attributing a difference to the agent. Matched views omit whole-run wall time because it does not measure just the selected tasks.

Reading the provenance column¶

Each field is marked by how Nullius obtains it. Missing evidence is not a zero.

Marker	Means	Fails by
measured	Read verbatim from `result.json`, the trajectory or the trace.	Going missing: you see —, not a wrong value.
derived	Arithmetic over measured values. Deterministic, no guessing.	Compounding an upstream gap silently.
inferred	A heuristic: a regex, a keyword rule or a pattern over free text.	Being confidently wrong.
unavailable	The run never recorded it, and the app declines to invent it.	Nothing. Absence is shown as absence.

Reading the portability column¶

Any Harbor run reads as is; a run from another harness once an importer has converted it. A run of the standard is written in Harbor's shape by Nullius itself: its tokens, cost and gateway log from the relay's record of each model call, its verdict from the benchmark's own scorer. A metric that needs one dataset's fields is marked.

Marker	Means	Reads from
any	Uses common run fields when the producer records them; availability still varies.	Harness files: `result.json` (its reward, exception and `agent_result`), `config.json`, `trajectory.json`, `trace.json`.
mixed	Part of the rule is harness-level, part needs a per-dataset reader.	`outcome`: the exception branch travels, the failure detail does not. `f2p`: filled only for a benchmark whose entry describes its report (SWE-bench, Terminal-Bench).
swe-bench	Needs fields only SWE-bench emits. Blank elsewhere.	`verifier/report.json` (`resolved`, the patch flags, `FAIL_TO_PASS` / `PASS_TO_PASS`) and the gold patch.

The two solve rates travel. Both count outcome == resolved: SWE-bench from its report, every other dataset from verifier_result.rewards.reward, and only once something in verifier/ proves the verifier ran.

Where the numbers come from¶

Layer 0 · import

Harness¶

nullius/importers/

A run Harbor never wrote (ATIF trajectories, an OpenTelemetry export), converted into a job directory first. On such rows the importer decides what a number is. A standard run is written in Harbor's shape by Nullius itself (nullius/standard/job.py).

Layer 1 · run

Job¶

nullius/runs/parse.py

One benchmark invocation. Aggregates its trials; owns solve rates and totals.

Layer 2 · task

Trial¶

nullius/runs/parse.py

One agent against one task. Outcome, tests, phase timings, tokens, cost.

Layer 3 · transcript

Trajectory¶

nullius/traces/trajectory.py

The ATIF step log: messages, tool calls, observations, per-step metrics.

Layer 4 · clocks

Trace¶

nullius/traces/trace.py

An optional sidecar with a start and end per call, the only per-span timing, present when the run captured one.

Layer 5 · conduct

Behavior¶

nullius/traces/behavior.py

How the agent worked, not whether it won. Computed over layer 3.

Job metrics, one row per run¶

Field	What it is	Source	Works on
`raw_solve_rate`	Resolved ÷ all trials: Harbor's headline, which counts infra failures as misses.	derived	any
`adjusted_solve_rate`	Resolved ÷ non-infrastructure trials. Only rate limits, API errors and setup errors leave the denominator; unknowns remain. Read alongside the raw rate and infrastructure count.	derived	any
`n_infra`	Trials removed from the adjusted denominator as infrastructure failures.	derived	any
`mean_score`	The mean graded score, weighted by each trial's number of scored parts when present. When no trial passes or fails, it is the headline and the rates read —.	derived	mixed
`score_by_group · parts_by`	The mean score per value of the field the benchmark breaks it down by (a question's category), from `verifier/parts.json`.	derived	mixed
`n_regressions`	Trials whose patch broke a previously passing test.	derived	swe-bench
`counts`	Trials per outcome, as a stacked bar; on the run page each segment filters the trials.	derived	any
`total_input_tokens`	Prompt tokens across the run.	derived	any
`total_cache_tokens`	The part of those served from cache, as each trial's `result.json` records it.	derived	any
`total_output_tokens`	Completion tokens.	derived	any
`total_cost_usd`	The trials' recorded costs, summed: an estimate, not a bill. Shows — when no trial recorded one.	derived	any
`duration_min`	The whole run, start to finish.	derived	any
`agent_label`	The agent's name: the class of a `module:Class` path, or the trials' own name when the config gives none.	measured	any
`agent_version`	The version the trial's `agent_info` reports.	measured	any
`is_control`	An `oracle` or `nop` run: a check on the tasks, kept out of rankings and totals. Its row reads as a verdict, not a rate. A standard run is one only when its benchmark declares that control. An imported runner's arm without that declaration is not a control. On a benchmark that scores, a control's score of 1 reads as accepted and 0 as rejected.	measured	any
`is_running`	The job records running or pending trials. Numbers are partial; a managed execution's terminal status takes precedence on the page.	measured	any
`n_planned / n_retries`	Planned trials from the run record or task count, and retries recorded by the harness. Repeated attempts are separate trials, not retries.	measured	any
`dataset`	The dataset folder, `name@version` for a registry fetch, or the task names' shared prefix.	measured	any

Trial metrics, one row per task attempt¶

Field	What it is	Source	Works on
`outcome`	One of eleven classes: from the exception if there was one, else from the verifier's verdict. Full vocabulary below.	inferred	mixed
`score · n_parts`	A graded score the benchmark gave, and how many parts it averages. With no reward and no exception, the outcome is Scored.	measured	mixed
`reward`	The verifier's raw reward. Where no report says more, 1.0 or above is a solve.	measured	any
`f2p_pass / f2p_fail`	Checks that had to start passing: SWE-bench's FAIL_TO_PASS, or every test of a Terminal-Bench task.	measured	mixed
`p2p_pass / p2p_fail`	PASS_TO_PASS: checks that had to keep passing, the regression guard.	measured	swe-bench
`is_regression`	The patch broke a test that passed before it.	derived	swe-bench
`task_broken`	On the feasibility page: `nop`, which changes nothing, failed passing tests: the task is broken as shipped, or its verifier unstable.	derived	swe-bench
`patch_exists / patch_applied`	Whether a patch was produced, and whether it applied.	measured	swe-bench
`dur_total`	Trial wall clock.	measured	any
`dur_env / dur_agent_setup / dur_agent / dur_verifier`	The four phases Harbor records, drawn in order. An imported trial charges its whole window to `dur_agent`.	measured	any
`dur_other`	`dur_total` minus the phases: on Harbor, image pull, container start and teardown; on an imported script run, the wait for the judge; on a standard run, the unit's setup, the benchmark's driver and its scorer.	derived	any
`n_input_tokens`	Prompt tokens for the trial.	measured	any
`n_cache_tokens`	The cached part of the prompt tokens: a subset, never added to them.	measured	any
`cost_usd`	The cost `result.json` records: on Harbor, priced by the agent from its own table; on a standard run, the relay's record: the provider's own figure where its reply carried one, else an estimate from a price table. A benchmark option's embeddings (a search index its driver builds) are the run's, in `.work/relay-benchmark`, never a trial's. None recorded reads —, never `$0.00`.	derived	any
`n_steps / n_tool_calls`	Trajectory steps and tool calls; empty placeholder calls are not counted.	measured	any
`exception_type`	What ended the trial, if anything. It decides the infra / agent split.	measured	any
`n_rate_limited_retries`	429s counted in `agent/gateway.log`: on Harbor, the ones the agent's SDK retried and survived; on a standard run, every model call the relay saw refused with 429. No log reads not recorded, never 0.	inferred	any

Trajectory metrics, inside a single run¶

Field	What it is	Source	Works on
`calls[].name`	The tool invoked, verbatim. Not comparable across agents: each names its tools its own way.	measured	any
`calls[].category`	Plan, search, read, edit, execute or other, matched from the tool name; for a benchmark's MCP tool, lookup or change, from its first word. Compare on this, never on the name.	inferred	any
`calls[].target`	The call's subject: a named argument, or the paths inside a patch payload.	inferred	any
`calls[].failure`	Failure text spotted in the observation.	inferred	any
`category_freq`	The run's tool mix, as chips by category.	inferred	any
`name_freq`	The tool names behind each category, in the agent's own words. Evidence of what a category folded, never a comparison key.	measured	any
`files_touched`	Files the agent edited, made workspace-relative so two agents' sets compare.	inferred	any
`n_failed_calls`	Tool calls that returned an error.	inferred	any
`token_series`	Input tokens per LLM call, with the cached share shaded, from per-step metrics. A single call is drawn as one usage bar.	measured	any
`cache`	Cache reads ÷ every input token. Anthropic-style usage has its writes rebuilt from prompt growth; the file's own ratio shows when it differs.	inferred	any
`steps[].dur`	Seconds until the next step opened, shown on the selected step in the conversation.	derived	any
`has_step_clock`	False when the step timestamps span under a second. Step durations and time to first edit are then left out, not shown as zeros.	derived	any
`total_reasoning_tokens`	`None` when nothing reports it: "not reported" and "zero reasoning" are different claims.	measured	any
`last_agent_message`	The agent's closing message, behind a toggle in the task comparison.	measured	any

Trace metrics, per-span clocks when a capture exists¶

The one place with a start and end per call. Spans are measured by instrumentation, or rebuilt from a transcript and marked inferred. Without a trace the trial page does not draw this.

Field	What it is	Source	Works on
`span.start_ns / end_ns`	One call's extent. Bars sit on one global axis, so a child starting before its parent is shown, not clamped.	measured	any
`span.duration_ns`	`None` means no known end, not a zero.	measured	any
`span.kind`	One of nine: `llm_request`, `reasoning`, `output`, `tool`, `subagent`, `session`, `user_message`, `compaction`, `group`. It colours the bar and the legend.	measured	any
`kind group`	A span that only brackets a same-named child, re-kinded on load so one call draws as one row. Charged no time.	inferred	any
`span.timing_quality`	`measured`, `inferred` or `unknown`. A trace with nothing measured is labelled as such, not dropped.	measured	any
`totals`	Displayed verbatim, never recomputed. Spans nest, so summing them double-counts; the producer already resolved that.	measured	any
`row.left_pct / width_pct`	A bar's position and width as a share of the axis: the whole trace, or the zoomed span.	derived	any
`hide / hide_span`	Labels and single calls left out, each with its subtree. In the query string, so a view is a link and is what a save writes.	measured	any
`charts / Bar`	Span durations on log-spaced bins, by count and by summed span time, split by kind. A share of span time, never of the run.	derived	any

Behaviour metrics, how the agent worked¶

Computed here over the trajectory's tool calls, when present. They describe conduct, not correctness.

Field	What it is	Source	Works on
`first_edit_call`	The first call that changed anything; the calls before it were exploration.	inferred	any
`time_to_first_edit`	The same, in seconds, only when the step clock is usable.	derived	any
`n_redundant_reads`	Reads or searches repeating an exact earlier one.	inferred	any
`n_repeated_failed_calls`	Calls repeating a tool and target that had already failed.	inferred	any
`edit_test_cycles`	Edits followed by running something.	inferred	any
`exploration_per_edit`	Search and read calls per edit.	derived	any
`localization.hit / missed / extra`	Files edited against the gold patch's files. Missed is the fix it never went near.	inferred	swe-bench

Outcome vocabulary¶

Eleven classes. The three marked yes under Infrastructure leave the adjusted solve rate's denominator.

Outcome	Label	Infrastructure	Meaning
`resolved`	Resolved	no	The verifier accepted the attempt.
`scored`	Scored	no	A score and no pass or fail: a token F1, partial credit. The score is the result; such a run has no solve rate.
`patch_failed_tests`	Patch failed tests	no	Applied cleanly, tests still failed.
`failed_verification`	Failed verification	no	The verifier said no, on a benchmark with no patch detail: Terminal-Bench, or one read only by its reward.
`patch_not_applied`	Patch did not apply	no	Malformed context lines, or edits against the wrong revision.
`no_patch`	No patch produced	no	Finished without changing anything.
`agent_error`	Agent error	no	Timed out, filled its context, truncated its output, refused, or exited non-zero. A result, counted against the agent.
`rate_limited`	Rate limited	yes	An HTTP 429 ended the trial.
`api_error`	API error	yes	Any other provider failure: overloaded, unreachable, out of quota, key rejected.
`setup_error`	Setup error	yes	The environment or image never came up, or the unit was not run for want of a secret or docker it needs, so the agent never ran.
`unknown`	Unknown	no	No verdict, or an exception on neither list. Counted, charged to nobody; the trial page shows the recorded type.

Caveats¶

Cost is an estimate, and only where one was recorded. A Harbor agent prices its own usage; a standard run's relay keeps the provider's figure when the reply carries one and estimates otherwise. No bill is read. A trial without a cost shows —, and a run's total covers only the trials that have one.
The infra/agent split is an explicit list. Which exceptions leave the adjusted denominator and which count against the agent is pinned in classify.py and tested against Harbor's own classes; anything else is Unknown.
Tool names don't compare across agents. One writes bash, another a whole sentence. Compare on category.
A flat step clock hides a run's interior. When every step carries the same timestamp, the app leaves step durations out rather than drawing zeros.
Large evidence loads in parts. Span detail and conversation content arrive in bounded chunks.
Trace totals are the producer's, not ours. Spans nest, so summing them double-counts.
The verifier has to be able to say no. A nop run scoring 0 is what makes a passing score mean something. Check it on any new dataset with the builder's controls.