Supported benchmarks

τ³-bench¶

Customer service with domain tools and a simulated user: airline, retail, telecom, banking.

Key	`tau3-bench`
Category	Customer service
Origin	Official
Runs on	Nullius, Harbor
Tasks	375
Paper	τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment, arXiv · 2025
Needs, ports proven	`agent.tools/v1`, `agent.conversation/v1`

Datasets¶

Dataset	Key	Tasks	From	Verifier
τ³-bench	`tau3-bench`	375	GitHub sierra-research/tau2-bench Harbor hub `sierra-research/tau3-bench`	τ²'s evaluator: the database state and the conversation's assertions.

On the standard¶

Control	Written by
`nop`	Nullius
`oracle`	Nullius

Versions¶

Version	Setups as published, whose model	Options
`tau3-bench`	Original · τ³-bench LLM agent · Like the paper, Like the leaderboard or Any of your choice	`agent_reasoning_effort`, `retrieval`, `task_split`, `upstream_agent`, `user_reasoning_effort`

Options¶

Option	Values, default in bold	What it sets	Setups
`agent_reasoning_effort`	`default`, `none`, `minimal`, `low`, `medium`, `high`, `xhigh`	How much reasoning the benchmark's own agent model uses, with `default` keeping the upstream settings.	`original`
`retrieval`	`bm25`, `bm25_grep`, `grep_only`, `no_knowledge`, `full_kb`, `golden_retrieval`, `openai_embeddings`, `openai_embeddings_grep`, `alltools`, `terminal_use`, `terminal_use_write`	How the agent searches the bank's documents in banking_knowledge tasks.	`original`, `yours`
`task_split`	`base`, `train`, `test`, `small`, `full`	Which group of tasks to run within the selected domain.	`original`, `yours`
`upstream_agent`	`llm_agent`, `llm_agent_gt`, `llm_agent_solo`	Whether the benchmark's own agent runs normally, receives the reference actions, or works alone with the user's tools.	`original`
`user_reasoning_effort`	`default`, `none`, `minimal`, `low`, `medium`, `high`, `xhigh`	How much reasoning the simulated user's model uses, with `default` keeping the upstream settings.	`original`, `yours`

Option	Value	Refused without (`host`: the driver's machine; `embeddings`: a model the provider serves)
`retrieval`	`openai_embeddings`	embeddings: text-embedding-3-large
`retrieval`	`openai_embeddings_grep`	embeddings: text-embedding-3-large
`retrieval`	`alltools`	embeddings: text-embedding-3-large; host: sandbox
`retrieval`	`terminal_use`	host: sandbox
`retrieval`	`terminal_use_write`	host: sandbox

Good to know¶

Scores depend on the simulated user's and the evaluator's models: keep both fixed when comparing.
Conversations run in the upstream's own runner, release v1.0.1, at its defaults: 200 steps, 10 tool errors, seed 300.
Original is the upstream's own agent with your model, as in a leaderboard submission.
Several attempts per task report pass^k, as the leaderboard's 4 trials do. Each attempt is a new conversation.
A task whose retries all fail is an infrastructure error, not a score.
A tool call after the conversation has ended returns an error. An agent that keeps calling until its time runs out is not scored.

Upstream limitations¶

The agent must answer the user through the user tool, or the episode ends early.
Tool calls must be one at a time: the evaluator refuses concurrent ones.

Nullius limitations¶

All five domains run: airline, retail, telecom, telecom-workflow and banking_knowledge. The mock domain and voice mode are not run.
banking_knowledge's search method is an option, bm25 by default. Scores compare only between runs with the same search. The qwen and reranker methods are not offered.
Another agent replaces the upstream's agent. It gets the upstream's instructions and policy, plus two lines from Nullius: where its tools are, and that its last message in a turn goes to the user.

Cite¶

BibTeX

@misc{barres2025tau2,
      title={$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment},
      author={Victor Barres and Honghua Dong and Soham Ray and Xujie Si and Karthik Narasimhan},
      year={2025},
      eprint={2506.07982},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2506.07982},
}

@misc{yao2024tau,
      title={$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains},
      author={Shunyu Yao and Noah Shinn and Pedram Razavi and Karthik Narasimhan},
      year={2024},
      eprint={2406.12045},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2406.12045},
}

@article{shi2026tau,
  title={$\tau$-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge},
  author={Shi, Quan and Zytek, Alexandra and Razavi, Pedram and Narasimhan, Karthik and Barres, Victor},
  journal={arXiv preprint arXiv:2603.04370},
  year={2026}
}