Supported benchmarks

GSM8K¶

Grade-school math word problems that take two to eight steps of basic arithmetic, each with a single numeric answer.

Key	`gsm8k`
Category	Math
Origin	Official
Runs on	Nullius
Tasks	1319
Paper	Training Verifiers to Solve Math Word Problems, arXiv · 2021
Needs, ports proven	`agent.task/v1`

Datasets¶

Dataset	Key	Tasks	From	Verifier
GSM8K test	`test`	1319	GitHub openai/grade-school-math	The upstream's extract_answer and is_correct, applied to the agent's last message and the problem's reference answer.

On the standard¶

Control	Written by
`nop`	Nullius
`oracle`	the authors

Versions¶

Version	Setups as published, whose model	Options
`gsm8k`	Original · No agent · Any of your choice	`temperature`

Options¶

Option	Values, default in bold	What it sets	Setups
`temperature`	pattern `default\\|(?:0\\|[1-9][0-9]*)(?:\.[0-9]+)?`, default `default`	Sampling randomness. Provider default sends no temperature; enter a non-negative decimal to override it. Accepted values depend on the model.	`original`

Good to know¶

Answers must end with a line '#### '. The number is compared as text, so 18.00 does not match 18.
The test split is run. The Socratic variant has the same questions and answers, with different worked explanations.

Nullius limitations¶

The prompt asking for the '#### ' line is Nullius's wording. The paper's models were fine-tuned to write it.
The paper's trained verifier, which picks the best of several answers, is not reproduced. Several attempts per problem report pass@k instead.

Cite¶

BibTeX

@article{cobbe2021gsm8k,
  title={Training Verifiers to Solve Math Word Problems},
  author={Cobbe, Karl and Kosaraju, Vineet and Bavarian, Mohammad and Chen, Mark and Jun, Heewoo and Kaiser, Lukasz and Plappert, Matthias and Tworek, Jerry and Hilton, Jacob and Nakano, Reiichiro and Hesse, Christopher and Schulman, John},
  journal={arXiv preprint arXiv:2110.14168},
  year={2021}
}