Supported benchmarks

HumanEval¶

Python functions written from their signature and docstring, each checked by its unit tests.

Key	`humaneval`
Category	Coding
Origin	Official
Runs on	Nullius
Tasks	164
Paper	Evaluating Large Language Models Trained on Code, arXiv · 2021
Needs, ports proven	`agent.workspace/v1`

Datasets¶

Dataset	Key	Tasks	From	Verifier
HumanEval	`humaneval`	164	GitHub openai/human-eval	The upstream's check_correctness: the problem's prompt, the agent's completion and the problem's own check(candidate), run as one program.

On the standard¶

Control	Written by
`nop`	Nullius
`oracle`	the authors

Versions¶

Version	Setups as published, whose model	Options
`humaneval`	Original · No agent · Any of your choice	`temperature`

Options¶

Option	Values, default in bold	What it sets	Setups
`temperature`	pattern `default\\|(?:0\\|[1-9][0-9]*)(?:\.[0-9]+)?`, default `default`	Sampling randomness. Provider default sends no temperature; enter a non-negative decimal to override it. Accepted values depend on the model.	`original`

Good to know¶

One attempt per problem gives pass@1. With n attempts per problem, the run reports pass@1 to pass@n. The paper sampled 200.

Upstream limitations¶

Original asks a chat model for the code. The paper's Codex models, used through raw completion, are not available.

Nullius limitations¶

The tests run the agent's code. On the docker backend they run in a throwaway container with no network. On local processes they run on this machine, as you, for 3 seconds at most.
Only solution.py is tested. Other files the agent writes are ignored.
The task given to agents, completing the function in solution.py, is Nullius's wording.

Cite¶

BibTeX

@article{chen2021codex,
  title={Evaluating Large Language Models Trained on Code},
  author={Chen, Mark and Tworek, Jerry and Jun, Heewoo and Yuan, Qiming and Pinto, Henrique Ponde de Oliveira and Kaplan, Jared and others},
  journal={arXiv preprint arXiv:2107.03374},
  year={2021}
}