Supported benchmarks

LongMemEval

Long-term chat memory: questions over timestamped chat histories.

Key	longmemeval
Category	Memory
Origin	Official
Runs on	Nullius
Tasks	1500
Paper	LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory, arXiv · 2024
Needs, ports proven	cap.memory/v1

Datasets

Dataset	Key	Tasks	From	Verifier
LongMemEval-S (cleaned)	longmemeval-s	500	Hugging Face xiaowu0162/longmemeval-cleaned	LongMemEval's own judge, GPT-4o (gpt-4o-2024-08-06): yes or no per answer.
LongMemEval-M (cleaned)	longmemeval-m	500	Hugging Face xiaowu0162/longmemeval-cleaned	LongMemEval's own judge, GPT-4o (gpt-4o-2024-08-06): yes or no per answer.
LongMemEval oracle · evidence sessions only	longmemeval-oracle	500	Hugging Face xiaowu0162/longmemeval-cleaned	LongMemEval's own judge, GPT-4o (gpt-4o-2024-08-06): yes or no per answer.

On the standard

Control	Written by
nop	Nullius
oracle	Nullius

Versions

Version	Setups as published, whose model	Options
longmemeval	Original · Original memory · Like the paper or Any of your choice	history_format, merge_key_expansion, reading, retriever, separate_notes, topk, useronly

Options

Option	Values, default in bold	What it sets	Setups
history_format	json, nl	Whether conversation history is shown to the model as JSON or as lines of text labelled by speaker.	original, model
merge_key_expansion	none, merge, replace	Whether retrieved conversations are shown in full, replaced by their user text, or combined with that text.	original, model
reading	con, direct	Whether the model answers from recalled memory directly or first identifies relevant facts and reasons through them.	original, model, yours
retriever	full-history-session, full-history-turn, no-retrieval, flat-bm25-session, flat-bm25-turn, flat-contriever-session, flat-contriever-turn, flat-stella-session, flat-stella-turn	Whether the model receives the full conversation history, no history, or sessions or turns selected by a retrieval method.	original, model
separate_notes	false, true	Write a separate note for each retrieved session or turn before reasoning. Original memory only.	original, model
topk	1, 3, 5, 10, 20, 30, 50, 100, 1000	The maximum number of conversation sessions or turns the model receives when answering a question.	original, model
useronly	false, true	Whether conversation history includes only user messages or both user and assistant messages.	original, model

Option	Value	Refused without (host: the driver's machine; embeddings: a model the provider serves)
retriever	flat-contriever-session	host: cuda_gpu
retriever	flat-contriever-turn	host: cuda_gpu
retriever	flat-stella-session	host: cuda_gpu
retriever	flat-stella-turn	host: cuda_gpu

Good to know

• The S, M and oracle files run cleaned, and S and M also as first released. The paper's numbers come from the files as first released.
• An M file is 2.7 GB and needs about 5.5 GB of free disk. STANDARD_DATA_DIR moves the datasets to a disk with room.
• Reading is chain-of-note by default, or direct. Separate notes work with Original memory only.
• Original memory offers the paper's Contriever and Stella retrievers and needs a CUDA GPU. Stella needs more than 11 GB of GPU memory.

Upstream limitations

• Scores compare with the paper only with its reader, GPT-4o or GPT-4o-mini, and its judge, GPT-4o.
• A reader you choose must take about 126,200 tokens. The history is not cut to fit, so a longer recall fails at the model.
• An empty reply from the reader fails the question.
• Three combinations fail as published: merge expansion with a full-history retriever, user-only with flat-bm25-turn, and separate notes with chain-of-note reading and nl history.
• The judge, a model, rejects about one right answer in twenty, and passes an empty answer now and then.

Nullius limitations

• The GTE retriever, the index and query expansions, and the open-weight readers are not offered.

Cite

BibTeX

@article{wu2024longmemeval,
      title={LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory},
      author={Di Wu and Hongwei Wang and Wenhao Yu and Yuwei Zhang and Kai-Wei Chang and Dong Yu},
      year={2024},
      eprint={2410.10813},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.10813},
}