Supported benchmarks

LoCoMo

Very long-term conversational memory over conversations of 19 to 32 sessions.

Key	locomo
Category	Memory
Origin	Official
Runs on	Nullius
Tasks	10
Paper	Evaluating very long-term conversational memory of llm agents, arXiv · 2024
Needs, ports proven	cap.memory/v1

Datasets

Dataset	Key	Tasks	From	Verifier
LoCoMo-10	locomo10	10	GitHub snap-research/locomo	LoCoMo's own task_eval/evaluate_qa.py: the token F1 of each answer against the reference; no judge model.

On the standard

Control	Written by
nop	Nullius
oracle	Nullius

Versions

Version	Setups as published, whose model	Options
locomo	Original · Original memory · Like the paper or Any of your choice	rag_mode, retrieval, retriever, top_k

Options

Option	Values, default in bold	What it sets	Setups
rag_mode	dialog, observation, summary	Whether the memory stores conversation turns, extracted facts, or session summaries.	yours
retrieval	none, dialog, observation, summary	Whether the model receives the whole conversation or retrieves relevant turns, extracted facts, or session summaries.	original, model
retriever	dragon, dpr	The embedding model used to find relevant entries when retrieval is enabled.	original, model
top_k	2, 5, 10, 25, 50	The maximum number of entries the model receives per question when retrieval is enabled.	original, model

Option	Value	Refused without (host: the driver's machine; embeddings: a model the provider serves)
retrieval	dialog	host: cuda_gpu
retrieval	observation	host: cuda_gpu
retrieval	summary	host: cuda_gpu

Good to know

• Each conversation is one task, with its questions inside. The smallest run is one conversation.
• The score is the mean token F1 over the questions, as in the paper. There is no pass or fail.
• What goes into the memory is an option: the conversation itself by default, or the authors' released facts or session summaries.

Upstream limitations

• The 446 adversarial questions are left out, because the published file has no answer for them. 1,540 of the 1,986 questions remain.
• The reader's model name must contain gpt-4 or gpt-3.5, as LoCoMo's own code requires. Any other model fails every question.
• The paper's readers, gpt-4-turbo and gpt-3.5-turbo, now point to newer models. Scores can differ from the paper's for that reason.
• The reader gets 32 tokens per answer. A reasoning model can spend them all reasoning, and that question fails.
• One unreadable reply stops the whole conversation, because LoCoMo's script writes its answers only at the end.
• Original memory runs the paper's retrieval setup and needs a CUDA GPU. It writes facts and summaries anew each run, so they differ from the authors' released ones.

Nullius limitations

• The paper's other readers are not offered. Claude 3 Sonnet and Gemini Pro 1.0 are retired, and the Llama 3 and Mistral readers run only as local weights.

Cite

BibTeX

@article{maharana2024evaluating,
  title={Evaluating very long-term conversational memory of llm agents},
  author={Maharana, Adyasha and Lee, Dong-Ho and Tulyakov, Sergey and Bansal, Mohit and Barbieri, Francesco and Fang, Yuwei},
  journal={arXiv preprint arXiv:2402.17753},
  year={2024}
}