Published November 14, 2026 | Version 1.2.2

StaleBench: data, frozen evaluator, harness and result records

Authors/Creators

  • 1. Causal Dynamics Lab

Description

Release archive for the paper StaleBench: Isolating Stale Retrieval in Agent Memory, and Pricing the Fix (Findings of EACL 2027).

StaleBench isolates stale retrieval in LLM agent memory: an agent that stores facts over time holds several versions of the same fact, and a similarity retriever cannot prefer the current one. The benchmark is built on 892 real updating facts from Wikidata validity intervals via TempLAMA, with currency labels defined by the source and never by a retrieval score.

Contents:

  • release/: the eight benchmark configurations as statement, query and key JSONL for both splits, the alias inventories, a datasheet, a manifest with per-file row counts and MD5s, the frozen scorer evaluate.py, reference baselines, and the cache of every LLM call made for the paper.
  • stalebench/: the harness, retention policies, system runners for Mem0, Graphiti, LangMem and ERASE, probe gates, and the scripts that generate every number and table in the paper.
  • results/: the result records the paper's numbers are generated from.
  • v2/: the builder for the natural alias inventory (Wikidata aliases and Wikipedia redirects) and the inventories the harness reads.

Licences: code under Apache-2.0; data under CC0 1.0, except alias_inventory_natural.json, which includes Wikipedia redirect titles and is CC BY-SA 4.0. See LICENSE and LICENSE-DATA.md.

Version 1.2.2 is identical to the public repository at https://github.com/thenexuslab/stalebench-release and matches the camera-ready paper: the numbers generator regenerates the paper's numbers.tex and every table exactly, and the figure script reproduces its figures. Relative to 1.2.1 only four files change (the two generator scripts, the package version, one README line). Relative to 1.2, the archive also carries the natural alias inventory builder and its outputs, which the harness needs to re-run the natural-alias configurations. The benchmark files under release/ and the result records are unchanged across all versions.

Files

stalebench-1.2.2.zip

Files (12.4 MB)

Name Size
md5:038ca57c7464aff9084f03f23ab7734a
12.4 MB Preview Download

Additional details

Related works