ESM-BENCH: A Benchmark for Evaluating Whether AI Agents Understand Earth System Model Physics and Code

Can AI coding agents understand not only the syntax of Earth System Model (ESM) code but also the physics it encodes? Recent benchmark work has substantially im- proved evaluation for software engineering, scientific programming, and scientific agents, but those settings rarely test repository-level edits to production ESM code under explicit physical-validity constraints. Climate science, by contrast, demands more than executable syntax: a code change can look plausible as software while still targeting the wrong routine, implementing the wrong physical mechanism, or introducing locally non-physical logic.

We introduce ESM-BENCH, a benchmark that directly probes this physics-and- code understanding gap. ESM-BENCH currently contains 107 tasks and covers 3 open-source ESM repositories, including Noah-MP, CLM5, SUMMA, ParFlow, MOM6, VIC, WRF, and E3SM. Together these repositories represent Fortran 90, Fortran 2008, and C across land-surface, ocean, hydrology, atmosphere, and coupled Earth-system domains. Tasks span 4 scientifically grounded categories: physics-based bug fixes (PBF, n= 39), process representation modifications (PRM, n= 47), parameterization scheme selection (PSS, n= 15), and parameter opti- mization (PO, n= 6). Each task provides (1) a physics-driven problem description; (2) the full source file contents at the parent commit; (3) the ground-truth unified diff from the real developer commit; and (4) review metadata for physics-aware human evaluation, including a six-item rubric. The benchmark is organized as two tracks: Track A tests repository localization (can the agent find the right file?), while Track B tests physics-aware patch synthesis at three oracle-derived hint levels, namely baseline (description + source files), oracle-localized (adds subroutine and line hints), and expert-assisted (adds edit context + Fortran idioms), enabling controlled measurement of whether the bottleneck is code navigation, code generation, or domain knowledge. The central design principle is that physics-aware patch review and exact-match recovery are independent dimensions: an agent can recover the rough intent of a fix without reproducing the exact developer diff, and it can emit a syntactically valid diff while still making a locally non-physical change. ESM-BENCH measures both. Every task carries explicit commit-time provenance metadata, enabling model-specific temporal filtering and contamination audits.