Replication Package for "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents"
Authors/Creators
Description
Replication package for the paper "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents" by Tural Mehtiyev, Biruk Tadesse, Lawrence Arkoh, and Wesley Assunção (North Carolina State University).
LLM-based coding agents such as Claude Code read repository instruction files (also called context files or agent configuration files) like AGENTS.md and CLAUDE.md in every session. When the code moves and the file is not updated, the file keeps giving stale instructions, a form of documentation debt. The study examines three real-world stale instructions from open-source repositories (a test command in pyflow, a code location in cog, and a procedure in aghub). It runs 22 maintenance tasks under three versions of each instruction (stale, corrected, removed) with Claude Haiku 5.5 and Claude Sonnet 5.5, and analyzes 732 agent trajectories tool call by tool call. Agents fixed the bug in nearly every run under every version, so task success hid the cost: the stale test command was followed in 78 of 80 stale runs, often ran no tests while appearing to succeed, and cost Claude Sonnet 5.5 about two extra tool calls per run.
Contents: case-search protocol and coding-agent labels; notes on the three studied repositories; the 22 tasks; the experiment protocol with all amendments; transcripts and evaluator outputs of all runs with an attempt ledger; container definitions; scripts for running agents, grading, and analysis; and scripts/reproduce.py, which regenerates the reported numbers, tables, and data figures from the released records. See README.md in the archive.
Files
replication.zip
Additional details
Software
- Programming language
- Python