Published November 6, 2026 | Version 1.0.0

Replication Package for "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents"

  • 1. ROR icon North Carolina State University

Description

Replication package for the paper "When Instruction Files Go Stale: Documentation Debt and Hidden Cost in Coding Agents" by Tural Mehtiyev, Biruk Tadesse, Lawrence Arkoh, and Wesley Assunção (North Carolina State University).

LLM-based coding agents such as Claude Code read repository instruction files (also called context files or agent configuration files) like AGENTS.md and CLAUDE.md in every session. When the code moves and the file is not updated, the file keeps giving stale instructions, a form of documentation debt. The study examines three real-world stale instructions from open-source repositories (a test command in pyflow, a code location in cog, and a procedure in aghub). It runs 22 maintenance tasks under three versions of each instruction (stale, corrected, removed) with Claude Haiku 5.5 and Claude Sonnet 5.5, and analyzes 732 agent trajectories tool call by tool call. Agents fixed the bug in nearly every run under every version, so task success hid the cost: the stale test command was followed in 78 of 80 stale runs, often ran no tests while appearing to succeed, and cost Claude Sonnet 5.5 about two extra tool calls per run.


Contents: case-search protocol and coding-agent labels; notes on the three studied repositories; the 22 tasks; the experiment protocol with all amendments; transcripts and evaluator outputs of all runs with an attempt ledger; container definitions; scripts for running agents, grading, and analysis; and scripts/reproduce.py, which regenerates the reported numbers, tables, and data figures from the released records. See README.md in the archive.

Files

replication.zip

Files (31.6 MB)

Name Size
md5:dd7e446160c8eeae865145c3d67cdd00
31.6 MB Preview Download

Additional details

Software

Programming language
Python