From Black Box to Toolbox: Human-Centered Explainability in Developer-Authored Agent Rules

As AI agents permeate software development, developers are embracing markdown-based rule files (e.g., AGENTS.md) to provide the organizational context that general-purpose models often lack. This emerging practice, nonetheless, is undermined by a critical explainability challenge. Through a qualitative study of 12 professional software developers who author agent rules for their respective projects, we found that developers lack a reliable way to verify whether an agent consistently follow specific rules. Further more, they struggle to attribute observable agent behaviors to individual instructions in agent rule files. Instead, they often rely on subjective "vibe checks" to test these rules. Our findings also highlight a lack of centralized governance of agent rules for common libraries, leading to redundant work and the erosion of a shared understanding about agent behavior. We argue for a transition toward rigorous rule traceability and library-level governance to transform agent customization from an ad-hoc craft into an evidence-based practice.