Published October 29, 2026 | Version v1

Test Weakening: Measuring Repair without Eroding Tests

Authors/Creators

  • 1. Independent researcher

Description

A coding agent can make visible tests pass by changing their expected answers. Test Weakening measures whether repair preserves the evidence used to accept it. We study 30 synthetic JavaScript tasks with executable checks for original assertions, held-out behavior, and mutation strength. The study runs 11 scripted policies over 660 episodes including repeats. Reference repair earns 100.0 [88.6, 100.0] percent full success. Expected-answer rewriting earns 100.0 [88.6, 100.0] percent public-test success but 0.0 [0.0, 11.4] percent full success. Brackets are 95 percent Wilson intervals over tasks. A semantics-preserving test reformat earns 0.0 [0.0, 11.4] percent full success, exposing a lexical preservation limit. A separately labelled simulated population sweep compares public-only, original-test, and full acceptance rules. It measures scoring sensitivity under fixed behavior, not changes in learned agents. No learned model, human study, or production deployment is evaluated.

Files

test-weakening-paper.pdf

Files (91.2 kB)

Name Size
md5:f3c564accb999e21c2d907f67a488609
72.1 kB Preview Download
md5:3f37d9e23a1d57a5c366dfc009989ccb
19.1 kB Download

Additional details

Related works