Patching the Protocol: A Taxonomy of Pre-Registration Repairs in LLM-Proposer Experiments
Description
Pre-registration is increasingly proposed as a trust mechanism for machine-learning research involving LLM-generated content, but its canonical form assumes that a frozen protocol can be executed unchanged. Software-intensive studies violate this assumption: the protocol is partly implemented, and implementations carry defects. We report a three-case study of protocol repairs (``patches'') from a research program in which an LLM proposes executable decision rules and a programmatic verifier gates them. In three domains –- production scheduling, incentive mechanism design, and task assignment –- the protocol required exactly one substantive repair each, and no repair touched a judgment criterion, threshold, or collected data. We propose a two-axis taxonomy of patches (defect class: environmental / feasibility / correctness x timing: pre-data / post-data), introduce the notion of a verifiable non-impact claim for post-data repairs, and distill a nine-item checklist for disciplined protocol repair. We argue that the credibility of automated-research pipelines depends less on whether patches occur than on whether they are auditable.