Does the Train-Gain/Generalization Relation Transfer? A Pre-Registered Cross-Domain Test of Inverted-U Behavior in Verifier-Bound Rule Discovery
Description
A verifier-bound rule-discovery engine –- an LLM proposes executable rules, a programmatic verifier gates them, a single-shot isolated holdout scores them –- was run in three domains under one pre-registered methodology. The first domain (combinatorial scheduling) reported an inverted-U relation between training improvement and holdout generalization: moderate training gains generalized best, and the largest gains overfit. This paper reports the pre-registered cross-domain test of that relation. Under the frozen protocol (quartile buckets over margin-positive pairs; criterion: middle buckets' mean win rate exceeds the ends' by 0.05; $ 2$ of $ 2$ participating domains required), the criterion replicates: scheduling $0.35/0.60/0.70/0.48$ (mid $0.650$ vs.\ ends $0.413$) and MCS task assignment $0.36/0.93/1.00/1.00$ (mid $0.964$ vs.\ ends $0.679$). The second domain (MCS incentive design) is excluded by the protocol's own rule –- its verifier passed 3 of 2,000 LLM rules and 0 of the human-designed references, a threshold effect that leaves no margin-positive pairs. The substantive finding is not the verdict but the shape divergence beneath it: in scheduling the top quartile declines (48\ overfitting), while in assignment the top quartile saturates at 100\ different mechanisms. We read the pattern –- hard 0–1 verification collapses generalization, mid-severity simulation lets the top quartile overfit, feasibility-by-construction transfers it –- as evidence that verifier strictness determines the fate of the largest training gains, and we report the two-level verdict (criterion: replicated; mechanism: not shared) as the result. Pre-registrations, ledgers, and analysis scripts are available from the authors upon reasonable request via email.