Verifier-Bound Rule Discovery: A Pre-Registered Evaluation of LLM Proposers for Combinatorial Scheduling
Description
Large language models are increasingly used as proposers of human-readable rules in scientific discovery and combinatorial optimization (FunSearch, ARCHE, SVBRD-LLM). Yet the LLM-proposer systems we surveyed share a methodological gap: none reports a same-budget random-proposer control, none pre-registers its acceptance criteria, and none evaluates at distribution scale –- ARCHE validates on three hand-picked cases, and SVBRD-LLM, the closest system to ours, is outperformed on accuracy by a black-box LSTM while reporting no significance tests. We apply a verifier-bound rule-discovery framework in which an LLM proposes interpretable rules (linear weight vectors and sandboxed programs), an independent programmatic verifier checks domain-hard properties, holdout evaluation is physically isolated and executed exactly once, and every acceptance criterion is frozen before experiments run. On an industrial scheduling domain, under a 20-evaluation budget, the LLM proposer reaches train quality far above a same-budget random control (median new-overdue 11 vs.\ 14, vs.\ production baseline 17), but its holdout win rate is 55/100 bootstrap pairs –- an inconclusive band by our pre-registered criteria. The failure mode is nevertheless systematic and stable: a decoupling between train improvement and generalization –- the proposer finds a train-beating rule in 81/100 pairs, but 47\