A benchmark framework to guide automated cyclic peptide discovery
Authors/Creators
- 1. Biomate AI
- 2. Department of Gastrointestinal Medical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
- 3. Department of Structural Biology, St. Jude Children's Research Hospital, Memphis, TN
- 4. BioMate AI
Description
Automated cyclic-peptide discovery runs a design–make–test–learn loop in which a
model chooses what to synthesise and measure next, so the whole enterprise rests on a
benchmark that can tell real learning from an artefact. That is harder than it sounds: an
apparent gain can come from the starting predictor or a nearby measured analogue rather
than from feedback; an improvement in one chemical series or laboratory need not
survive the move to the next; and a control meant to catch a rigged benchmark can just as
easily condemn a sound one. Here we present a benchmark framework that pulls these
questions apart with coordinated controls — frozen-model, one-shot, shuffled-feedback
and retrieval comparisons for the source of a gain; acquisition versus held-out-source
ranking for its transfer; and planted-valid and planted-shortcut tests for the validity check
itself. Applied to the largest public permeability and binding datasets, the controls are
revealing. Most of the selection advantage arrives before any feedback, and the
reproducible 47.3-hit gain proves to sit almost entirely in a single data source. Held-out-
source ranking improves only marginally (Spearman gain 0.075) and cannot be separated
from shuffled feedback. The registered validity gate falsely condemns a quarter to a third
of clean benchmarks, where a null-calibrated gate condemns just 3.0–8.3% and still
catches planted shortcuts (area under the receiver-operating curve, AUROC, 0.65 and
0.70); the same scrutiny leaves a generation benchmark inconclusive, its physics verifier
below its own floor (0.071 against 0.10). The framework turns benchmark interpretation
into an empirical object of study: trust comes from controls, not from a headline metric.