Published November 10, 2026 | Version v2

A benchmark framework to guide automated cyclic peptide discovery

  • 1. Biomate AI
  • 2. Department of Gastrointestinal Medical Oncology, The University of Texas MD Anderson Cancer Center, Houston, TX
  • 3. Department of Structural Biology, St. Jude Children's Research Hospital, Memphis, TN
  • 4. BioMate AI

Description

Automated cyclic-peptide discovery runs a design–make–test–learn loop in which a

model chooses what to synthesise and measure next, so the whole enterprise rests on a

benchmark that can tell real learning from an artefact. That is harder than it sounds: an

apparent gain can come from the starting predictor or a nearby measured analogue rather

than from feedback; an improvement in one chemical series or laboratory need not

survive the move to the next; and a control meant to catch a rigged benchmark can just as

easily condemn a sound one. Here we present a benchmark framework that pulls these

questions apart with coordinated controls — frozen-model, one-shot, shuffled-feedback

and retrieval comparisons for the source of a gain; acquisition versus held-out-source

ranking for its transfer; and planted-valid and planted-shortcut tests for the validity check

itself. Applied to the largest public permeability and binding datasets, the controls are

revealing. Most of the selection advantage arrives before any feedback, and the

reproducible 47.3-hit gain proves to sit almost entirely in a single data source. Held-out-

source ranking improves only marginally (Spearman gain 0.075) and cannot be separated

from shuffled feedback. The registered validity gate falsely condemns a quarter to a third

of clean benchmarks, where a null-calibrated gate condemns just 3.0–8.3% and still

catches planted shortcuts (area under the receiver-operating curve, AUROC, 0.65 and

0.70); the same scrutiny leaves a generation benchmark inconclusive, its physics verifier

below its own floor (0.071 against 0.10). The framework turns benchmark interpretation

into an empirical object of study: trust comes from controls, not from a headline metric.

Files

A benchmark framework to guide automated cyclic peptide discovery.pdf

Files (3.0 MB)