A Graded Evaluation of RNA Structure Awareness in Foundation Models: From Stem-Loop Discrimination to Partner Specificity

Three-rung composition-controlled evaluation of RNA structure awareness in ten foundation models (five RNA-pretrained, five DNA-pretrained), revealing that partner-level structural knowledge requires both architectural inductive bias and learned weights.

52 Rfam families across tRNAs, rRNAs, ribozymes, riboswitches, snRNAs, and CRISPR repeats, evaluated with SHA-frozen preregistered protocols at three levels of stringency. Rung 1 (stem/loop discrimination) applies a nucleotide-stratified permutation null that absorbs 3x of the naive signal, exposing GC enrichment as the dominant confound. Rung 2 (dinucleotide-stratified null) eliminates all but one family (HCV IRES III). Rung 3 (partner specificity) tests whether models resolve which specific position pairs with which, using within-stem derangement nulls (1,000 derangements per stem, K >= 3 eligible pairs).

Two models clear the third rung. RiNALMo (650M) achieves the highest mean perturbation specificity (PS = 0.227, 28/29 qualifying families, partner-is-max precision 87.4%). ERNIE-RNA (86M) follows (PS = 0.117, 28/30 families, 88.3%). An untrained control (randomized weights, preserved architecture) produces PS = 2.5 x 10^-8, confirming seven orders of magnitude separating learned from architectural signal. NT v2 and DNABERT-2 encode structure through attention-contact correlation (rho = 0.32, 0.26) without embedding-level partner specificity. DNABERT-2's BPE tokenizer limits evaluation to 8 of 52 families; none pass the positive control gate. Evo (7B) shows the weakest signal despite 500x the parameters of the next-largest SSM. RiNALMo also shows the strongest probing signal of any model (balanced accuracy = 0.623, delta = +0.123 above chance at layer 32).

This deposit contains the full per-family result files (Phase 6 perturbation specificity for all 10 models plus untrained control, Phase 6 synthetic derangement results, Phases 1-5 expanded Rfam including RiNALMo, HTT repeat-expansion case study), 52 Rfam family structures, preregistration documents with commit SHAs (Phase 1: 694b43b, Phase 2: bd4b3fd, Phase 6: c19aa59), evaluation scripts, and the manuscript (LaTeX source and compiled PDF).