A simple distance rule beats genomic language models at operon prediction

Language models trained on bacterial genomes are evaluated on operon identification and on protein–protein interaction (PPI) prediction, and good scores are read as evidence that they learn genome organisation. Yet genes in one operon lie on the same strand with short gaps between them, and interacting proteins are often encoded by neighbouring genes. We re-ran the 12 protein and genomic language models of the BacBench benchmark and asked how much of their performance gene position (strand, intergenic distance and adjacency) explains. An untuned rule that uses only strand and intergenic distance exceeds every model's BacBench score on long-read operon maps (0.826 against at most 0.776 AUROC), and it outperforms the operon predictor of the genomic language model gLM on gLM's own evaluation set. At fixed position, the models' embedding similarity carries little or no operon signal, so scores on these operon benchmarks leave open whether the models have learned operon structure. For PPI, the labels from STRING are themselves built partly from genome organisation, and position alone outperforms most models zero-shot, although five of the six models we re-ran for this task keep significant AUROC signal beyond it. In supervised probes, the advantage of the genomic language model Bacformer Large over the protein language model it builds on disappears once position features are added. Benchmarks of genomic language models should report position baselines and what the models add to position.

Code, result tables, logs and figures: doi:10.5281/zenodo.23484463.