A living proteomics benchmark for comprehensive evaluation of deep learning-based de novo peptide sequencing tools

Introduction

De novo peptide sequencing identifies peptides from MS/MS data without relying on a reference proteome database. It is a powerful strategy for the discovery of novel peptides, and it is particularly valuable in applications where massive search spaces make database searching less effective such as immunopeptidomics and metaproteomics. 
 
Recent advancements in deep learning methods for sequential data have led to the emergence of multiple new de novo tools. However, their evaluation and comparison remain challenging due to the lack of standardized metrics and universal test datasets. This complicates determining the contexts where de novo sequencing is effective and leaves users without clear guidance on selecting the most suitable tool for their needs, highlighting the necessity of a unified benchmarking platform.

Methods

We have developed a comprehensive, community-driven benchmarking approach for evaluating de novo sequencing tools.
 
We compiled a diverse collection of 26 MS/MS datasets covering 18 million MS/MS spectra, encompassing various organisms, experimental conditions, and PTMs. Spectra were annotated with ground truth peptide labels through a standardized bioinformatics workflow that combines sequence database searching and PSM rescoring. To maintain the benchmark’s long-term validity and avoid potential overfitting, the data have been divided into public and private subsets.
 
To enable consistent benchmarking, we encapsulate each de novo tool within an individual Apptainer container with an isolated, reproducible environment. These containers employ standardized input and output formats, allowing diverse tools to be seamlessly integrated into a single automated workflow.

Preliminary Data

The benchmark includes 16 de novo sequencing tools—AdaNovo, BiATNovo, Casanovo, Contranovo, DeepNovo, DePS, GCNovo, GraphNovo, InstaNovo, Novor, PEAKS, PepNet, SearchNovo, SMSNet, Spectralis, and π-HelixNovo—each configured by its original developers for optimal performance. Predictions are evaluated at both the amino acid and peptide levels. Additionally, we assess the number of de novo predicted peptides that match the reference species proteome, as this can reveal correct peptide identification even when missed by database searching.
 
The benchmark spans various MS instrument vendors, multiple proteases, and a range of PTMs, including phosphorylation, TMT, and SILAC-labeled peptides. We cover multiple species, including those with proteomes highly divergent from human and common model organisms. Additionally, we incorporate diverse proteomics applications such as immunopeptidomics, single-cell proteomics, metaproteomics, and antibody sequencing.
 
Our benchmarking results indicate that deep learning-based de novo tools perform reliably under standard conditions, although their effectiveness can vary depending on algorithmic design and training dataset properties. Recent transformer-based tools like Casanovo perform well on human and commonly studied species data, as well as in immunopeptidomics and single-cell proteomics. However, peptides with rare or unsupported PTMs (e.g., phosphorylation) remain a significant challenge for most models, often resulting in less accurate sequencing or misidentifications.
 
Performance also declines when data originates from MS instruments differing from the commonly used Thermo Orbitrap. For example, tools show up to a 50% reduction in predictive performance on instruments with an ion mobility dimension, such as Bruker timsTOF. Similarly, non-tryptic proteases reduce peptide precision by 30–50% across various tools.
 
This benchmark is the first comprehensive evaluation of the de novo tools across diverse conditions, providing key insights into their current capabilities, strengths, and areas needing improvement. It serves as a valuable resource for the community, guiding tool selection and charting the course for future developments in de novo sequencing.

Novel Aspect

Large-scale, objective, and dynamic evaluation of the effectiveness and limitations of de novo peptide sequencing.