Comprehensive benchmark and architectural analysis of deep learning models for Nanopore sequencing basecalling

Nanopore-based DNA sequencing relies on basecalling the electric current signal. Basecalling1
requires neural networks to achieve competitive accuracies. To improve sequencing accuracy further,2
new models are continuously proposed. However, benchmarking is currently not standardized, and3
evaluation metrics and datasets used are defined on a per publication basis, impeding progress in4
the field. To standardize the process of benchmarking, we unified existing benchmarking datasets5
and defined a rigorous set of evaluation metrics. We benchmarked the latest seven basecaller models6
and analyzed their deep learning architectures. Our results show that overall Bonito has the best7
architecture for basecalling. We find, however, that species bias in training can have a large impact8
on performance. Our comprehensive evaluation of 90 novel architecture demonstrates that different9
models excel at reducing different types of errors and using RNNs (LSTM) and a CRF decoder are10
the main drivers of high performing models.