Dedupuplicate_UMI is a tool written in rust to remove alignment-free exact-duplicate FASTQ read-pairs using the UMI sequences-library approach. Duplicates are identified using a concatenation of R1, R2 and UMI sequences approach. For duplicate molecules the read-pair with the highest total base PHRED quality score is retained.

Exact duplicate reads (including identical UMI sequences) are typically the result of over-amplification during PCR in the sequencing library preparation. For quantification and some assembly-based approaches you want these overamplified sequences removed.

However, identical read sequences without the same UMI can occur naturally and may represent independent molecules that were sequenced multiple times. These should generally NOT be removed. The probability of observing such biologically relevant duplicates increases with deeper sequencing. Typically the occurrence of erroneous sequence duplicates is linked to input DNA concentrations and the number of amplification cycles used in the library prep.

Exact duplicates should only be removed when both the read sequences and the UMI sequence are identical (amplification artefact).

Reads are considered duplicates when the following combination is identical: R1 sequence + R2 sequence + UMI

For the older R3 system this becomes: R1 sequence + R2 sequence + R3 sequence

• Binary rust build based on x86_64-unknown-linux-musl for maximum HPC compatibility.
• It writes out the last sequences found of a duplicate set having the highest TOTAL qualityscore.
• Input FASTQ files must follow the standard 4-line FASTQ format, starting at the first record.
  • Sequences in R1 R2 (and R3) should be in same order and NOT INTERLEAVED!!
• UMI location (header or separate FASTQ file) is detected automatically unless --noUMI is used
• If using --noUMI we just filter exact duplicates. This most likely filters too harsh since exact duplicates can occur in natural good quality datasets. Especially at high sequencing depths.
• Gzip I/O is now handled natively via rust::flate2 (zlib-ng backend).
  • No external gzip/zcat/pigz dependency required anymore from version 1.4r and up.
  • Gzip is the only parallelised part of the tool and its greedy for the number of cores. Limit the number of cores using --max-cores when needed.

• FASTQ files in fixed 4-line format having read-pairs in separate files (DO NOT use INTERLEAVED FASTQ files)!
• Since the rust version 1.4r Gzip I/O is now handled natively via rust::flate2 (zlib-ng backend). No external gzip/zcat/pigz dependency required anymore.

Processing time is typically limited by disk I/O. The script writes the read pair with the highest summed base quality from each set of exact duplicates (keeping the best readset).

Deduplication stores unique reads as keys in memory. Memory usage scales with the number of unique molecules in the dataset. In practice MAXIMUM memory usage is approximately: ~2 × size of the uncompressed R1 sequence file

Example benchmark on a large dataset:

Dataset: 69 million paired-end reads (clusters) in R1 R2 R3 (R1.gz ≈ 6 GB)

Server: PowerEdge R750, Intel Xeon Gold 6354 (72 threads), 256 GB memory running Ubuntu 24.04.3 LTS.

Results:

• Read + deduplicate: ~6 minutes
• Compression + writing: ~5 minutes
• Total runtime: ~11 minutes

Processing speed is largely limited by gzip (de)compression and disk I/O. Details in the performance_test directory.

a.bossers@uu.nl // alex.bossers@wur.nl

Script is provided AS IS under GPL-3.

We did our best to verify that the results are legitimate. However, the output should be considered erroneous, so you should check your results! The authors, nor their institutions/employers, are in any way direct or indirect responsible for the direct or indirect damages caused by using this tool. Use it at your own responsibility.