DNAformer Datasets

Bar-Lev, D., Orr, I., Sabary, O., Etzion T., & Yakkobi, E.   Scalable and robust DNA-based storage via coding theory and deep learning. 2024.

Datasets description

This document provides an overview of the 5 datasets introduced in this work. For each dataset we provide both the raw .fastq files with the sequenced reads, as well as a file that includes the processed binned reads that were obtained by the binning step described in the paper. 

The dataset is provided under similar license as the code repository, with scripts for loading and processing the data at: https://github.com/itaiorr/Deep-DNA-based-storage.git

The datasets

The data was synthesized using Twist Bioscience and are differentiated by the sequencing technology used. Two Illumina datasets, both generated by Illumina miSeq. The reads in these two datasets were sequenced with paired-end sequencing, while the merging (stitching) was done with PEAR software. We include both raw reads and stitched reads in our repository under the names:

1. Pilot Illumina dataset 

   1. BinnedPilotIllumina.txt  - include the pilot dataset in binned format. 
   2. P-Pilot_S2_L001_R1_001.fastq.gz - include the pilot reads pre-stitching as obtained from Illumina miSeq. 
   3. P-Pilot_S2_L001_R2_001.fastq.gz - include the pilot reads pre-stitching as obtained from Illumina miSeq.
   4. Pilot_Illumina_raw_reads.fastq - includes the reads post-stitching. 

    

2. Test Illumina dataset

   1. BinnedTestIllumina.txt - Includes the test dataset in binned format. 
   2. F1-Full-Pool_S1_L001_R1_001.fastq.gz - includes the Illumina reads pre-stitching as obtained from Illumina miSeq. 
   3. F1-Full-Pool_S1_L001_R2_001.fastq.gz-  includes the Illumina reads pre-stitching as obtained from Illumina miSeq. 
   4. test_illumina_raw_reads.fastq - Includes the reads post-stitching. 

Three Nanopore datasets, all generated by Oxford Nanopore Technologies MinION under the names:

1. Pilot Nanopore dataset

   1. BinnedPilotNanopore.txt  - reads in binned format. 
   2. raw_reads_pilot_nanopore.zip - original basecalled reads as obtained from ONT MinION.
   3. Pilot_RawSignals_1_5.zip , Pilot_RawSignals_6_10.zip , Pilot_RawSignals_11_13.zip - raw nanopore signals as obtained from ONT MinION. 

2. Test Nanopore first flowcell dataset (termed in the paper as “Nanopore single flowcell”). 

   1. BinnedNanoporeFirstFlowcell.txt - reads in binned format. 
   2. test_pool_nanopore_single.zip - original basecalled reads as obtained from ONT MinION. 
   3. NanoporeFirstFlowcellRawSignals.zip - raw nanopore signals as obtained from ONT MinION.

3. Test Nanopore second flowcells dataset

   1. BinnedNanoporeSecondFlowcell.txt  - reads in binned format. 
   2. test_nanopore_second_flowcell_part001.zip , test_nanopore_second_flowcell_part002.zip- original basecalled reads as obtained from ONT MinIN.
   3. NanoproeSecondFlowcellRawSignals_1_5.zip , NanoproeSecondFlowcellRawSignals_6_10.zip , NanoproeSecondFlowcellRawSignals_11_15.zip - raw nanopore signals as obtained from ONT MinION.

Additionally, for completeness, we also included a file with the processed and binned reads of the test Nanopore dataset of the combined two flowcells dataset (termed in the paper as “Nanopore two flowcells”). This can be found in the file BinnedNanoporeTwoFlowcells.txt. 

Detailed description

The binned format was created using the binning step described in the paper. Each cluster of reads appears in the file with a header followed by the reads. More specifically:

1. The header consists of 2 lines; the first corresponds to the encoded sequence of the clusters, and the second is a line of 18x“*” that should be ignored

2. The reads in the clusters are provided after the header, where each read is given in a separate line

3. Each cluster ends with two empty lines

Data processing

To ease the processing of our datasets, we also provide the following Python scripts (see https://github.com/itaiorr/Deep-DNA-based-storage)

1. Preprocessor.py - includes our preprocessing procedure of the raw reads. The procedure detects  and truncates the primers 

2. Parser.py - parses the file of the binned reads and creates two Python dictionaries. In the first dictionary each key is an encoded sequence,  and the value is a  list of the reads in the cluster. In the second dictionary the keys are the index and the value is a list of the reads in the cluster.