ARGA Data Retrieval

????Data are stored in many different shapes and sizes everywhere across the internet, and often without consistency. The goal of the Australian Reference Genome Atlas (ARGA) project is to show users where to obtain data they may be interested in, and how it compares to data in other locations. To do this, we first need to collect the metadata and standardise it such that all pieces of metadata can be interpreted similarly. ARGA has created a bespoke in-house data retrieval pipeline for this purpose. The data retrieval pipeline consists of three phases: download, processing, and conversion; this pipeline is driven by configuration files outlined for each database of interest. Downloading is currently split into distinct categories: URL, crawling, and script retrieval. Script retrieval is the most diverse, being used for both API access and scraping static data from web pages. Processing is then implemented based on specific file processing, processing that needs to be applied to all files, and final stage processing that combines files into a final output file representative of the entire target database. Some datasets require little to no processing, but are always represented with a single output file. Using this final file, mapping is implemented to convert columns to standardised names within the ARGA schema through the conversion process. Here we detail the data ingestion pipeline and demonstrate how it will be implemented to index more datasets and adapt to handle any complex data challenges ahead.