Catalogue of Life Repackaged and Sorted hash://sha256/7ba03a18029f51f07c73b1dd792a5a7054b1a439b30c20a4b438ed7a84b26e0c hash://md5/f9d660d798d7d81a8a6a204fcba0c350

Description

Technical info

# Introduction

The Catalogue of Life [@Banki_2023] is "[...] is an assembly of expert-based global species checklists with the aim to build a comprehensive catalogue of all known species of organisms on Earth. [...]"

This data publication contains a verifiable copy of the Catalogue of Life as well as a reverse sorted version of the NameUsage.tsv table. The aims of this publication are to:

1. provide a signed citation [@Elliott_2023] for a copy of Catalogue of Life
2. prepare Catalogue of Life to be included in the Nomer Corpus of Taxonomic Resources [@Poelen_2023a]. 
3. pre-process the Catalogue of Life to facilitate optimized indexing and offline taxonomic name alignments using tools like Nomer [@Poelen_2023b]. 

Overall, the publications aims to facilitate taxonomic name alignment using the wealth of information provided by the Catalogue of Life to help enable fast, reproduceable, offline-enabled alignment of namelists with taxonomic resources of known provenance (or origin). 

An example of an application facilitated by this publication is the Taxonomic Name Alignment tool as provided through https://github.com/globalbioticinteractions/name-alignment-template. This template repository implements an automated workflow using GitHub Action to align scientific names in csv/tsv files and darwin core archive with common taxonomic name lists like Catalogue of Life, NCBI Taxonomy, Integrated Taxonomic Information System (ITIS), and GBIF Backbone taxonomy.

## Methods

To capture and process the Catalogue of Life, the following steps were taken:

1. track and archive a copy of Catalogue of Life
2. reverse sort NameUsage.tsv
3. assign an alias to the processed resources

Steps 1-3 are captured and documented using Preston, a biodiversity data tracker. Preston not only helps to documents the steps, but also includes the digital resources that were used and produced. 

### Track and Archive

To track and archive a copy of Catalogue of Life, the following command was issued:

~~~
preston track https://download.catalogueoflife.org/col/xr_latest_coldp.zip
~~~

With this, a copy of https://download.catalogueoflife.org/col/xr_latest_coldp.zip is downloaded and their sha256 checksum (or hash) is calculated. Also, the download process is captured machine-readable rdf/nquads statement. 

The content id (or sha256 hash) of the copy included in this publication can be found using:

~~~
preston alias \
  --anchor hash://sha256/7ba03a18029f51f07c73b1dd792a5a7054b1a439b30c20a4b438ed7a84b26e0c \
  --remote https://zenodo.org/deposit/19460413/files \
   https://download.catalogueoflife.org/col/xr_latest_coldp.zip \
 | head -n1
~~~

and is 

~~~
<https://download.checklistbank.org/col/xr_latest_coldp.zip> <http://purl.org/pav/hasVersion> <hash://sha256/2efcf557bb213a26c8035a783c076eb18acd809a90b64cdd278ae5ded0611c13> <urn:uuid:c02514cc-4a51-4fb7-9aae-08ad8d10df5d> .
~~~

meaning that `hash://sha256/2efcf557bb213a26c8035a783c076eb18acd809a90b64cdd278ae5ded0611c13` is the content id (or sha256 hash) of the content produced by `https://download.catalogueoflife.org/col/xr_latest_coldp.zip` at the time this publication was compiled.

### Reverse Sort NameUsage.tsv

The second step was to pre-process (or transform) part of the tracked Catalogue of Life data package and reverse sort their content. We use the following bash script to do this processing:

~~~
(sed -u 1q; LC_ALL=C sort -r) | gzip
~~~

The script first prints the header (i.e., `sed -u 1q`), then reverse sorts the following content (i.e., `LC_ALL=C sort -r`). Finally, the output is compressed using gzip.

This script is part of this data publication, and can be retrieved via:

~~~
preston cat \
 --remote https://zenodo.org/record/19460413/files/ \
 hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd
~~~

To help make it easier to reference the script, an alias was created using

~~~
preston alias \
 urn:example:reverse-sort.sh \
 hash://sha256/03292e3e40a04c565c83debd26ed521516f158a29c011b57a96d1c720c40b6cd
~~

Following the authoring of the reverse sort script, and documenting their alias (i.e. 'urn:example:reverse-sort.sh`), we applied the script to the acquired copy of the Catalogue of Life using:

~~~
preston cat \
  --remote https://zenodo.org/record/19460413/files/ \
  'zip:hash://sha256/2efcf557bb213a26c8035a783c076eb18acd809a90b64cdd278ae5ded0611c13!/NameUsage.tsv'\
 | preston bash\
 --anchor hash://sha256/7ba03a18029f51f07c73b1dd792a5a7054b1a439b30c20a4b438ed7a84b26e0c\
 --remote https://zenodo.org/record/19460413/files/\
 -c urn:example:reverse-sort.sh 
~~~

The result of this process was the content identified by sha256 hash 

~~~
hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc
~~~

as documented in line 22 of the associated preston processing log in 

~~~
preston cat\
  --remote https://zenodo.org/record/19460413/files/\
 'line:hash://sha256/8ba35deafc847f0d5d69d357241a431b7fd9b6f2735189575b2b7168d523caa9!/L22' 
~~~

### Assign An Alias to Processed Resources 

The alias 'col:NameUsage.tsv.gz' was defined to help make it easier to point to the result using:

~~~
preston alias\
  col:NameUsage.tsv.gz\
  hash://sha256/1008433bfa5fe7fb059a720eddfe995e18b3e0e8f25ac0c990f1477c177d18cc
~~~

With this, the following command was executed to list the first three lines of the produced resource:

~~~
preston cat \
 --anchor hash://sha256/7ba03a18029f51f07c73b1dd792a5a7054b1a439b30c20a4b438ed7a84b26e0c \
 --remote https://zenodo.org/record/19460413/files/ \
 col:NameUsage.tsv.gz \
 | gunzip \
 | head -n3
~~~

Where `preston cat ...` prints the produced resource, `gunzip` uncompresses the result, and the first three lines are selected using `head -n3`. 

The result of the operation is shown below:

| col:ID | col:alternativeID | col:nameAlternativeID | col:sourceID | col:parentID | col:basionymID | col:status | col:scientificName | col:authorship | col:rank | col:notho | col:originalSpelling | col:uninomial | col:genericName | col:infragenericEpithet | col:specificEpithet | col:infraspecificEpithet | col:cultivarEpithet | col:combinationAuthorship | col:combinationAuthorshipID | col:combinationExAuthorship | col:combinationExAuthorshipID | col:combinationAuthorshipYear | col:basionymAuthorship | col:basionymAuthorshipID | col:basionymExAuthorship | col:basionymExAuthorshipID | col:basionymAuthorshipYear | col:namePhrase | col:nameReferenceID | col:publishedInYear | col:publishedInPage | col:publishedInPageLink | col:gender | col:genderAgreement | col:etymology | col:code | col:nameStatus | col:accordingToID | col:accordingToPage | col:accordingToPageLink | col:referenceID | col:scrutinizer | col:scrutinizerID | col:scrutinizerDate | col:extinct | col:temporalRangeStart | col:temporalRangeEnd | col:environment | col:species | col:section | col:subgenus | col:genus | col:subtribe | col:tribe | col:subfamily | col:family | col:superfamily | col:suborder | col:order | col:subclass | col:class | col:subphylum | col:phylum | col:kingdom | col:ordinal | col:branchLength | col:link | col:nameRemarks | col:remarks | col:modified | col:modifiedBy | clb:merged |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| zylcJ5C5xwfQ5_i9Ss9 |  |  | 37384 | KVV3T |  | accepted | BOLD:ADE5700 |  | unranked |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | Tenebrionidae | Tenebrionoidea | Polyphaga | Coleoptera |  | Insecta | Hexapoda | Arthropoda | Animalia |  |  | http://bins.boldsystems.org/index.php/Public_BarcodeCluster?clusteruri=BOLD:ADE5700 |  | Family names applied to this BIN: Tenebrionidae (1). |  |  | true |
| z6CIok51k99C3_1pENr |  |  | 37384 | 7NF3Y |  | accepted | BOLD:AAZ3374 |  | unranked |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | Gastropoda |  | Mollusca | Animalia |  |  | http://bins.boldsystems.org/index.php/Public_BarcodeCluster?clusteruri=BOLD:AAZ3374 |  | Order names applied to this BIN: Stylommatophora (1). |  |  | true |

# Results

As described in our methods, this publication derived the resource with alias col:NameUsage.tsv.gz and content id 
  ```hash://sha256/07ed13f5e9d3e091871cb40d11e4789e92c2453a8a761e6d60556ade24360292``` . This resource contains a reverse-sorted copy of the NameUsage.tsv provided in the Catalogue of Life data package retrieved from https://download.catalogueoflife.org/col/xr_latest_coldp.zip with content identifier `hash://sha256/2efcf557bb213a26c8035a783c076eb18acd809a90b64cdd278ae5ded0611c13`.

The following tools were used to process the Catalogue of Life resource:

 | tool name | 
 | --- | 
 | [preston](https://github.com/bio-guoda/preston) |
 | [bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) | 
 | [gzip](https://en.wikipedia.org/wiki/Gzip) |   
 | [sed](https://en.wikipedia.org/wiki/Sed) |
 | [head](https://en.wikipedia.org/wiki/Head_(Unix)) |
 | [sort](https://en.wikipedia.org/wiki/Sort_(Unix)) |
: Tools used in this data publication

# Discussion 

This publication is intended to facilitate re-use of the Catalogue of Life data package in taxonomic name alignment workflows. While the primary goal was to generate a resource for use in Nomer v0.6.2 [@Poelen_2023b], other usage can be imagined such as:

1. Lots of Copies Keeps Stuff Safe (LOCKSS [@Maniatis_2005]): keep an identical copy of Catalogue of Life data package outside of the Catalogue of Life infrastructure.
2. demonstrating how data transformation processes can be documented using Preston
3. making a streamable copy of a reverse-sorted copy of Catalogue of Life available via https://zenodo.org/record/19460413/files/07ed13f5e9d3e091871cb40d11e4789e92c2453a8a761e6d60556ade24360292 for use in workflows like looking up the first record that contain _Enhydra lutris_ (Sea otter):

~~~
 curl -L 'https://zenodo.org/record/19460413/files/07ed13f5e9d3e091871cb40d11e4789e92c2453a8a761e6d60556ade24360292'\
   | gunzip\
   | grep "Enhydra lutris"\
   | head -n1
~~~

# Acknowledgements

This work stands on the shoulders of contributors to open source software and openly accessible datasets. Thank you!