A biodiversity dataset graph: Biological Associations in TaxonWorks hash://sha256/a4d651aac5220487835e6178511886e98b845b2d98cb7c5447fb2b042e0654d2 hash://md5/849edbe55e31e54ea5cdaba0188c5655

The intended use of this archive is to facilitate (meta-)analysis of Biological Associations captured in TaxonWorks [1]. TaxonWorks is an integrated web-based workbench for taxonomists and biodiversity scientists. It allows you to capture, organize, and enrich your data; share it with collaborators; and package it for analysis and publication. 

This dataset provides versioned snapshots of the TaxonWorks network as tracked by Preston [2,3,4] between 2023-08-16 and 2023-08-16 using:

preston track -u https://sfg.taxonworks.org

. In addition, this dataset provides a processed version of the biological associations using the "preston tw-stream" command as generated by the following bash script:

#!/bin/bash
#
# Generates GloBI interaction JSON Lines from provided provenance log as generated by preston tw-stream.
#

/usr/local/bin/preston cat hash://sha256/ff5e709305e593c87711e897b6341b94e775e2f312aa6d4ae5ed6120babd6f5e\
 | /usr/local/bin/preston tw-stream

The script itself was executed using:

cat transform.sh | preston bash

The execution of this transform.sh script (with content id hash://sha256/ea6c131a7436c9080b9380c7712dd32654a564fbb6f0568ff97bd7a8f28337a4), as well as their results, is captured within this datasets also. A rdf/quads formatted machine readable version of the workflow execution description can be found via:

preston cat hash://sha256/ab7550368905e7c919e70a306efbb97719a1edbba2cfe4c4515f635ebc0be4bb 

And, the resulting JSON Lines file has content id (or signature) hash://sha256/93c748c9e88a8c221f4d9cf4de37f0506d8427a4383cf1ee2078983cdeb31a14 and is also included as interactions.json to facilitate access. 

The first json record can be generated using:

preston cat hash://sha256/93c748c9e88a8c221f4d9cf4de37f0506d8427a4383cf1ee2078983cdeb31a14\
 | head -n1\
 | jq .

or, provided that the interactions.json has content id starting with hash://sha256/da34e70...

cat interactions.json\
 | head -n1\
 | jq .

This produces the following (formatted) json object:

{
  "http://www.w3.org/ns/prov#wasDerivedFrom": "hash://sha256/f1c123b38bdfab129d8a393b06842eddae823bf08a2c91b1100399940b582a9a",
  "http://www.w3.org/1999/02/22-rdf-syntax-ns#type": "application/vnd.taxonworks+json",
  "referenceId": "https://sfg.taxonworks.org/api/v1/sources/49013",
  "interactionId": "https://sfg.taxonworks.org/api/v1/biological_associations/84748",
  "taxonRootsResolved": 2,
  "referenceResolved": true,
  "referenceCitation": "@article{49013,\n  author = {Abate, T.},\n  booktitle = {Journal of Applied Entomology},\n  journal = {Journal of Applied Entomology},\n  month = {mar},\n  day = {31},\n  pages = {278-285},\n  title = {The bean fly <i>Ophiomyia phaseoli</i> Tryon (Diptera: Agromyzidae) and its parasitoids in Ethiopia.},\n  volume = {111(3)},\n  year = {1991},\n  stated_year = {1991},\n  year_suffix = {a},\n  issn = {0044-2240}\n}\n",
  "interactionTypeId": "gid://taxon-works/BiologicalRelationship/14",
  "interactionTypeName": "Primary host",
  "sourceTaxonName": "Eupelmus",
  "sourceTaxonId": "gid://taxon-works/TaxonName/456381",
  "sourceTaxonRank": "genus",
  "sourceTaxonAuthorship": "Dalman, 1820",
  "sourceTaxonPath": "Root | Animalia | Arthropoda | Insecta | Hymenoptera | Chalcidoidea | Eupelmidae | Eupelminae | Eupelmus",
  "sourceTaxonPathIds": "gid://taxon-works/TaxonName/455455 | gid://taxon-works/TaxonName/703648 | gid://taxon-works/TaxonName/703653 | gid://taxon-works/TaxonName/703658 | gid://taxon-works/TaxonName/455456 | gid://taxon-works/TaxonName/455458 | gid://taxon-works/TaxonName/455504 | gid://taxon-works/TaxonName/455506 | gid://taxon-works/TaxonName/456381",
  "sourceTaxonPathNames": "nomenclatural rank | kingdom | phylum | class | order | superfamily | family | subfamily | genus",
  "targetTaxonName": "Agromyzidae",
  "targetTaxonId": "gid://taxon-works/TaxonName/513569",
  "targetTaxonRank": "family",
  "targetTaxonAuthorship": "",
  "targetTaxonPath": "Root | Animalia | Arthropoda | Insecta | Diptera | Agromyzidae",
  "targetTaxonPathIds": "gid://taxon-works/TaxonName/455455 | gid://taxon-works/TaxonName/703648 | gid://taxon-works/TaxonName/703653 | gid://taxon-works/TaxonName/703658 | gid://taxon-works/TaxonName/513567 | gid://taxon-works/TaxonName/513569",
  "targetTaxonPathNames": "nomenclatural rank | kingdom | phylum | class | order | family"
}

In this example, a claim is made that, according to https://sfg.taxonworks.org/api/v1/sources/49013 [6]  Eupelmus (a parasitic wasp) has a primary host in the family of Agromyzidae (leaf-miner flies). 

In total, 138,246 such claims can be found in the generated resource with alias interactions.json and content id starting with hash://sha256/93c748c... .

In addition, the archive consists of 256 individual parts (e.g., preston-00.tar.gz, preston-01.tar.gz, ...) to allow for parallel file downloads. The archive contains three types of files: index files, provenance logs and data files. In addition, index files have been individually included in this dataset publication to facilitate remote access. Index files provide a way to links provenance files in time to establish a versioning mechanism. Provenance files describe how, when, what and where the TaxonWorks content was retrieved. For more information, please visit https://preston.guoda.bio or https://doi.org/10.5281/zenodo.1410543 .  

To retrieve and verify the downloaded TaxonWorks biodiversity dataset graph, first concatenate all the downloaded preston-*.tar.gz files (e.g., cat preston-*.tar.gz > preston.tar.gz). Then, extract the archives into a "data" folder. Alternatively, you can use the preston[2] command-line tool to "clone" this dataset using:

java -jar preston.jar clone --remote https://zenodo.org/record/8253729/files

After that, verify the index of the archive by reproducing the following provenance log history:

 java -jar preston.jar history --log tsv

to be:

hash://sha256/a4d651aac5220487835e6178511886e98b845b2d98cb7c5447fb2b042e0654d2	http://www.w3.org/ns/prov#wasDerivedFrom	hash://sha256/ab7550368905e7c919e70a306efbb97719a1edbba2cfe4c4515f635ebc0be4bb
hash://sha256/ab7550368905e7c919e70a306efbb97719a1edbba2cfe4c4515f635ebc0be4bb    http://www.w3.org/ns/prov#wasDerivedFrom    hash://sha256/ff5e709305e593c87711e897b6341b94e775e2f312aa6d4ae5ed6120babd6f5e    
urn:uuid:0659a54f-b713-4f86-a917-5be166a14110    http://purl.org/pav/hasVersion    hash://sha256/ff5e709305e593c87711e897b6341b94e775e2f312aa6d4ae5ed6120babd6f5e    

To check the integrity of the extracted archive, confirm that each line produce by the command "preston verify" produces lines as shown below, with each line including "CONTENT_PRESENT_VALID_HASH". Depending on hardware capacity, this may take a while.

java -jar preston.jar verify

Note that a copy of the java program "preston", preston.jar, is included in this publication. The program runs on java 8+ virtual machine using "java -jar preston.jar", or in short "preston". 

Files in this data publication:

--- start of file descriptions ---

-- description of archive and its contents (a rendition of this file) --
README

-- biological associations indexed from TaxonWorks expressed in a GloBI [5] compatible JSON Lines file --
interactions.json

-- executable java jar containing preston [2,3,4] v0.7.4-SNAPSHOT. --
preston.jar

-- preston archives containing TaxonWorks data files, associated provenance logs and a provenance index --
preston-[00-ff].tar.gz -- individual provenance index files --

2a5de79372318317a382ea9a2cef069780b852b01210ef59e06b640a3539cb5a
f98d36a9dc7bd833c93b3b61130865628f7bc2f7bb0920e95afcd16fba3dc6a8
29306c5c144c3d7fd21be344d8b6b554b6f6efa3b8f8f5c0b27cdf0e88785652

--- end of file descriptions ---

References 
[1] TaxonWorks is an integrated web-based workbench for taxonomists and biodiversity scientists. (TaxonWorks, https://sfg.taxonworks.org) accessed from 2023-08-16 to 2023-08-16 with provenance hash://sha256/a4d651aac5220487835e6178511886e98b845b2d98cb7c5447fb2b042e0654d2.
[2] https://preston.guoda.bio, https://doi.org/10.5281/zenodo.1410543 . 
[3] MJ Elliott, JH Poelen, JAB Fortes (2020). Toward Reliable Biodiversity Dataset References. Ecological Informatics. https://doi.org/10.1016/j.ecoinf.2020.101132 hash://sha256/136c3c1808bcf463bb04b11622bb2e7b5fba28f5be1fc258c5ea55b3b84f482c
[4] MJ Elliott, JH Poelen, JAB Fortes (2023). Signing data citations enables data verification and citation persistence. Scientific Data. https://doi.org/10.1038/s41597-023-02230-y hash://sha256/f849c870565f608899f183ca261365dce9c9f1c5441b1c779e0db49df9c2a19d 
[5] Poelen JH, Simons JD, Mungall CJ. (2014). Global Biotic Interactions: An open infrastructure to share and analyze species-interaction datasets. Ecological Informatics. https://doi.org/10.1016/j.ecoinf.2014.08.005.
[6] Abate, T. (1991) The bean fly Ophiomyia phaseoli Tryon (Diptera: Agromyzidae) and its parasitoids in Ethiopia. Journal of Applied Entomology 111(3), 278–285."

This work is funded in part by grant NSF OAC 1839201, NSF DBI 1901932, NSF DBI 1901926, and NSF DBI 2102006 from the National Science Foundation.