Datasets and software for article "Predicting coarse-grained representations of biogeochemical cycles from metabarcoding data"

Datasets for Tabigecy article (reviewed)

This repository contains additional information for the article "Predicting coarse-grained representations of biogeochemical cycles from metabarcoding data". This information allows the results presented in the article to be reproduced.

The version 2 of this archive contains experiments performed following the review of the article. To have access to the data for the experiments presented in the draft version look at the version 1 of this archive.

This archive contains:

- article_data_review.zip: updated results on the two salt cavern datasets:

• input files used for the article experiments:
  • bordenave_et_al_2013.tsv, bordenave_et_al_2013_abundance.csv and bordenave_et_al_2013_group.tsv for the Bordenave et al. dataset.
  • schwab_et_al_2022.tsv, schwab_et_al_2022_abundance.tsv and schwab_et_al_2022_sample_grouping.tsv for the Schwab et al. dataset.
• output folders from Tabigecy run on these inputs:
  • output_bordenave: output folder for Bordenave et al. dataset.
  • output_schwab: output folder for Schwab et al. dataset.
• scripts used to create plots:
  • create_pca.R to create PCA biplot and correlation plot from the results of both datasets.
  • bordenave_create_figure_article.py to create polar plots for the Bordenave et al. dataset.
  • schwab_create_figure_article.py to create polar plots for the Schwab et al. dataset.
• several folders containing svg files for the article figures: bordenave_figure, experiment_figure, schwab_figure and workflow_figure.

- comparison_metabolic.zip: it contains the comparison between bigecyhmm and METABOLIC on two metagenomic datasets (Glass et al. 2021 and Diamond et al. 2019). Protein sequences associated with the metagenomes of these datasets were retrieved from the NCBI database and searched with both bigecyhmm and METABOLIC.

• glass_dataset:
  • input: protein sequences of the organisms from the metagenomics dataset.
  • metabolic_output: output files created by METABOLIC.
  • bigecyhmm_output: output files created by bigecyhmm.
  • ncbi_dataset.tsv: name of the organisms used in the dataset.
  • compute_metrics.py: Python script used to compute metrics by comparing the predictions on major functions of carbon, sulfur and nitrogen cycles.
  • metabolic_bigecyhmm_glass_metrics.txt: results of compute_metrics.py.
• diamond_dataset:
  • input: protein sequences of the organisms from the metagenomics dataset.
  • metabolic_output: output files created by METABOLIC.
  • bigecyhmm_output: output files created by bigecyhmm.
  • ncbi_dataset.tsv: name of the organisms used in the dataset.
  • compute_metrics.py: Python script used to compute metrics by comparing the predictions on major functions of carbon, sulfur and nitrogen cycles.
  • metabolic_bigecyhmm_diamond_metrics.txt: results of compute_metrics.py.
• Figures created by this comparison: comparison_metabolic.pdf and comparison_metabolic.svg.

 - comparison_metagenomes.zip: compressed archive containing the comparison between Tabigecy predictions and PICRUSt2, Tax4Fun2 against metagenomes. It is based on the comparison made in PICRUST2 manuscript (source data can be found at this GitHub repository). The analysis was performed on the seven datasets present in the manuscript (blueberry, cameroon, hmp, indian, mammal, ocean and primate). Content of the archive:

• input_tabigecy: input files for tabigecy (abundance and taxonomic affiliation files) and the 16S rRNA sequences used by FROGs to generate the taxonomic affiliaitons.
• output_tabigecy.zip: zip archive containing output files created by Tabigecy.
• comparison:
  • 16S_vs_MGS_metrics: computed metrics for each dataset for the different tools considered.
  • mgs_validation: expected ground truth obtained by creating functional profiles from the metagenomes using HUMAnN2. Created by the authors of PICRUSt2 manuscript.
  • panfp_out: PanFP outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
  • picrust1_out: PICRUSt1 outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
  • picrust2_out: PICRUSt2 outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
  •  picrust2_scrambled_out: PICRUSt2 outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
  • piphillin_out: Piphillin outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
    
  • possible_ko: Possible KO predicted by the different methods.
  • tabigecy_out: output of tabigecy (corresponding to the "output_3_visualisation/function_abundance/hmm_functional_profile.tsv" in the output folder of the dataset).
  • tax4fun2_out: Tax4fun2 outputs on the seven datasets generated by the authors of PICRUSt2 manuscript.
  • 16S_vs_MGS_KO_validations.R: R script to compute the comparison metrics according to the folder and ko.txt.gz file present in this folder.
  • all_ko_f_score_df.pdf: output file generated by  the Python script create_boxplot.py.
  • create_boxplot.py: Python script using pandas, seaborn and matplotlib to generate all_ko_f_score_df.pdf.
  • ko.txt.gz: KO database.
• Figures created by this comparison: comparison_picrust.pdf and comparison_picrust.svg.

- software.zip: compressed archive containing the code of the tools developed and used in the article:

• bigecyhmm-0.1.6.zip: contains the code of bigecyhmm version 0.1.6 used in the article.
• esmecata-0.6.4.zip: contains the code of esmecata version 0.6.4 used in the article.
• tabigecy-0.1.1.zip: contains the code of tabigecy version 0.1.1 used in the article.
• METABOLIC-master.zip: contains the code of METABOLIC version 0.4 used in the article. 

- db_used_by_metabolic.zip: database files used with METABOLIC to predict cycle functions. All the corresponding files were obtained through the run_to_setup.sh script of METABOLIC and were put in the METABOLIC code folder. METABOLIC HMM database, METABOLIC template and Motif files can be found in the METABOLIC-master.zip file.

Perform experiments for the salt caverns datasets

Analyses performed in the article can be reproduced by running tabigecy on the input files of the article_data archive.

To do so, add the EsMeCaTa input file (either bordenave_et_al_2013.tsv or schwab_et_al_2022.tsv`) to the parameter --infile and the abundance file (either bordenave_et_al_2013_abundance.csv or  schwab_et_al_2022_abundance.tsv) to the parameter --inAbundfile. The precomputed database is required and can be given with the parameter --precomputedDB. The database can be downloaded from Zenodo.

Commands for the Bordenave et al. dataset:

 nextflow run ArnaudBelcour/tabigecy --infile bordenave_et_al_2013.tsv --inAbundfile bordenave_et_al_2013_abundance.csv --precomputedDB /path/to/esmecata_database.zip --outputFolder output_bordenave --coreBigecyhmm xx

Commands for the Schwab et al. dataset:

nextflow run ArnaudBelcour/tabigecy --infile schwab_et_al_2022.tsv --inAbundfile schwab_et_al_2022_abundance.tsv --precomputedDB /path/to/esmecata_database.zip --outputFolder output_schwab --coreBigecyhmm xx

To decrease the runtime of the workflow, it is advised to give several cores to `--coreBigecyhmm xx`. With 5 cores, the runtime of the workflow is around 13 minutes.

To create polar plots, call the two Python scripts at the same location where the input files and output folder are (inside article_data folder):

python3 bordenave_create_figure_article.py

python3 schwab_create_figure_article.py

To create the PCA and correlation plots, launch the R script on the same location:

Rscript create_pca.R

Perform comparison experiments

METABOLIC comparison

For both metagenomic datasets, input folders (containing fasta files of the organism protein sequences) were used as input for both bigecyhmm and METABOLIC.

bigecyhmm -i input -o bigecyhmm_output -c 10

perl METABOLIC-G.pl -in input -o metabolic_output -t 10 -kofam-db small

Then to compute the metrics, launch the Python script:

python3 compute_metrics.py

It compares the functions predicted in the METABOLIC_Figures_Input/Nutrient_Cycling_Diagram_Input folder of METABOLIC output folder to the functions present in the diagram_input folder of bigecyhmm output folder.

Metagenome comparison

FROGs was used on the 16S rRNA sequences of the seven datasets (from the source data of the GitHub repository) to predict taxonomic affiliations and generate the "*_esmecata.tsv" file. The abundance files were retrieved from the GitHub repository. All these files were moved to the input_tabigecy folder. 

Tabigecy was applied to the seven datasets using files from the input_tabigecy folder, for example:

 nextflow run ArnaudBelcour/tabigecy --infile input_tabigecy/blueberry_esmecata.tsv --inAbundfile input_tabigecy/blueberry_abundance.tsv --precomputedDB /path/to/esmecata_database.zip --outputFolder blueberry_output --coreBigecyhmm xx

For each dataset, the file "dataset_output/output_3_visualisation/function_abundance/hmm_functional_profile.tsv" was retrieved from tabigecy output folder and added to the folder tabigecy_out. Then the comparison metrics were computed using (with the folders containing the KO predictions for the other tools):

Rscript 16S_vs_MGS_KO_validations.R

Using the result files in folder 16S_vs_MGS_metrics, a boxplot is generated by the Python script:

python3 create_boxplot.py

Metadata

The experiments were performed with the following tool versions: