# Preparation Of Annotation Resources
This document provides links to download the annoation sources used in aiDIVA. If it is necessary to further prepare these files it is explained and shown in the respective section with a coode snippet.
\
\
For some of the annotation sources there are no GRCh38/hg38 files available in these cases we lifted the GRCh37/hg19 sources manually to the new assembly using the tool [CrossMap](https://github.com/liguowang/CrossMap). For this manual liftover we also provide code snippets.


## Reference Genome Assembly
GRCh38:
<br>
[http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz](http://hgdownload.cse.ucsc.edu/goldenPath/hg38/bigZips/hg38.fa.gz)

```
gunzip hg38.fa.gz
bgzip hg38.fa

samtools dict hg38.fa.gz
samtools faidx hg38.fa.gz
```

## Annotation Databases
INFO: Please be adviced that some of the annotation sources aiDIVA uses are only free for non-commercial and academic use!

If for some reason you choose to exclude one or more of the Annotation sources shown here you have to make sure to modify the source code accordingly. If an annotation source is used as a feature for the random forest (RF) and you exclude it in your analysis you have to train a new model before you can use the model RF in aiDIVA.


### CADD
GRCh38:
<br>
https://kircherlab.bihealth.org/download/CADD/v1.6/GRCh38/whole_genome_SNVs.tsv.gz

```
wget -c -O grch38_whole_genome_SNVs.tsv.gz https://kircherlab.bihealth.org/download/CADD/v1.6/GRCh38/whole_genome_SNVs.tsv.gz

python3 prepare_CADD_vcf.py grch38_whole_genome_SNVs.tsv.gz grch38_CADD_snvs_v16.vcf

bgzip grch38_CADD_snvs_v16.vcf
tabix -p vcf grch38_CADD_snvs_v16.vcf.gz

rm grch38_whole_genome_SNVs.tsv.gz
```


### Capice
GRCh38:
<br>
https://zenodo.org/record/3928295/files/capice_v1.0_build37_snvs.tsv.gz

Needs manual liftover!!!
https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz

```
wget -c https://zenodo.org/record/3928295/files/capice_v1.0_build37_snvs.tsv.gz

python3 prepare_Capice_vcf.py capice_v1.0_build37_snvs.tsv.gz grch37_capice_v1_snvs.vcf

bgzip grch37_capice_v1_snvs.vcf
tabix -p vcf grch37_capice_v1_snvs.vcf.gz

CrossMap.py vcf hg19ToHg38.over.chain.gz grch37_capice_v1_snvs.vcf.gz hg38.fa.gz grch38_capice_v1_snvs.vcf

bgzip grch38_capice_v1_snvs.vcf
tabix -p vcf grch38_capice_v1_snvs.vcf.gz

rm grch37_capice_v1_snvs.vcf.gz
rm capice_v1.0_build37_snvs.tsv.gz
```


### ClinVar
GRCh38:
<br>
https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz

```
wget -c https://ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/clinvar.vcf.gz

python3 prepare_ClinVar_vcf.py clinvar.vcf.gz grc38_clinvar.vcf

bgzip grch38_clinvar.vcf
tabix -p vcf grch38_clinvar.vcf.gz

rm clinvar.vcf.gz
```


### Condel
!!! Account needed !!!
Please first download the FannsDB database from here: [https://bbglab.irbbarcelona.org/fannsdb/](https://bbglab.irbbarcelona.org/fannsdb/) 
\
\
Afterwards the `prepare_Condel_vcf.py` script found in the _scripts_ folder can be used to create the VCF annotation file. For the annotation the VCF file needs to be _bgzipped_ and _indexed_

Needs manual liftover!!!
https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz

The following code snippet shows the necessary commands

GRCh38:

```
python3 prepare_Condel_vcf.py fannsdb.tsv.gz grch37_fannsDB_Condel.vcf

bgzip grch37_fannsDB_Condel.vcf
tabix -p vcf grch37_fannsDB_Condel.vcf.gz

CrossMap.py vcf hg19ToHg38.over.chain.gz grch37_fannsDB_Condel.vcf.gz hg38.fa.gz grch38_fannsDB_Condel.vcf

bgzip grch38_fannsDB_Condel.vcf
tabix -p vcf grch38_fannsDB_Condel.vcf.gz

rm grch37_fannsDB_Condel.vcf.gz
rm fannsdb.tsv.gz
```


### Eigen phred
GRCh38:
\
https://zenodo.org/record/3256671/files/Eigen_hg38_coding_annot_04092016.tab.gz

```
wget -c https://zenodo.org/record/3256671/files/Eigen_hg38_coding_annot_04092016.tab.gz

python3 prepare_EigenPhred_vcf.py Eigen_hg38_coding_annot_04092016.tab.bgz grch38_eigen_phred_coding.vcf

bgzip grch38_eigen_phred_coding.vcf
tabix grch38_eigen_phred_coding.vcf.gz

rm Eigen_hg38_coding_annot_04092016.tab.bgz
```


### FATHMM XF
GRCh38:
<br>
https://fathmm.biocompute.org.uk/fathmm-xf/fathmm_xf_coding_hg38.vcf.gz

```
wget -c https://fathmm.biocompute.org.uk/fathmm-xf/fathmm_xf_coding_hg38.vcf.gz

python3 prepare_Fathmm_XF_vcf.py fathmm_xf_coding_hg38.vcf.gz grch38_fathmm_xf_coding.vcf

bgzip grch38_fathmm_xf_coding.vcf
tabix -p vcf grch38_fathmm_xf_coding.vcf.gz

rm fathmm_xf_coding_hg38.vcf.gz
```


### GnomAD (oe_lof and homAF)
GRCh38:
\
https://storage.googleapis.com/gcp-public-data--gnomad/release/2.1.1/constraint/gnomad.v2.1.1.lof_metrics.by_gene.txt.bgz

Needs manual liftover!!!
https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz

```
wget -c https://storage.googleapis.com/gnomad-public/release/2.1.1/constraint/gnomad.v2.1.1.lof_metrics.by_gene.txt.bgz

zcat gnomad.v2.1.1.lof_metrics.by_gene.txt.bgz | sed -n '1d;p' | awk -v OFS="\t" -F "\t" '{ print $$75,$$76,$$77,$$24}' > gnomad_OE.bed
cat gnomad_OE.bed | sort -k1,1 -k2,2n -k3,3n -t '	' | sed '/NA/s/\bNA//g' > gnomAD_OE_sorted.bed

CrossMap.py vcf hg19ToHg38.over.chain.gz gnomAD_OE_sorted.bed hg38.fa.gz gnomAD_OE_grch38.bed

cat gnomad_OE_grch38.bed | sort -k1,1 -k2,2n -k3,3n -t '	' | sed '/NA/s/\bNA//g' | awk 'NF==4' > gnomAD_OE_grch38_sorted.bed

rm gnomad.v2.1.1.lof_metrics.by_gene.txt.bgz
rm gnomAD_OE_sorted.bed
```

\
GRCh38:
\
https://storage.googleapis.com/gcp-public-data--gnomad/release/2.1.1/liftover_grch38/vcf/genomes/gnomad.genomes.r2.1.1.sites.liftover_grch38.vcf.bgz

```
wget -c https://storage.googleapis.com/gcp-public-data--gnomad/release/2.1.1/liftover_grch38/vcf/genomes/gnomad.genomes.r2.1.1.sites.liftover_grch38.vcf.bgz

python3 prepare_gnomAD_vcf.py gnomad.genomes.r2.1.1.sites.liftover_grch38.vcf.bgz grch38_gnomAD_genomes_r211.vcf

bgzip grch38_gnomAD_genomes_r211.vcf
tabix -p vcf grch38_gnomAD_genomes_r211.vcf.gz
```
\
-- NOTE -- 
We switched to the gnomAD 3.1.2 genome dataset (unfortunately for this data you have to prepare each chromosome (1-22 and X) separately the same way as described for v2.1.1 and combine them afterwards into a huge VCF file for annotation):


### \[optional\] HGMD (needs license)
The possibility to use HGMD in aiDIVA is optional due to the fact that you need a license for it. If you choose to include HGMD in the aiDIVA analysis you can use the public/professional version of HGMD.


### Low Confidence Regions
GRCh38:
<br>
https://github.com/imgag/megSAP/raw/master/data/misc/low_conf_regions.bed

```
wget -c -O grch38_low_conf_region.bed https://github.com/imgag/megSAP/raw/master/data/misc/low_conf_regions.bed
```


### MutationAssessor
GRCh38:
<br>
https://web.archive.org/web/20160802103335/http://mutationassessor.org/r3/MA_scores_rel3_hg19_full.tar.gz

Needs manual liftover!!!
https://hgdownload.cse.ucsc.edu/goldenpath/hg19/liftOver/hg19ToHg38.over.chain.gz
```
wget -c https://web.archive.org/web/20160802103335/http://mutationassessor.org/r3/MA_scores_rel3_hg19_full.tar.gz
tar -xzf MA_scores_rel3_hg19_full.tar.gz

python3 prepare_MutationAssessor_vcf.py $(ls -m MA_scores_rel3_hg19_full/*.csv | tr -d '[:space:]') grch37_precomputed_MutationAssessor_unsort.vcf

# use VcfSort from ngs-bits to make sure that the created VCF is sorted
ngs-bits/VcfSort -in grch37_precomputed_MutationAssessor_unsort.vcf -out grch37_precomputed_MutationAssessor.vcf

bgzip grch37_precomputed_MutationAssessor.vcf
tabix -p vcf grch37_precomputed_MutationAssessor.vcf.gz

CrossMap.py vcf hg19ToHg38.over.chain.gz grch37_precomputed_MutationAssessor.vcf.gz hg38.fa.gz grch38_precomputed_MutationAssessor.vcf

bgzip grch38_precomputed_MutationAssessor.vcf
tabix -p vcf grch38_precomputed_MutationAssessor.vcf.gz

rm MA_scores_rel3_hg19_full.tar.gz
rm -r MA_scores_rel3_hg19_full/
rm grch37_precomputed_MutationAssessor_unsort.vcf
rm grch37_precomputed_MutationAssessor.vcf.gz
```


### PhastCons
For PhastCons is no further preparation necessary. Just make sure that it is correctly specified in the YAML configuration file.

GRCH38:
\
phastCons_primate: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons17way/hg38.phastCons17way.bw

phastCons_mammal: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons30way/hg38.phastCons30way.bw

phastCons_vertebrate: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phastCons100way/hg38.phastCons100way.bw

### PhyloP
For PhyloP is no further preparation necessary. Just make sure that it is correctly specified in the YAML configuration file.
\
\
GRCh38:
\
phyloP_primate: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phyloP17way/hg38.phyloP17way.bw

phyloP_mammal: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phyloP30way/hg38.phyloP30way.bw

phyloP_vertebrate: https://hgdownload.soe.ucsc.edu/goldenPath/hg38/phyloP100way/hg38.phyloP100way.bw


### Segment Duplication
GRCh38:
\
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/genomicSuperDups.txt.gz

```
wget -c -O hg38.genomicSuperDups.txt.gz https://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/genomicSuperDups.txt.gz

zcat hg38.genomicSuperDups.txt.gz | cut -f2,3,4,27 > grch38_segmentDuplication_unsort.bed

grep -v '#' grch38_segmentDuplication_unsort.bed | sort -k1,1 -k2,2n -k3,3n -t $'\t' > grch38_segmentDuplication.bed

rm hg38.genomicSuperDups.txt.gz
rm grch38_segmentDuplication_unsort.bed
```


### SimpleRepeat
GRCh38:
\
https://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/simpleRepeat.txt.gz

```
wget -c -O hg38.simpleRepeat.txt.gz https://hgdownload.soe.ucsc.edu/goldenPath/hg38/database/simpleRepeat.txt.gz

zcat hg38.simpleRepeat.txt.gz | cut -f2,3,4,11 > grch38_simpleRepeat_unsort.bed

grep -v '#' grch38_simpleRepeat_unsort.bed | sort -k1,1 -k2,2n -k3,3n -t '	' > grch38_simpleRepeat.bed

rm hg38.simpleRepeat.txt.gz
rm grch38_simpleRepeat_unsort.bed
```


### RepeatMasker
GRCh38:
\
http://www.repeatmasker.org/genomes/hg38/RepeatMasker-rm405-db20140131/hg38.fa.out.gz

```
wget -c http://www.repeatmasker.org/genomes/hg38/RepeatMasker-rm405-db20140131/hg38.fa.out.gz

python3 prepare_RepeatMasker_bed.py hg38.fa.out.gz grch38_repeatmasker.bed

rm hg38.fa.out.gz
```


### REVEL
GRCh38:
\
https://zenodo.org/records/7072866/files/revel-v1.3_all_chromosomes.zip

```
wget -c https://zenodo.org/records/7072866/files/revel-v1.3_all_chromosomes.zip

unzip revel-v1.3_all_chromosomes.zip

python3 prepare_REVEL_vcf.py revel_with_transcript_ids grch37_revel_v13_unsort.vcf grch38_revel_v13_unsort.vcf

ngs-bits/VcfSort -in grch38_revel_v13_unsort.vcf -out grch38_revel_v13.vcf

bgzip grch38_revel_v13.vcf
tabix -p vcf grch38_revel_v13.vcf.gz

rm revel-v1.3_all_chromosomes.zip
rm revel_with_transcript_ids
rm grch38_revel_v13_unsort.vcf
```


### SpliceAI
GRCh38
\
!!! Account needed !!!
\
\
Manually download the following files `spliceai_scores.masked.snv.hg38.vcf.gz` and `spliceai_scores.masked.indel.hg38.vcf.gz`
\
[https://basespace.illumina.com/s/otSPW8hnhaZR](https://basespace.illumina.com/s/otSPW8hnhaZR)

```
tabix -p vcf spliceai_scores.masked.snv.hg38.vcf.gz
tabix -p vcf spliceai_scores.masked.indel.hg38.vcf.gz
```


### AlphaMissense
GRCh38
\
!!! Google Account needed !!!
\
Manually donwload the following files `AlphaMissense_hg38.tsv.gz`
\
https://console.cloud.google.com/storage/browser/dm_alphamissense

```
tabix -p vcf AlphaMissense_hg38.tsv.gz
```
