Sensitive remote homology search by local alignment of small positional embeddings from protein language models

This repository contains model weights for ESM-2 3B 3Di, predicted profiles and 3Di sequences from the Pfam 32 clustered splits, and other data necessary to reproduce the analysis in the paper "Sensitive remote homology search by local alignment of small positional embeddings from protein language models" (https://www.biorxiv.org/content/10.1101/2023.07.26.550718v1). 

The Pfam 32 clustered splits were originally developed by Bileschi et al. 2022. If you use those datasets please cite the appropriate papers:

• Bileschi, Maxwell L., David Belanger, Drew H. Bryant, Theo Sanderson, Brandon Carter, D. Sculley, Alex Bateman, Mark A. DePristo, and Lucy J. Colwell. “Using Deep Learning to Annotate the Protein Universe.” Nature Biotechnology, February 21, 2022, 1–6. https://doi.org/10.1038/s41587-021-01179-w.

• Mistry, Jaina, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik L L Sonnhammer, Silvio C E Tosatto, et al. “Pfam: The Protein Families Database in 2021.” Nucleic Acids Research 49, no. D1 (January 8, 2021): D412–19. https://doi.org/10.1093/nar/gkaa913.

Details of individual files:

ESM-2_3B_3Di.pt: Trained weights for ESM-2 3B 3Di model, which converts protein sequences to 3Di sequences with approximately 64% accuracy.

pfamN_family_clans.tsv: Metadata on training and test sequences from Pfam 32 clustered splits, with columns:
sequence_name    split    family_id    clan

pfamN_pids.tsv: % identity of each test sequence from test split to the most similar sequence in the train split from the same family.

search_results.tar.gz:  

•     1_test_hits_hmmscan.tsv: aa seq query, aa hmmer profile (from re-aligned training sequence MSAs) target 
•     2_test_hits_phmmer.tsv: aa seq query, aa seq target
•     3_test_esm2_3B_msas_pnone_hmmsearch.tsv: pred hmmer profile (hmmbuild --pnone) query, aa seq target
•     4_test_esm2_3B_msas_hmmsearch.tsv: pred hmmer profile (hmmbuild default settings) query, aa seq target
•     5_test_hits_hhblits_esm2_3B_test_pep_aligned_train.tsv: pred hhsuite profile query, aa hhsuite profile (from re-aligned training sequence MSAs) target
•     6_test_hits_hhblits_esm2_3B_test_esm2_3B_train.tsv: pred hhsuite profile query, pred hhsuite profile target
•     7_test_train_3Di_pred_foldseek.tsv: pred 3Di query, pred 3Di target (Foldseek search run with default settings)
•     8_test_train_3Di_pred_foldseek_3Di_only.tsv: pred 3Di query, pred 3Di target (Foldseek search run with --alignment-type 0 option)
•     9_test_3Di_pred_3Di_aahmmscan.tsv: pred 3Di query, pred 3Di hmmer profile target (profile created by running MAFFT using 3Di matrix on each family to generate an MSA, then running hmmbuild from HMMER3, search was run using hmmscan from HMMER3)
•     10_test_3Di_pred_3Di_hmmscan.tsv: pred 3Di query, pred 3Di hmmer3Di profile target (profile created by running MAFFT using 3Di matrix on each family to generate an MSA, then running hmmbuild from HMMER3Di, search was run using hmmscan from HMMER3Di)
   

pfamN_clustered_splits_data.tar.gz: Pfam 32 clustered splits splits converted to profile and 3Di-style positional embeddings .

•     test_3Di_pred: predicted 3Di sequences in fastas by family
•     test_3Di_pred_aligned: MAFFT MSAs by family of predicted 3Di sequences
•     test_3Di_pred.fasta: concatenation of all files in test_3Di_pred directory
•     test_3Di_pred_hmm: 3Di_hmmbuild run on each file in test_3Di_pred_aligned
•     test_esm2_3B.hhm: concatenation of all files in the test_esm2_3B_profiles directory
•     test_esm2_3B_msas: 40-sequence MSAs for each test sequence, sampled from the ESM2 3B masked probability distribution.
•     test_esm2_3B_msas.hmm: concatenation of all files in the test_esm2_3B_msas_hmm directory
•     test_esm2_3B_msas_hmm: hmmbuild on all the files in the test_esm2_3B_msas directory
•     test_esm2_3B_msas_hmm_pnone: hmmbuild --pnone on all the files in the test_esm2_3B_msas directory
•     test_esm2_3B_msas_pnone.hmm: concatenation of all files in the test_esm2_3B_msas_hmm_pnone directory
•     test_esm2_3B_profiles: hhsuite hhm files for each test sequence, sampled from the ESM2 3B masked probability distribution.
•     test_pep_aligned: MAFFT MSAs for test sequences by family,
•     test_pep.fasta: test sequence unaligned amino acid fasta
•     test_pep_hmm: hmmbuild on all files in test_pep_aligned
•     train_3Di_pred: predicted 3Di sequences in fastas by family
•     train_3Di_pred_aahmm: hmmbuild (from HMMER3) run on each file in test_3Di_pred_aligned
•     train_3Di_pred_aahmm.hmm: concatenation of all files in train_3Di_pred_aahmm
•     train_3Di_pred_aligned: MAFFT MSAs by family of predicted 3Di sequences
•     train_3Di_pred.fasta: concatenation of all files in train_3Di_pred directory
•     train_3Di_pred.hmm: concatenation of all files in train_3Di_pred_hmm
•     train_3Di_pred_hmm: 3Di_hmmbuild run on each file in train_3Di_pred_aligned 
•     train_esm2_3B_msas: 40-sequence MSAs for each train sequence, sampled from the ESM2 3B masked probability distribution. 
•     train_esm2_3B_profiles: hhsuite hhm files for each train sequence, sampled from the ESM2 3B masked probability distribution. (note that we don't include the concatenated hhm file or the related hhsuite database because it is too big for Zenodo. The database can easily be built from the MSAs and profiles)
•     train_pep_3Di_hmm: hmm files built using 3Di_hmmbuild on the amino acid MSAs.
•     train_pep_aligned: MAFFT MSAs of sequences from individual families from the training set.
•     train_pep_aligned_a3m.ffdata: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep_aligned_a3m.ffindex: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep_aligned_cs219.ffdata: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep_aligned_cs219.ffindex: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep_aligned_hhm.ffdata: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep_aligned_hhm.ffindex: hhsuite-database file for MAFFT MSAs of training sequences
•     train_pep.fasta: All unaligned train peptide sequences
•     train_pep_hmm: Directory of hmm profiles made with hmmbuild from files in the directory train_pep_aligned
•     train_pep.hmm: Concatenation of files in train_pep_hmm