Word embeddings of PubMed abstracts

We developed a software package called marea (marea adamantly resists egregious acronyms) that implements all necessary natural language processing (NLP) steps to prepare the titles and abstracts of PubMed articles as input for word embedding algorithms. marea filters PubMed articles for relevance and applies PubTator Central concept recognition to the titles and abstracts of relevant articles. After concept replacement, the final phase eliminates punctuation and stop words, and reduces the vocabulary size.

Filtering relevant PubMed articles

NCBI's FTP site makes available gzipped XML files containing titles, abstracts, and metadata for all PubMed articles. marea downloads the annual baseline and daily update files, and parses them to extract the fields of interest for each article: PubMed ID, MeSH descriptors (if any), keywords (if any), and year of publication. For entries that have multiple dates with different years, the earliest one is recorded. To select articles for a particular search, the marea user provides a set of high-level MeSH descriptor ids. The MeSH descriptors defining the scope of the research described herein were D009369 (Neoplasms) and D011494 (PKs). Any article marked with at least one of these descriptors or any subcategory of these descriptors is considered relevant. An article is also judged relevant if it has a keyword that matches a label or synonym of the search descriptors or their subcategories. Some PubMed articles have neither MeSH descriptors nor keywords; some have no abstract. Any article that has no abstract is irrelevant for the search regardless of its MeSH descriptors or keywords. 

Concept replacement

The original word2vec method operates on individual words (tokens). However, many medical concepts span multiple tokens. For instance, non-small-cell lung carcinoma would be treated by word2vec as three or five tokens (depending on how the hyphen is handled), but it represents a single medical concept. For this reason, recent approaches collapse multi-word concepts into a single token prior to embedding by replacing the multiword concepts with a single concept id. For instance, non-small-cell lung carcinoma can be replaced by its MeSH id D002289.

PubTator Central from the National Center for Biotechnology Information (National Library of Medicine) offers data for concept recognition in PubMed articles. Annotated categories include chemicals, diseases, genes, cell lines, SNPs, and species, as well as other categories marea does not track, such as DNAMutation and ProteinMutation. Using PubTator Central character offsets, the software replaces each phrase recognized in the title or abstract with the identifier of the corresponding concept. Diseases and chemical names are normalized to MeSH ids, genes and proteins to NCBI Gene ids, cell lines to Cellosaurus, SNPs to dbSNP RS ids, and species to NCBI Taxonomy ids. The one exception is the human species, NCBI taxon 9606, which we decided to skip. PubTator Central annotations would have substituted 9606 for man, woman, boy, girl, father, mother, patient, and similar words. We chose to preserve the distinctions of gender and age expressed in terms for humans, as these factors are certainly significant in the medical context.

Text preprocessing

After concept replacement, marea cleans up the text of PubMed titles and abstracts to make it more suitable for word embedding. The tokenizer deletes all punctuation symbols, including hyphens and underscores within words: the parts of a compound word become separate tokens. marea removes stop words, whether lowercase or capitalized. Uppercase acronyms of length ≥ 2, even those that coincide with stop words, are not changed. For example, the acronym ALL (acute lymphocytic leukemia) is retained while all and All are eliminated. We started with the stop word list for English in the Natural Language Toolkit (nltk version 3.5) Python library and added some new stop words. Any letter of the alphabet that occurs as a single-character token is a stop word. To further reduce the size of the vocabulary, tokens that remain after stop word removal are lemmatized with the WordNet  lemmatizer from nltk. The lemmatizer reduces words to their base form, for example plural nouns are simplified to the singular. (Unlike stemming, lemmatizing a word always returns a complete word, not a truncated word stem.) The last step of text preprocessing converts everything to lowercase, to avoid near-duplicate embeddings for upper-, lower-, and mixed-case forms of the same word.

Word embedding

The word embedding method based on the word2vec algorithm is performed on the preprocessed corpus to embed words to vectors. We used the EMBeddInG GENerator (embiggen), a Python 3 software library developed by our group for word embedding based on word2vec and node embedding based on the node2vec algorithm. In the current project, the skip-gram model was used for word2vec with the parameters window size = 5, minimum count (minimum word frequency) = 5, batch size = 128, negative samples = 20 and dimension = 100.