Data files are extracted from zip files, parsed from csvs, and imported into a MySQL database (see parser.py).
Duplicate records are then de-duplicated by taking only the most recent version for each case ID (see dedupe.py).
Indications are normalized by matching to UMLS terms by string matching. Cross-references to Human Phenotype Ontology are pulled from UMLS and xrefs to Monarch Disease Ontology (MONDO) are pulled from MONDO using the UMLS xrefs. (See normalize_indications.py)
Drugs names are normalized first by applying a few simple string cleaning operations (strip, fix slashes and periods). Then they are attempted to be matched to rxnorm by exact string matching. Those that don't match are run against rxnorm's approximate matching service, and are accepted if the score is higher than 67/100. The matched rxnorm CUIs are then mapped to the their Ingredient level rxnorm ID. (See normalize_drugs.py)
Indications are then retrieved for each drug ingredient and filtered to require a minimum of 20 individual occurances. (See get_indications.py)