Generalizable and Scalable Multistage Biomedical Concept Normalization Leveraging Large Language Models

Background: Biomedical entity normalization is critical to biomedical research because the richness of
free-text clinical data, such as progress notes, can often be fully leveraged only after translating words and
phrases into structured and coded representations suitable for analysis. Large Language Models (LLMs),
in turn, have shown great potential and high performance in a variety of natural language processing (NLP)
tasks, but their application for normalization remains understudied.
Methods: We applied both proprietary and open-source LLMs in combination with several rule-based nor-
malization systems commonly used in biomedical research. We used a two-step LLM integration approach,
(1) using an LLM to generate alternative phrasings of a source utterance, and (2) to prune candidate UMLS
concepts, using a variety of prompting methods. We measure results by Fβ , where we favor recall over
precision, and F1.
Results: We evaluated a total of 5,523 concept terms and text contexts from a publicly available dataset
of human-annotated biomedical abstracts. Incorporating GPT-3.5-turbo increased overall Fβ and F1 in nor-
malization systems +16.5 and +16.2 (OpenAI embeddings), +9.5 and +7.3 (MetaMapLite), +13.9 and +10.9
(QuickUMLS), and +10.5 and +10.3 (BM25), while the open-source Vicuna model achieved +20.2 and
+21.7 (OpenAI embeddings), +10.8 and +12.2 (MetaMapLite), +14.7 and +15 (QuickUMLS), and +15.6
and +18.7 (BM25).
Conclusions: Existing general-purpose LLMs, both propriety and open-source, can be leveraged to greatly
improve normalization performance using existing tools, with no fine-tuning.