Named Entity Recognition (NER) from Scholarly Literatures

Open Notebook in Google Colab

Introduction

The Named Entity Recognition (NER) framework has been designed to extract important entities such as countries, species, diseases, and genes from scientific and technical documents with a strong focus on accuracy. It has been built with flexibility and scale in mind, the NER pipeline uses a modular approach, allowing each entity type to be processed independently. It is also extendable, meaning it can be easily adapted to new types of entities or different academic domains.

Key features of the NER Pipeline:

•  Modular – separate logic for each entity type
•  Scalable – capable of processing large corpora
•  Extendable – easily customized for new entity types or domains

Why Use Named Entity Recognition?

NER helps to turn complex, unstructured scientific text into structured, machine-readable data. This is especially valuable in fields like biology, climate science, medicine and policy, where identifying and classifying real-world concepts is crucial.

Key benefits include:

• Creating structured indexes of scientific knowledge
•  Supporting automated literature reviews and meta-analyses
•  Tracking geographical and biological entities in research
• Analyzing relationships between entities through co-occurrence and network visualizations
• Connecting extracted information with ontologies and knowledge graphs for deeper analysis

This pipeline offers a powerful foundation for research, enabling faster insights and more efficient handling of domain-specific literature.

Architecture

The pipeline is comprised of the following stages:

1.      Preprocessing

a.       PDF-to-text conversion

b.      Discarding the reference

c.       Sentence segmentation

2.      Entity Recognition

a.       Selection of the most appropriate NER model for each entity type

b.      Support for both general-purpose and biomedical-specific corpora

3.      Entity Validation

a.       Filter and verify using curated databases, regex, APIs, and ontology checks

b.      Multiple layers of validation reduce false positives and enhance accuracy

4.      Output Generation

a.       Structured CSV files

b.      Visualizations: word clouds, co-occurrence plots, interactive graphs

NER Models and Validation Methods (Entity-wise)

Each entity class has its own specialized extraction strategy, including tailored models and validation steps.

1.      Country Extraction

a.       NER Model: spaCy en_core_web_lg

 A large English NLP model capable of identifying country and geopolitical names.

b.      Validation: Babel’s Official Country List

 Filters extracted names against a curated list of standardized countries to remove false positives (e.g., city names or common words).

2.      Species Extraction

a.       NER Model: scispaCy en_core_sci_md

 Trained on scientific literature, effective at identifying Latin binomial names like Homo sapiens, E. coli.

b.      Validation:

  i.      Regex Pattern Matching for Genus–species format

  ii.      GBIF API verification to confirm species existence and correctness

3.      Disease Extraction

a.       NER Model: scispaCy en_ner_bc5cdr_md

 Specialized in biomedical disease and chemical terms.

b.      Validation:

  i.      Disease Ontology (DOID)

 ii.      Fuzzy Matching (≥95%) using rapidfuzz to catch near-matches and spelling variants

4.      Gene Extraction

a.       NER Model: scispaCy en_ner_bionlp13cg_md. Recognizes gene/protein names, cell lines, and biomedical entities

b.      Validation: HGNC complete gene symbol set (TSV). Ensures names are real, up-to-date, and standardized

Output and Visualizations

The pipeline generates structured data and visual interpretations:

a)      CSV Files

•  Frequency tables for each entity type

b)      Word clouds

• Based on frequency data
• Highlight high-frequency terms

c)      Co-occurrence Plots

• Graphs showing how entities co-appear in the corpus
• Help discover conceptual linkages

d)      Interactive Network Graphs

•  Force-directed layouts
• Nodes represent entities, edges represent co-occurrence strength
• Suitable for presentations and exploratory analysis

Conclusion: The framework presented for entity extraction from a scholarly literature corpus has provided insights into the use of natural language processing techniques and pre-trained models in a smarter, faster, and more structured manner. The results obtained helped extract entities such as plants, chemicals, locations, genes, etc., which will further aid in contextualizing and conducting downstream analysis for information retrieval, knowledge graph creation, and understanding the interrelationships between different entities.