Semantic Shift NLP

This release provides a reproducible research pipeline for diachronic lexical semantic change detection using contextualized language models.

Contents

• End-to-end preprocessing pipeline for temporally separated news corpora
• Target lexicon construction via frequency and stopword filtering
• Sentence-level context extraction
• Contextual embedding generation using BERT/DistilBERT
• Semantic drift quantification using cosine-based displacement metrics
• Ranked semantic shift outputs with qualitative analysis support

Reproducibility

All scripts, dependencies, and configurations required to reproduce the reported results are included in this release. The repository is structured to support extension to additional time slices, languages, and transformer architectures.

Citation

A CITATION.cff file is provided at the repository root for academic referencing.