BERT-MultiCulture-DEID: A Set of Models Fine-Tuned on identifiers from 9 Language Variants

Background

De-identification is the process of removing PII from data to protect individual privacy. This step is crucial in healthcare research, where clinical texts often contain sensitive details such as patient names, addresses, contact information, and other identifiers. Proper de-identification enables the safe use of clinical notes for research, analytics, and machine learning without compromising patient confidentiality.

Traditional rule-based or heuristic approaches, as well as models fine-tuned only on English identifiers, often lack generalization across identifiers from different cultures. For example, a model trained on English PII may struggle to recognize names, addresses, or other identifiers in Mandarin, Spanish, Hindi, French, or Bengali text, resulting in missed or incorrectly masked PII. Handling this diversity and multicultural variability is therefore a critical challenge in clinical de-identification.

To address these challenges, we developed a BERT-MultiCulture-DEID set of models (BERT, ClinicalBERT, and ModernBERT), fine-tuned on identifiers extracted from nine language variants, including English (US, Great Britain, Australia, Canada), Mandarin, Spanish, Hindi, French, and Bengali. These models use contextual embeddings to accurately detect and mask PII.

Model Description

The models are based on a bidirectional transformer architecture and fine-tuned for token classification. The provided inference pipeline is implemented in Python using the Hugging Face Transformers library and is compatible with both CPU and GPU execution. These are binary classification models for token-level de-identification, where a label of 0 indicates that the token is not PII and a label of 1 indicates that the token is PII. The categories of identifiers include names, phone numbers, hospitals, cities, states, addresses, countries, companies, universities, dates, emails, and other sensitive identifiers.

We fine-tuned 3 variants of BERT:

• BERT
• ClinicalBERT
• ModernBERT

Technical Implementation

The models were fine-tuned on 500 notes from each language variant for 10 epochs with a learning rate of 2 × 10-⁵ and weight decay of 0.01 on the MIMIC-III dataset. During preprocessing, masked identifiers in the dataset were replaced with synthetic surrogates generated using the Faker library to preserve text structure. Please see the associated publication "Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches" for more details.

Usage

Access to the trained models is restricted until official publication on PhysioNet; interested researchers may contact the corresponding author (Dr. Mohamed Abdalla, email: mabdall2@ualberta.ca) to request access, subject to appropriate approvals.

Citing our work

Please use the following BibTex entry to cite us if you use our models or any of the associated analyses:

@inproceedings{zambare2026fair,
  title     = "Towards Fair and Efficient De-identification: Quantifying the Efficiency and Generalizability of De-identification Approaches",
  author    = "Zambare, Noopur and Aghakasiri, Kiana and Lin, Carissa and Ye, Carrie and Mitchell, J. Ross and Abdalla, Mohamed",
  booktitle = "Findings of the Association for Computational Linguistics: EACL 2026",
  year      = "2026",
  publisher = "Association for Computational Linguistics"
}