MedFSN-Bench: A Benchmark Corpus for Medical Device Safety

MedFSN-Bench is a benchmark corpus constructed from Field Safety Notices (FSNs) issued within medical device vigilance systems. FSNs are manufacturer-issued regulatory communications describing device malfunctions, associated risks, health hazards, and corrective actions identified during post-market surveillance.

The corpus was developed to support research on Natural Language Processing (NLP) and Large Language Models (LLMs) for medical device safety analysis.

MedFSN-Bench contains three benchmark tasks:

Task 1: Medical Device Classification

• Input: Intended-use descriptions and candidate label space
• Output: Device category and subcategory

Task 2: Health Effect Prediction

• Input: Device problem descriptions, risk descriptions, and candidate label space derived from IMDRF Annex F terminology
• Output: Appropriate IMDRF Annex F health effect terms

Task 3: Device Problem Classification

• Input: Device problem descriptions, risk descriptions, corrective actions, and candidate label space derived from IMDRF Annex A terminology
• Output: Appropriate IMDRF Annex A device problem terms

The corpus was constructed from publicly available Field Safety Notices collected from the German medical device vigilance system between 2013 and 2025. A hybrid human–LLM annotation workflow was employed, followed by LLM-as-a-Judge verification and additional human quality checks on a subset of the data.

Corpus Statistics:

• Total instances: 874
• Task 1: 612 instances
• Task 2: 192 instances
• Task 3: 70 instances

The dataset is intended for benchmarking NLP and LLM systems in medical device safety surveillance, regulatory document understanding, post-market surveillance, and medical device vigilance research.

This resource is released to support academic research, education, and reproducible evaluation of AI systems in the medical device safety domain. The benchmark was derived from publicly available regulatory safety communications and is intended solely for scientific, educational, and non-commercial research purposes. Ownership and copyright of the original source documents remain with their respective rights holders. Users are responsible for complying with any applicable licensing, copyright, and usage conditions associated with the original source materials.

Citation

If you use MedFSN-Bench in your research, please cite the associated publication:

Md Moin Uddin and Mouzhi Ge. 2026. MedFSN-Bench: A Benchmark Corpus for Medical Device Safety. In Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (BCB '26), June 30–July 03, 2026, Rende (CS), Italy. ACM, New York, NY, USA.

DOI: https://doi.org/10.1145/3807503.3819497

BibTeX:

@inproceedings{uddin2026medfsnbench,
author = {Uddin, Md Moin and Ge, Mouzhi},
title = {MedFSN-Bench: A Benchmark Corpus for Medical Device Safety},
booktitle = {Proceedings of the 17th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (BCB '26)},
year = {2026},
location = {Rende (CS), Italy},
publisher = {Association for Computing Machinery},
doi = {10.1145/3807503.3819497},
isbn = {979-8-4007-2653-8/2026/06}
}