Published May 18, 2026 | Version v3
Dataset Open

TCGA-242-Pathology-Benchmark: External Validation Dataset for "The Digital Registrar"

Authors/Creators

  • 1. ROR icon China Medical University Hospital
  • 2. ROR icon Cedars-Sinai Medical Center

Description

This dataset contains the Independent External Validation (IEV) cohort used in the study:

Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs. Diagnostics. 2026; 16(11):1644.

It comprises 242 de-identified surgical pathology reports selected from The Cancer Genome Atlas (TCGA) program, extracted by Kefeli, J. & Tatonetti, N. (2025, please see reference) covering five distinct categories: Breast, Colorectal, Esophagus, Stomach, and "Others" (including synchronous bilateral cancers).
This benchmark serves as a "Gold Standard" for evaluating Large Language Models (LLMs) on clinical extraction tasks, providing raw text, expert-verified ground truth annotations, and baseline model outputs.
 

If you use this code or the dataset in your research, please cite:

Chow N-H, Chang H, Chen H-K, Lin C-Y, Liu Y-L, Tseng P-Y, Shiu L-J, Chu Y-W, Chung P-C, Chang K-P. Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs. Diagnostics. 2026; 16(11):1644. https://doi.org/10.3390/diagnostics16111644
 
BibTeX
 
@Article{diagnostics16111644,
AUTHOR = {Chow, Nan-Haw and Chang, Han and Chen, Hung-Kai and Lin, Chen-Yuan and Liu, Ying-Lung and Tseng, Po-Yen and Shiu, Li-Ju and Chu, Yen-Wei and Chung, Pau-Choo and Chang, Kai-Po},
TITLE = {Digital Registrar: A Schema-First Framework for Multi-Cancer Privacy-Preserving Pathology Abstraction via Local LLMs},
JOURNAL = {Diagnostics},
VOLUME = {16},
YEAR = {2026},
NUMBER = {11},
ARTICLE-NUMBER = {1644},
URL = {https://www.mdpi.com/2075-4418/16/11/1644},
ISSN = {2075-4418},
ABSTRACT = {Background/Objectives: Free-text surgical pathology reports hinder automated cancer registry entry and secondary analytics. This study introduces a clinically governed schema layer for interoperability, testing whether a locally-deployable Large Language Model (LLM) pipeline can deliver robust registry-grade extraction across institutions. Methods: We developed a College of American Pathologists (CAP)-aligned clinical ontology encompassing 10 cancer types, 192 per-organ scalar fields, key biomarkers, and nested structures for lymph nodes and margins. Encoded via Declarative Self-improving Python (DSPy) signatures with grammar-constrained decoding using DSPy v3.2.1, this model-agnostic pipeline was benchmarked on 893 internal reports against a pathologist-adjudicated gold standard. External validation utilized 242 The Cancer Genome Atlas (TCGA) reports. Hardware feasibility was confirmed on a single 48-gigabyte (GB) Graphics Processing Unit (GPU), ensuring suitability for privacy-preserving on-premises deployment. Results: Using the gpt-oss-20b model, the framework achieved 92.0% macro-mean exact-match accuracy on internal data, demonstrating near-perfect run-to-run reliability. Critical prognostic indicators, including breast estrogen receptor/progesterone receptor (ER/PR) (98.7%) and margin positivity (>93%), maintained high fidelity. On the external TCGA cohort, accuracy was 77.5%, rising to 88.0% after excluding structurally silent fields absent in older narratives. Operationally, the model processed reports in 40–70 s, optimally balancing speed and accuracy. Conclusions: This schema-first abstraction layer successfully decouples clinical logic from specific Artificial Intelligence (AI) models. By reliably transforming narrative reports into machine-readable structures, it establishes a portable privacy-preserving foundation for automated cancer surveillance, institutional data reuse, and future multimodal clinical systems.},
DOI = {10.3390/diagnostics16111644}
}


Files

tcga.zip

Files (956.8 kB)

Name Size Download all
md5:2a308d8ca0022b14e168371c22a97731
956.8 kB Preview Download

Additional details

Software

Programming language
Python
Development Status
Active

References

  • Kefeli, J. & Tatonetti, N. (2024). "TCGA-Reports: A machine-readable pathology report resource for benchmarking text-based AI models." Patterns 5(3). DOI: 10.1016/j.patter.2024.100933
  • Chow, N.-H., Chang, H., Chen, H.-K., et al. (2025). "A Multicancer AI Framework for Comprehensive Cancer Surveillance from Pathology Reports" medRxiv. DOI: 10.1101/2025.10.21.25338475