# Cross-lingual ICD-11 coding consistency benchmark

A reproducible benchmark for evaluating the **cross-lingual consistency of large language models (LLMs) on ICD-11 coding**. This release accompanies the manuscript *“Cross-lingual consistency of large language models in ICD-11 coding: a six-language benchmark.”*

Measurement date: **June 2026**. ICD-11 release: **2025-01** (WHO MMS).

---

## What this benchmark does
For each ICD-11 entity, the model is given the entity's official WHO definition in a given language and must return a single ICD-11 MMS code. By repeating this across six languages and several models, the benchmark measures (a) chapter-level and graded coding accuracy and (b) cross-lingual consistency (Krippendorff's α and agreement with the English assignment).

- **Entities:** 300, stratified across 23 ICD-11 chapters (residual/“unspecified”, external-cause, supplementary, and extension chapters excluded; leaf entities with a definition ≥60 characters).
- **Languages (6):** English, Spanish, French, Chinese, Arabic, Russian.
- **Models evaluated (this run):** gpt-5.4 (2026-03-05), gpt-5.4-nano (2026-03-17), gemini-2.5-flash, gemini-2.5-flash-lite, claude-sonnet-4-6, claude-haiku-4-5 (2025-10-01). Queried at temperature 0, one response per item.

> Because model scores date quickly, the benchmark is designed for re-use: the item set, prompts, and scoring code are fixed, so new models can be measured on the identical task. Please record the new models' version identifiers and the run date.

---

## Contents
```
benchmark_release/
├── README.md                      ← this file
├── code/
│   ├── pilot_icd11_v4.py          ← end-to-end runner (sampling → multilingual fetch → query → score)
│   ├── make_figures_from_csv.py   ← regenerate all figures directly from results_prod.csv
│   └── make_figures_from_summary.py ← figures from hard-coded summary values (reference only)
├── figures/
│   ├── fig1_chapter_accuracy.png
│   ├── fig2_alpha.png
│   ├── fig3_graded_accuracy.png
│   ├── fig4_english_concordance_heatmap.png
│   └── prisma_flow.png
└── data/                          ← data files (WHO definition text excluded; see Licensing)
    ├── entities_manifest.json         ← 300 sampled entities: code + official ICD-API URL + English title (NO WHO definitions)
    └── results_prod.csv           ← per-call log (entity, language, model, version, prediction, grade, raw text)
```

### ⚠ A note on the item set and WHO definitions
This release includes both data files. `results_prod.csv` is the per-call log from which the manuscript's numbers are computed, and `entities_manifest.json` lists the 300 sampled entities as code + official ICD-API URL + English title. **The WHO ICD-11 definition text is deliberately not included**, to respect the ICD-11 license (CC BY-ND 3.0 IGO; see Licensing). To obtain the official multilingual titles and definitions, fetch each entity's `url` from the WHO ICD-API with an `Accept-Language` header, or re-run the pipeline (below), which fetches them for you.

---

## Reproducing the figures and numbers
With `results_prod.csv` present:
```bash
pip install matplotlib numpy krippendorff
python code/make_figures_from_csv.py
```
This reads `results_prod.csv` directly (no transcribed numbers), writes `fig1`–`fig4`, and prints the exact chapter-level accuracy, Krippendorff's α, and English-concordance values used in the manuscript.

## Re-running the full benchmark (optional)
To evaluate new or additional models, set five API credentials as environment variables — `ICD_CLIENT_ID`, `ICD_CLIENT_SECRET` (WHO ICD-API), and the model-vendor keys `OPENAI_API_KEY`, `GOOGLE_API_KEY`, `ANTHROPIC_API_KEY` — then:
```bash
pip install requests numpy krippendorff
python code/pilot_icd11_v4.py
```
Because this release ships only `entities_manifest.json` (no `entities_prod.json` cache), the runner re-samples the 300 items under the fixed `SEED` and fetches their official six-language titles and definitions from the WHO ICD-API, writing its own `entities_prod.json`; it then logs every call to `results_prod.csv`. The run is resumable (re-running continues from where it stopped) and verifies model connectivity before starting. Edit the `MODELS` list at the top to add models; keep `N_ENTITIES`, `SEED`, and `LANGUAGES` unchanged to reproduce the published item set. (`entities_manifest.json` holds the exact 300 entity URLs, so you can instead fetch definitions for those URLs directly, without re-sampling.)

---

## Method summary (for quick reference)
- **Sampling:** per chapter, all descendants are retrieved via the ICD-API `include=descendant` option, shuffled under a fixed seed, and filtered to eligible leaves until a per-chapter quota is met (total 300).
- **Prompt:** the model receives the definition in the target language and returns `{"code":"…"}`. Instruction wording is native for English/Spanish/French; for Chinese/Arabic/Russian the instruction is in English while the definition is in the target language (a documented limitation).
- **Scoring:** four nested levels — exact, stem (before the decimal), three-character, and chapter (first character). Chapter level is the primary accuracy outcome.
- **Consistency:** Krippendorff's α (nominal) across the six languages, plus per-language agreement with the model's English assignment.

---

## Licensing
- **Code** (`code/`): [suggested: MIT License — add a LICENSE file].
- **Results and item set** (`results_prod.csv`, `entities_manifest.json`): [suggested: CC BY 4.0]. These hold entity codes, official ICD-API URLs, English titles, and the authors' own model outputs and scores.
- **ICD-11 content**: © World Health Organization, licensed under **CC BY-ND 3.0 IGO**. To respect this license, **the WHO ICD-11 definition text is not redistributed in this release**; `entities_manifest.json` lists only entity codes, official ICD-API URLs, and English titles, and the official multilingual titles and definitions are retrieved by the user directly from the WHO ICD-API. No WHO endorsement is implied. See https://icd.who.int and the ICD-11 license terms.

> Note: WHO's CC BY-ND license prohibits distributing *adaptations* of ICD-11 content. To stay clearly within the license, this release does **not** ship the definition text at all: it provides entity codes/URLs (plus the authors' own model outputs and scores) and lets users re-fetch the official definitions from the ICD-API.

---

## Suggested citation
> [Author(s)]. Cross-lingual consistency of large language models in ICD-11 coding: a six-language benchmark. [Journal / year]. Dataset and code: [Zenodo DOI].

ICD-11 itself: World Health Organization. International Classification of Diseases, Eleventh Revision (ICD-11). Geneva: WHO; 2022. License: CC BY-ND 3.0 IGO.

---

## Checklist before depositing
- [x] `data/` contains `results_prod.csv` and `entities_manifest.json` (WHO definition text excluded for CC BY-ND compliance)
- [ ] Add a `LICENSE` file for the code (e.g., MIT)
- [ ] Add a data-license note (e.g., CC BY 4.0) for `results_prod.csv` and `entities_manifest.json`
- [ ] Upload to Zenodo (or institutional repository), obtain the DOI
- [ ] Paste the DOI into the manuscript's **Data availability** statement
