Published July 18, 2023
| Version 1.0.0
Dataset
Open
RheumaMIR
Authors/Creators
- 1. Fundación Investigación Biomédica Hospital Clínico San Carlos
Contributors
- 1. 0000-0002-6126-8786
- 2. 0000-0002-2869-7861
Description
This dataset accompanies the research paper entitled: Harnessing ChatGPT and GPT-4 for Evaluating the Rheumatology Questions of the Spanish Access Exam to Specialized Medical Training.
The dataset contains 145 rheumatology-related questions extracted from the Spanish MIR exams held between the academic years 2009-2010 to 2022-2023. The questions are evaluated by ChatGPT, GPT-4, BARD and CLAUDE. Six rheumatologists assess the clinical reasoning of ChatGPT and GPT-4.
The dataset is made up of the following columns:
| Column | Description |
| Id | Question identifier |
| Question (ES) | MIR exam question in Spanish |
| Question (EN) | Translation of `Question (ES)` column |
| Year | Academic year of the question (from 2009-2010 to 2022-2023) |
| Question Type | Case or factual question |
| Genre | Male, Female, Does not apply, No sex (newborn) |
| Invalidated question | 0,1 (invalidated question by the Spanish Minister of Health) |
| Official answer | Official answer given by the Spanish Minister of Health |
| GPT-4 answer | Answer provided by GPT-4 |
| Correct answer GPT-4 | 0, 1 (Whether the answer provided by GPT-4 is correct) |
| Clinical reasoning GPT-4 (ES) | Clinical reasoning provided by GPT-4 |
| Clinical reasoning GPT-4 (EN) | Translation of `Clinical reasoning GPT-4 (ES)` column |
| Eval1_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the first evaluator |
| Eval2_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the second evaluator |
| Eval3_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the third evaluator |
| Eval4_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the fourth evaluator |
| Eval5_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the fifth evaluator |
| Eval6_GPT4 | The score of the `Clinical reasoning GPT-4 (ES)` column given by the sixth evaluator |
| ChatGPT answer | Answer provided by ChatGPT |
| Correct answer ChatGPT | 0, 1 (Whether the answer provided by ChatGPT is correct) |
| Clinical reasoning ChatGPT (ES) | Clinical reasoning provided by ChatGPT |
| Clinical reasoning ChatGPT (EN) | Translation of `Clinical reasoning ChatGPT (ES)` column |
| Eval1_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the first evaluator |
| Eval2_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the second evaluator |
| Eval3_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the third evaluator |
| Eval4_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the fourth evaluator |
| Eval5_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the fifth evaluator |
| Eval6_ChatGPT | The score of the `Clinical reasoning ChatGPT (ES)` column given by the sixth evaluator |
| Disease category (ES) | The disease that the question addressed (Bone metabolism, Infective arthritis, Microcrystalline arthritis, Others, Rheumatoid arthritis, Scleroderma, Spondyloarthropathies, Systemic lupus erythematosus, Vasculitis ) |
| Disease category (EN) | Translation of `Disease category (ES)` column |
| CLAUDE answer | Answer provided by CLAUDE |
| Correct answer CLAUDE | 0, 1 (Whether the answer provided by CLAUDE is correct) |
| Clinical reasoning CLAUDE (ES) | Clinical reasoning provided by CLAUDE |
| Clinical reasoning CLAUDE (EN) | Translation of `Clinical reasoning CLAUDE (ES)` column |
| BARD answer | Answer provided by BARD |
| Correct answer BARD | 0, 1 (Whether the answer provided by BARD is correct) |
| Clinical reasoning BARD (ES) | Clinical reasoning provided by BARD |
| Clinical reasoning BARD (EN) | Translation of `Clinical reasoning BARD (ES)` column |
The translations of the questions and the clinical reasoning from Spanish into English was done with DeepL
Files
Files
(587.5 kB)
| Name | Size | Download all |
|---|---|---|
|
md5:151d66c1bcd848b4c6175367cd55225d
|
587.5 kB | Download |