---
dataset_info:
- config_name: anatomy
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: business_ethics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: clinical_knowledge
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: college_chemistry
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: college_computer_science
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: college_mathematics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: college_medicine
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: college_physics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: econometrics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: electrical_engineering
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: formal_logic
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: global_facts
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_chemistry
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_mathematics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_physics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_statistics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: human_aging
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: logical_fallacies
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: machine_learning
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: miscellaneous
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: philosophy
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: professional_accounting
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: public_relations
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: virology
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: conceptual_physics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_us_history
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: astronomy
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_geography
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: high_school_macroeconomics
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
- config_name: professional_law
  features:
  - name: question
    dtype: string
  - name: choices
    sequence: string
  - name: answer
    dtype: int64
  - name: error_type
    dtype: string
  - name: source
    dtype: string
  - name: correct_answer
    dtype: string
  - name: potential_reason
    dtype: string
  splits:
  - name: test
    num_examples: 100
configs:
- config_name: anatomy
  data_files:
  - split: test
    path: anatomy/data-*
- config_name: business_ethics
  data_files:
  - split: test
    path: business_ethics/data-*
- config_name: clinical_knowledge
  data_files:
  - split: test
    path: clinical_knowledge/data-*
- config_name: college_chemistry
  data_files:
  - split: test
    path: college_chemistry/data-*
- config_name: college_computer_science
  data_files:
  - split: test
    path: college_computer_science/data-*
- config_name: college_mathematics
  data_files:
  - split: test
    path: college_mathematics/data-*
- config_name: college_medicine
  data_files:
  - split: test
    path: college_medicine/data-*
- config_name: college_physics
  data_files:
  - split: test
    path: college_physics/data-*
- config_name: econometrics
  data_files:
  - split: test
    path: econometrics/data-*
- config_name: electrical_engineering
  data_files:
  - split: test
    path: electrical_engineering/data-*
- config_name: formal_logic
  data_files:
  - split: test
    path: formal_logic/data-*
- config_name: global_facts
  data_files:
  - split: test
    path: global_facts/data-*
- config_name: high_school_chemistry
  data_files:
  - split: test
    path: high_school_chemistry/data-*
- config_name: high_school_mathematics
  data_files:
  - split: test
    path: high_school_mathematics/data-*
- config_name: high_school_physics
  data_files:
  - split: test
    path: high_school_physics/data-*
- config_name: high_school_statistics
  data_files:
  - split: test
    path: high_school_statistics/data-*
- config_name: human_aging
  data_files:
  - split: test
    path: human_aging/data-*
- config_name: logical_fallacies
  data_files:
  - split: test
    path: logical_fallacies/data-*
- config_name: machine_learning
  data_files:
  - split: test
    path: machine_learning/data-*
- config_name: miscellaneous
  data_files:
  - split: test
    path: miscellaneous/data-*
- config_name: philosophy
  data_files:
  - split: test
    path: philosophy/data-*
- config_name: professional_accounting
  data_files:
  - split: test
    path: professional_accounting/data-*
- config_name: public_relations
  data_files:
  - split: test
    path: public_relations/data-*
- config_name: virology
  data_files:
  - split: test
    path: virology/data-*
- config_name: conceptual_physics
  data_files:
  - split: test
    path: conceptual_physics/data-*
- config_name: high_school_us_history
  data_files:
  - split: test
    path: high_school_us_history/data-*
- config_name: astronomy
  data_files:
  - split: test
    path: astronomy/data-*
- config_name: high_school_geography
  data_files:
  - split: test
    path: high_school_geography/data-*
- config_name: high_school_macroeconomics
  data_files:
  - split: test
    path: high_school_macroeconomics/data-*
- config_name: professional_law
  data_files:
  - split: test
    path: professional_law/data-*
license: cc-by-4.0
task_categories:
- question-answering
language:
- en
pretty_name: MMLU-Redux
size_categories:
- 1K<n<10K
---


# Dataset Card for MMLU-Redux

<!-- Provide a quick summary of the dataset. -->

MMLU-Redux is a subset of 3,000 manually re-annotated questions across 30 MMLU subjects. 

## Dataset Details

### Dataset Description

<!-- Provide a longer summary of what this dataset is. -->



- **Curated by:** Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile van Krieken, Pasquale Minervini
<!-- - **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed] -->
- **Language(s) (NLP):** English
- **License:** CC-BY-4.0

### Dataset Sources 

<!-- Provide the basic links for the dataset. -->

- **Repository:** https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux
- **Paper:** https://arxiv.org/abs/2406.04127
<!-- - **Demo [optional]:** [More Information Needed] -->

## Uses

<!-- Address questions around how the dataset is intended to be used. -->

### Direct Use

<!-- This section describes suitable use cases for the dataset. -->

[More Information Needed]

### Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the dataset will not work well for. -->

[More Information Needed]

## Dataset Structure

<!-- This section provides a description of the dataset fields, and additional information about the dataset structure such as criteria used to create the splits, relationships between data points, etc. -->

[More Information Needed]

## Dataset Creation

### Curation Rationale

<!-- Motivation for the creation of this dataset. -->

[More Information Needed]

### Source Data

<!-- This section describes the source data (e.g. news text and headlines, social media posts, translated sentences, ...). -->

Source data is from the MMLU dataset, downloaded from https://huggingface.co/datasets/cais/mmlu . 

#### Data Collection and Processing

<!-- This section describes the data collection and processing process such as data selection criteria, filtering and normalization methods, tools and libraries used, etc. -->

[More Information Needed]

#### Who are the source data producers?

<!-- This section describes the people or systems who originally created the data. It should also include self-reported demographic or identity information for the source data creators if this information is available. -->

[More Information Needed]

### Annotations [optional]

<!-- If the dataset contains annotations which are not part of the initial data collection, use this section to describe them. -->

#### Annotation process

<!-- This section describes the annotation process such as annotation tools used in the process, the amount of data annotated, annotation guidelines provided to the annotators, interannotator statistics, annotation validation, etc. -->

[More Information Needed]

#### Who are the annotators?

<!-- This section describes the people or systems who created the annotations. -->

[More Information Needed]

#### Personal and Sensitive Information

<!-- State whether the dataset contains data that might be considered personal, sensitive, or private (e.g., data that reveals addresses, uniquely identifiable names or aliases, racial or ethnic origins, sexual orientations, religious beliefs, political opinions, financial or health data, etc.). If efforts were made to anonymize the data, describe the anonymization process. -->

[More Information Needed]

## Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

[More Information Needed]

### Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should be made aware of the risks, biases and limitations of the dataset. More information needed for further recommendations.

## Citation

<!-- If there is a paper or blog post introducing the dataset, the APA and Bibtex information for that should go in this section. -->

**BibTeX:**

```
@misc{gema2024mmlu,
      title={Are We Done with MMLU?}, 
      author={Aryo Pradipta Gema and Joshua Ong Jun Leang and Giwon Hong and Alessio Devoto and Alberto Carlo Maria Mancino and Rohit Saxena and Xuanli He and Yu Zhao and Xiaotang Du and Mohammad Reza Ghasemi Madani and Claire Barale and Robert McHardy and Joshua Harris and Jean Kaddour and Emile van Krieken and Pasquale Minervini},
      year={2024},
      eprint={2406.04127},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```


<!-- ## Glossary [optional] -->

<!-- If relevant, include terms and calculations in this section that can help readers understand the dataset or dataset card. -->

<!-- [More Information Needed]

## More Information [optional]

[More Information Needed]

## Dataset Card Authors [optional]

[More Information Needed]
 -->
## Dataset Card Contact

[More Information Needed]