Dataset Card for NTEU Multilingual Evaluation Dataset

Dataset Description

• Point of Contact: langtech@bsc.es

Dataset Summary

This evaluation dataset for Machine Translation was created by the NTEU - Neural Translation for the EU project. The evaluation dataset includes around 1,000 parallel sentences in the 24 official European languages. The original NTEU dataset has been cleaned and filtered by removing empty lines and near-duplicates, and it has been augmented with Catalan. The Catalan version was manually produced by a native Catalan translator from the original English and Spanish versions, and was sponsored by the AINA project.

Supported Tasks and Leaderboards

This dataset can be used to evaluate bilingual and multilingual machine translation systems for any combination of the 24 official European languages and Catalan in the legal domain.

Languages

The languages included in the dataset are the following:

Dataset Structure

Data Instances

A separate .txt file is provided for each language, with sentences aligned in the same order across all files. Each file uses the two-letter language code of its language as the file extension.

Data Fields

[N/A]

Data Splits

The dataset contains a single split: Test.

Dataset Creation

Curation Rationale

The aim of this dataset is to promote the evaluation of machine translation systems for the official European languages, plus Catalan.

Source Data

Initial Data Collection and Normalization

The data was originally extracted from EUR-Lex, the official online database of European Union law and other public documents of the European Union (EU), published in the 24 official languages of the EU. The Official Journal (OJ) of the European Union is also published on EUR-Lex.

Who are the source language producers?

EUR-Lex

Annotations

Annotation process

The dataset does not contain any annotations.

Who are the annotators?

[N/A]

Personal and Sensitive Information

No specific anonymisation process has been applied, personal and sensitive information may be present in the data. This needs to be considered when using the data for training models.

Considerations for Using the Data

Social Impact of Dataset

By providing this resource, we intend to promote the evaluation of machine translation systems including all the official European Languages and Catalan, thereby improving the accessibility and visibility of the Catalan language in Europe.

Discussion of Biases

No specific bias mitigation strategies were applied to this dataset. Inherent biases may exist within the data.

Other Known Limitations

The dataset contains data of a legal/administrative domain. Applications of this dataset in other domains would be of limited use.

Additional Information

Dataset Curators

Language Technologies Unit at the Barcelona Supercomputing Center (langtech@bsc.es).

Funding

This work has been promoted and financed by the Government of Catalonia through the Aina project.

Licensing Information

This work is licensed under an Attribution 4.0 International licence.

Citation Information

For more information about the NTEU Project, please refer to the following paper:

@inproceedings{bie-etal-2020-neural,
    title = "Neural Translation for the {E}uropean {U}nion ({NTEU}) Project",
    author = "Bi{\'e}, Laurent  and
      Cerd{\`a}-i-Cuc{\'o}, Aleix  and
      Degroote, Hans  and
      Estela, Amando  and
      Garc{\'i}a-Mart{\'i}nez, Mercedes  and
      Herranz, Manuel  and
      Kohan, Alejandro  and
      Melero, Maite  and
      O{'}Dowd, Tony  and
      O{'}Gorman, Sin{\'e}ad  and
      Pinnis, M{\={a}}rcis  and
      Rozis, Roberts  and
      Superbo, Riccardo  and
      Vasi{\c{l}}evskis, Art{\={u}}rs",
    editor = "Martins, Andr{\'e}  and
      Moniz, Helena  and
      Fumega, Sara  and
      Martins, Bruno  and
      Batista, Fernando  and
      Coheur, Luisa  and
      Parra, Carla  and
      Trancoso, Isabel  and
      Turchi, Marco  and
      Bisazza, Arianna  and
      Moorkens, Joss  and
      Guerberof, Ana  and
      Nurminen, Mary  and
      Marg, Lena  and
      Forcada, Mikel L.",
    booktitle = "Proceedings of the 22nd Annual Conference of the European Association for Machine Translation",
    month = nov,
    year = "2020",
    address = "Lisboa, Portugal",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2020.eamt-1.60/",
    pages = "477--478",
    abstract = "The Neural Translation for the European Union (NTEU) project aims to build a neural engine farm with all European official language combinations for eTranslation, without the necessity to use a high-resourced language as a pivot. NTEU started in September 2019 and will run until August 2021."
}

Contributions

[N/A]