vera.ai HorizonEU Project

Part of EU Open Research Repository
Published December 9, 2023 | Version v1
Conference paper Open

MULTITuDE: Large-Scale Multilingual Machine-Generated Text Detection Benchmark

Creators

  • 1. Kempelen Institute of Intelligent Technologies
  • 2. ROR icon MIT Lincoln Laboratory
  • 3. ROR icon Pennsylvania State University
  • 4. ROR icon University of Mississippi

Description

There is a lack of research into capabilities of recent LLMs to generate convincing text in languages other than English and into performance of detectors of machine-generated text in multilingual settings. This is also reflected in the available benchmarks which lack authentic texts in languages other than English and predominantly cover older generators. To fill this gap, we introduce MULTITuDE, a novel benchmarking dataset for multilingual machine-generated text detection comprising of 74,081 authentic and machine-generated texts in 11 languages (ar, ca, cs, de, en, es, nl, pt, ru, uk, and zh) generated by 8 multilingual LLMs. Using this benchmark, we compare the performance of zero-shot (statistical and black-box) and fine-tuned detectors. Considering the multilinguality, we evaluate 1) how these detectors generalize to unseen languages (linguistically similar as well as dissimilar) and unseen LLMs and 2) whether the detectors improve their performance when trained on multiple languages.

Files

2023.emnlp-main.616.pdf

Files (650.4 kB)

Name Size Download all
md5:4c9017c5b4803bf2e8ada9ac90e390a5
650.4 kB Preview Download

Additional details

Identifiers

arXiv
arXiv:2310.13606

Funding

European Commission
VIGILANT - Vital IntelliGence to Investigate ILlegAl DisiNformaTion 101073921
European Commission
vera.ai - vera.ai: VERification Assisted by Artificial Intelligence 101070093