Published February 26, 2020 | Version v2
Conference paper Open

An Evaluation Benchmark for Testing the Word Sense Disambiguation Capabilities of Machine Translation Systems

  • 1. University of Helsinki

Description

Lexical ambiguity is one of the many challenging linguistic phenomena involved in translation, i.e., translating an ambiguous word with its correct sense. In this respect, previous work has shown that the translation quality of neural machine translation systems can be improved by explicitly modeling the senses of ambiguous words. Recently, several evaluation test sets have been proposed to measure the word sense disambiguation (WSD) capability of machine translation systems. However, to date, these evaluation test sets do not include any training data that would provide a fair setup measuring the sense distributions present within the training data itself. In this paper, we present an evaluation benchmark on WSD for machine translation for 10 language pairs, comprising training data with known sense distributions. Our approach for the construction of the benchmark builds upon the wide-coverage multilingual sense inventory of BabelNet, the multilingual neural parsing pipeline TurkuNLP, and the OPUS collection of translated texts from the web. The test suite is available at http://github.com/Helsinki-NLP/MuCoW.

Files

LREC2020_mucow.pdf

Files (4.7 GB)

Name Size Download all
md5:007210a25f241ae7fe793108bc154c69
278.5 kB Preview Download
md5:2d7a259e050fc8288317cbc0665ce1fe
39.8 MB Preview Download
md5:9a59b8ce85c3612e5a0932cc0ecdcf95
507.5 MB Preview Download
md5:321cae6d182f368e599ceec81d851daa
1.2 GB Preview Download
md5:a6bfc46411b9be829a7606b711339e21
1.0 GB Preview Download
md5:88c726c11aee35f87107af01f77e04c9
1.4 GB Preview Download
md5:62d4f9016c09aed8b339ec8de3c381da
526.9 MB Preview Download