Multilingual Culture-Independent Word Analogy Datasets
Creators
- 1. University of Ljubljana, Ljubljana, Slovenia
- 2. Texta OU, Tallinn, Estonia; University of Tartu, Faculty of Arts and Humanities
- 3. University of Helsinki, Department of Finnish, Finno-Ugrian and Scandinavian Studies
- 4. University of Tartu, Faculty of Arts and Humanities; University of Latvia, Livonian institute
Description
In text processing, deep neural networks mostly use word embeddings as an input. Embeddings have to ensure that relations between words are reflected through distances in a high-dimensional numeric space. To compare the quality of different text embeddings, typically, we use benchmark datasets. We present a collection of such datasets for the word analogy task in nine languages: Croatian, English, Estonian, Finnish, Latvian, Lithuanian, Russian, Slovenian, and Swedish. We designed the monolingual analogy task to be much more culturally independent and also constructed cross-lingual analogy datasets for the involved languages. We present basic statistics of the created datasets and their initial evaluation using fastText embeddings.
Files
Ulčar_LREC_2020_2.pdf
Files
(118.4 kB)
Name | Size | Download all |
---|---|---|
md5:663482fa3b7c7458ba2bba5b77684e64
|
118.4 kB | Preview Download |