Dataset Open Access

Lexical Relations from the Wisdom of the Crowd 1.0

Ustalov, Dmitry

Loukachevitch, Natalia

A set of 300 most frequent nouns has been extracted from the Russian National Corpus. Then, each method or resource, including RuThes, produced at most five hypernyms, if possible. In case it is not possible, missing answers treated as empty results. This resulted in 9 322 unique non-empty subsumption pairs that have been passed for crowdsourcing annotation on the Yandex.Toloka microtask platform. Each pair has been annotated by seven different annotators whose mother tongue is Russian and the age is at least 20 by February 1, 2017.

The layout of the human intelligence task (HIT) design assumes the direct answer to a simple question: does the given pair of words represent a meaningful is-a relation? Since the crowd workers are not expert lexicographers and this question might be difficult for them, it has been rephrased as “Is it correct that a kitten is a kind of mammal?” (in Russian).

The answers have been aggregated using the Yandex.Toloka proprietary answer aggregation mechanism. As the result, 3 940 out of 9 322 pairs have been annotated as positive while the rest 5 382 have been annotated as negative.

Interestingly, the workers were more confident in negative answers rather than in the positive ones. These negative answers are extremely useful for both training and testing different relation extraction methods. To the best of our knowledge, this is the first dataset of this kind made for the Russian language using microtask-based crowdsourcing.

Ustalov, D.: Expanding Hierarchical Contexts for Constructing a Semantic Word Network. In: Computational Linguistics and Intellectual Technologies: Papers from the Annual conference "Dialogue". Volume 1 of 2. Computational Linguistics: Practical Applications. pp. 369–381. RSUH, Moscow, Russia (2017)
Files (16.6 MB)
Name Size
670.0 kB Download
15.9 MB Download
6.1 kB Download
  • Loukachevitch, Natalia (2011) Thesauri in Information Retrieval Tasks

  • Lyashevskaya, Olga et al. (2009) Frequency Dictionary of the Russian Language (on Russian National Corpus)

  • Ustalov, Dmitry (TBD) Expanding Hierarchical Contexts for Constructing a Semantic Word Network

All versions This version
Views 108108
Downloads 4343
Data volume 249.5 MB249.5 MB
Unique views 9898
Unique downloads 1919


Cite as