Dataset Open Access

Lexical Relations from the Wisdom of the Crowd 1.0

Ustalov, Dmitry

Other(s)
Loukachevitch, Natalia

A set of 300 most frequent nouns have been extracted from the Russian National Corpus. Then, each method or resource, including RuThes, produced at most five hypernyms, if possible. In case it is not possible, missing answers treated as empty results. This resulted in 9 322 unique non-empty subsumption pairs that have been passed for crowdsourcing annotation on the Yandex.Toloka microtask platform. Each pair has been annotated by seven different annotators whose mother tongue is Russian and the age is at least 20 by February 1, 2017.

The layout of the human intelligence task (HIT) design assumes the direct answer to a simple question: does the given pair of words represent a meaningful is-a relation? Since the crowd workers are not expert lexicographers and this question might be difficult for them, it has been rephrased as “Is it correct that a kitten is a kind of mammal?” (in Russian).

The answers have been aggregated using the Yandex.Toloka proprietary answer aggregation mechanism. As the result, 3 940 out of 9 322 pairs have been annotated as positive while the rest 5 382 have been annotated as negative.

Interestingly, the workers were more confident in negative answers rather than in the positive ones. These negative answers are extremely useful for both training and testing different relation extraction methods. To the best of our knowledge, this is the first dataset of this kind made for the Russian language using microtask-based crowdsourcing.

Files (16.6 MB)
Name Size
lrwc-1.0-aggregated.tsv
md5:83f09c5bb3f4730e9878f2182eb86088
670.0 kB Download
lrwc-1.0-assignments.tsv
md5:42dddb6bff4c709440a55b1e6e042cd9
15.9 MB Download
toloka-isa-50-skip-300-train-hit.tsv
md5:831878245bb9e5b41c08f4a5f58cc3de
6.1 kB Download
  • Loukachevitch, Natalia (2011) Thesauri in Information Retrieval Tasks
  • Lyashevskaya, Olga et al. (2009) Frequency Dictionary of the Russian Language (on Russian National Corpus)
  • Ustalov, Dmitry (TBD) Expanding Hierarchical Contexts for Constructing a Semantic Word Network

Share

Cite as