Published January 31, 2018 | Version v1
Dataset Open

RUSSE'2018: Human-Annotated Sense-Disambiguated Word Contexts for Russian

  • 1. University of Mannheim
  • 1. University of Hamburg
  • 2. National Research University Higher School of Economics
  • 3. University of Mannheim
  • 4. Scrapinghub
  • 5. Lomonosov Moscow State University
  • 6. ABBYY


This dataset contains human-annotated sense identifiers for 2562 contexts of 20 words used in the RUSSE'2018 shared task on Word Sense Induction and Disambiguation for the Russian language; part of the bts-rnc evaluation dataset. These sense identifiers are disambiguated as according to the sense inventory of the Large Explanatory Dictionary of Russian.

The annotation is done on December 1, 2017, on the Yandex.Toloka crowdsourcing platform. In particular, 80 pre-annotated contexts are used for training the human annotators, 2562 contexts are annotated by humans such that each context was annotated by 9 different annotators. The annotation reliability is indicated by a high value of Krippendorff's α = 0.83. After the annotation, every context was additionally inspected (“curated”) by the organizers of the shared task.

The following words are represented: акция (action / stock), байка (yarn / tale), гвоздика (carnation / nail), гипербола (hyperbole), град (avalanche), гусеница (grub), домино (domino), кабачок (marrow / pub), капот (hood), карьер (mine / career), кок (cook), крона (top / crown), круп (croup), мандарин (mandarine), рок (fate / rock), слог (syllable), стопка (glass, stack), таз (bowl), такса (rate / badger-dog), шах (shah / check).

The following files are included in this dataset:

  • Toloka assignments (training: tasks-train.tsv, annotation: tasks-test.tsv)
  • Toloka output (non-aggregated: assignments_01-12-2017.tsv.xz, aggregated: aggregated_results_pool_1036853__2017_12_01.tsv)
  • annotator agreement report (agreement.txt)
  • curated report (report-curated.tsv.xz and a supplementary file tasks-eval.tsv.xz)
  • the final aggregated dataset (bts-rnc-crowd.tsv)

The bts-rnc-crowd.tsv file has the following format: id, lemma, sense_id, left hand side context, word form, right hand side context, list of senses. The encoding is UTF-8 and the line breaks are LF (UNIX).


Panchenko, A., Lopukhina, A., Ustalov, D., Lopukhin, K., Arefyev, N., Leontyev, A., Loukachevitch, N.: RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language. In: Computational Linguistics and Intellectual Technologies: Papers from the Annual International Conference "Dialogue". pp. 547–564. RSUH, Moscow, Russia (2018)



Files (6.7 MB)

Name Size Download all
859.0 kB Download
100 Bytes Preview Download
1.0 MB Download
1.5 MB Download
222.6 kB Download
233.0 kB Download
2.8 MB Download
91.1 kB Download

Additional details