Dataset Open Access

RUSSE'2018: Human-Annotated Sense-Disambiguated Word Contexts for Russian

Ustalov, Dmitry

Researcher(s)
Panchenko, Alexander; Lopukhina, Anastasia; Ustalov, Dmitry; Lopukhin, Konstantin; Arefyev, Nikolay; Leontyev, Aleksey; Loukachevitch, Natalia

This dataset contains human-annotated sense identifiers for 2562 contexts of 20 words used in the RUSSE'2018 shared task on Word Sense Induction and Disambiguation for the Russian language; part of the bts-rnc evaluation dataset. These sense identifiers are disambiguated as according to the sense inventory of the Large Explanatory Dictionary of Russian.

The annotation is done on December 1, 2017, on the Yandex.Toloka crowdsourcing platform. In particular, 80 pre-annotated contexts are used for training the human annotators, 2562 contexts are annotated by humans such that each context was annotated by 9 different annotators. The annotation reliability is indicated by a high value of Krippendorff's α = 0.83. After the annotation, every context was additionally inspected (“curated”) by the organizers of the shared task.

The following words are represented: акция (action / stock), байка (yarn / tale), гвоздика (carnation / nail), гипербола (hyperbole), град (avalanche), гусеница (grub), домино (domino), кабачок (marrow / pub), капот (hood), карьер (mine / career), кок (cook), крона (top / crown), круп (croup), мандарин (mandarine), рок (fate / rock), слог (syllable), стопка (glass, stack), таз (bowl), такса (rate / badger-dog), шах (shah / check).

The following files are included in this dataset:

  • Toloka assignments (training: tasks-train.tsv, annotation: tasks-test.tsv)
  • Toloka output (non-aggregated: assignments_01-12-2017.tsv.xz, aggregated: aggregated_results_pool_1036853__2017_12_01.tsv)
  • annotator agreement report (agreement.txt)
  • curated report (report-curated.tsv.xz and a supplementary file tasks-eval.tsv.xz)
  • the final aggregated dataset (bts-rnc-crowd.tsv)

The bts-rnc-crowd.tsv file has the following format: id, lemma, sense_id, left hand side context, word form, right hand side context, list of senses. The encoding is UTF-8 and the line breaks are LF (UNIX).

Panchenko, A., Lopukhina, A., Ustalov, D., Lopukhin, K., Arefyev, N., Leontyev, A., Loukachevitch, N.: RUSSE'2018: A Shared Task on Word Sense Induction for the Russian Language. In: Computational Linguistics and Intellectual Technologies: Papers from the Annual International Conference "Dialogue". pp. 547–564. RSUH, Moscow, Russia (2018)
Files (6.7 MB)
Name Size
aggregated_results_pool_1036853__2017_12_01.tsv
md5:9861592150d8e5841ab49aa48cc3228a 		859.0 kB Download
agreement.txt
md5:58dd6fbee3dc0975107d6c54461436f5 		100 Bytes Download
assignments_01-12-2017.tsv.xz
md5:bc829cfdcffd3d06b98e48d3dbcf368f 		1.0 MB Download
bts-rnc-crowd.tsv
md5:ca6063fd367e4d9e361493b42bb54423 		1.5 MB Download
report-curated.tsv.xz
md5:7064569113d23770f58e93ad61fb13c3 		222.6 kB Download
tasks-eval.tsv.xz
md5:618b623d9c34b1d55e2a347c1fcc706a 		233.0 kB Download
tasks-test.tsv
md5:bb8a39021e269b8b8a7485f9c3c7a074 		2.8 MB Download
tasks-train.tsv
md5:ee463777c0f2825e08a70dcd74f09563 		91.1 kB Download
322
142
views
downloads
All versions This version
Views 322322
Downloads 142142
Data volume 115.2 MB115.2 MB
Unique views 287287
Unique downloads 6363
More info on how stats are collected.
Indexed in
Publication date:
January 31, 2018
DOI:
10.5281/zenodo.1117228

Zenodo DOI Badge

DOI 

10.5281/zenodo.1117228

Markdown 

[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1117228.svg)](https://doi.org/10.5281/zenodo.1117228)

reStructedText 

.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.1117228.svg
   :target: https://doi.org/10.5281/zenodo.1117228

HTML 

<a href="https://doi.org/10.5281/zenodo.1117228"><img src="https://zenodo.org/badge/DOI/10.5281/zenodo.1117228.svg" alt="DOI"></a>

Image URL 

https://zenodo.org/badge/DOI/10.5281/zenodo.1117228.svg

Target URL 

https://doi.org/10.5281/zenodo.1117228

Keyword(s):
word sense disambiguation word sense inducation lexical semantics human annotation gold standard crowdsourcing Russian
Meeting:
24th International Conference on Computational Linguistics and Intellectual Technologies (Dialogue 2018), Moscow, Russia, 30 May - 2 June 2018 (Session Dialogue Evaluation, Part Word Sense Induction and Disambiguation)
Related identifiers:
Compiled by
Documented by
Part of
Referenced by
Supplement to
Communities:
License (for files):
Creative Commons Attribution Share Alike 4.0 International

Versions

Version 1 10.5281/zenodo.1117228 Jan 31, 2018
Cite all versions? You can cite all versions by using the DOI 10.5281/zenodo.1117227. This DOI represents all versions, and will always resolve to the latest one. Read more.

Share

Cite as

Export