Crowdsourcing Topical Relevance with CrowdTruth

This repository contains the crowdsourcing annotations for topical relevance referenced in the following paper:

• Oana Inel, Giannis Haralabopoulos, Dan Li, Christophe Van Gysel, Zoltán Szlávik, Elena Simperl, Evangelos Kanoulas and Lora Aroyo: Studying Topical Relevance with Evidence-based Crowdsourcing. CIKM 2018.

If you find this data useful in your research, please consider citing:

@inproceedings{inel2018studying,
  title={Studying Topical Relevance with Evidence-based Crowdsourcing},
  author={Inel, Oana and Haralabopoulos, Giannis and Li, Dan and Van Gysel, Christophe and Szl{\'a}vik, Zolt{\'a}n and Simperl, Elena and Kanoulas, Evangelos and Aroyo, Lora},
  booktitle={Proceedings of the 27th ACM International Conference on Information and Knowledge Management},
  pages={1253--1262},
  year={2018},
  organization={ACM}
}

Running the notebooks

To run and regenerate the results, you need to install the stable version of the crowdtruth==2.0 package from PyPI using:
pip install crowdtruth==2.0