Crowdsourcing Document Similarity Judgements

This is the data obtained from crowdsourcing tasks which ask workers to provide similarity metrics between pairs of documents. Each document, as well as each pair, has a unique ID. We provide crowd workers with the pairs through three different task variations:

• Variation 1: We showed workers 5 pairs of documents and, for each, asked them to rate their similarity in a 4-level Likert scale (None, Low, Medium, High), tell us a confidence level of how sure they were (from 0 to 4) and a written reason as to why they chose that similarity level. For quality reasons, two of the 5 pairs were golden-standards, which means we knew their ratings already and checked the workers' responses. They had to give the golden pair with the higher similarity a higher score than the other golden pair, otherwise, their answer would be rejected.
• Variation 2: We repeated variation 1 but with a slight alteration: instead of a Likert scale for the similarity score, we asked for a Magnitude Estimation, which is any number above 0. It could be 1, 0.0001, 1000, 42, as long as it was coherent, as in a more similar pair had a higher score than a less similar pair and vice-versa;
• Variation 3: We showed workers 5 rankings. Each ranking had a main document and 3 auxiliary documents to be compared against the main one. They also had to report a confidence score and give a short written reason, just like variation 1. The first ranking is a golden-standard, and we knew the values for the 3 pairs in it (the pairs were the main document paired with each of the 3 auxiliary documents), and they had to give the golden pair with the highest similarity a higher rank than the one with the lower similarity.

The raw results from the tasks are recorded in the JSON file CrowdResults.json. For a description of its contents, please read the file CrowdResults_README.md.

These raw annotations from the crowd were then parsed into the three CSVs you see, each corresponding to the aggregated results from one of the task variations.

• final_scores_likert.csv is the resulting scores for each pair using the variation 1 tasks;
  • pair_id is a unique identifier for each pair;
  • similarity_alg is the similarity assigned to the pair of documents from an automated similarity algorithm;
  • relation is the type of relationship shown by the pair, where smaller values indicate more similar pairs;
  • similarity_crowd_simple_maj stores the simple majority result from the crowd's annotations;
  • similarity_crowd_simple_mean stores the mean of the crowd's annotations;
  • similarity_crowd_simple_median stores the median of the crowd's annotations;
• final_scores_magnitude.csv is the resulting scores for each pair using the variation 2 tasks;
  • pair_id is a unique identifier for each pair;
  • similarity_alg is the similarity assigned to the pair of documents from an automated similarity algorithm;
  • relation is the type of relationship shown by the pair, where smaller values indicate more similar pairs;
  • scaled_similarity_worker is the magnitude score scaled based on worker's behaviours
  • scaled_similarity_worker_docset is the magnitude score scaled based both on the worker's behaviour and on the pair
• final_scores_ranking.csv is the resulting scores for each pair using the variation 3 tasks;
  • pair_id is a unique identifier for each pair;
  • similarity_alg is the similarity assigned to the pair of documents from an automated similarity algorithm;
  • relation is the type of relationship shown by the pair, where smaller values indicate more similar pairs;
  • mean_similarity is the mean ranking from that value

This dataset was built and used as part of the TheyBuyForYou project.