Domain expert readability dataset

Judgments gathered from 10 experts through a web-based survey on the readability of publication abstracts. The abstracts used were a subset of the AMiner's DBLP citation nework v10 dataset (https://aminer.org/citation) in the discipline of data and knowledge management. In particular, abstracts containing the following keywords were used: "database", "machine learning", "information retrieval", "data management", "cloud computing", "data mining", "algorithms", "classification", "query processing", "networks", "indexing", "distributed systems".

After reading the abstract, each expert had to answer the following questions on a 5 point scale.

• Q1: Please rate how well-written the abstract is.
• Q2: Does the abstract contain linguistic errors?
• Q3: Please rate how clear the contribution of the paper is (based on the abstract).

For each question, the interpretation of the extreme scale values (i.e., 1 and 5) were provided. In particular, 1 = “very poorly written” / “so many ling. errors that make abstract incomprehensible” / “not clear at all” (Q1/Q2/Q3) and 5 = “excellently written” / “no errors” / “completely clear” (Q1/Q2/Q3).

The pairwise correlations (Kendall’s τ) of expert judgments on questions Q1-Q3 are presented in this table.

The contained dataset is a tsv file that includes the following fields:

• user_id: expert identifier
• paper_id: AMiner's identifier from DBLP citation nework v10 dataset
• rating_1: answer for Q1
• rating_2: answer for Q2
• rating_3: answer fro Q3 

Please cite:
Thanasis Vergoulis, Ilias Kanellos, Anargiros Tzerefos, Serafeim Chatzopoulos, Theodore Dalamagas, Spiros Skiadopoulos. A study on the readability of scientific publications. 23^rd International Conference on Theory and Practice of Digital Libraries. Oslo, Norway 2019 (to appear)