Exploring transparency in peer review: A study describing the content and tone of reviewers' confidential comments to editors

This data set was created to conduct - Exploring transparency in peer review: A study describing the content and tone of reviewers’ confidential comments to editors. Prior to data collection, the study received ethical approval the Dutch Association for Medical Education Ethics Review Board.  A preprint of the study is available on bioRxiv. 

Purpose: Recent calls to improve transparency in peer review have prompted examination of many aspects of the peer review process. Confidential comments to editors are a common component of many peer review systems that have clear implications for transparency, yet how reviewers use this component has escaped scrutiny in the published literature. Our study explores 1) how reviewers use the confidential comments section and 2) the alignment between comments to the editor and comments to authors with respect to content and tone. 

Methods: Our dataset consisted of 358 reviews of 168 manuscripts submitted between January 1, 2019 and August 24, 2020 to a top tier health professions education journal with a single blind review process. We first examined each review to determine whether the reviewer entered comments to the editor. Then, for the subset of reviews with comments, we used procedures consistent with conventional and directed qualitative content analysis to develop a coding scheme and code comments for content, tone, and section of the manuscript. For reviews in which the reviewer recommended reject, we coded for alignment between reviewers’ comments to the editor and to authors. We report descriptive statistics.

Results: Nearly half of the reviews contained comments to the editor (49%). Most of these comments (n=176) summarized the reviewers’ impression of the article (85%), which could include explicit reference to their recommended decision (44%) or comments on suitability for the journal (10%). The majority of comments addressed the quality of the argument (56%) or research design, methods, or data (51%). The tone of comments to the editor tended to be critical (40%) or constructive (34%). For the 48 reviews recommending reject, the majority of comments to editor contained content that also appeared in comments to the authors (65%); additional content tended to be irrelevant to the manuscript. Tone frequently aligned (85%).

Conclusion: Our findings indicate variability in how reviewers use the confidential comments to editor section in online peer review systems, though generally the way they use them suggests integrity and transparency to authors.