Assessment and prediction of the quality of scientific reviews: A text mining method

In this paper, we propose three metrics 𝑅𝑎𝑐𝑐, 𝑃𝑎𝑖, and Rcon to evaluate review quality. We collect the whole ICLR review corpus for the years 2017-2019 with all metadata. Utilizing the defined metrics, we evaluate the effectiveness of review process from multi-aspects. All three indicators declined as the submission increased. And reviewers are becoming more conservative in their recommendations. In addition, we proposed a model to predict the review quality. The result from Random Forest model shows that the review length, sentiment polarity, and confidence scores are three important factors that distinguish the quality of a review, which will assist meta-reviewers value more worthy reviews.