Can Physics Journal Article Quality Scores be Assigned by AI? The UK Case

This talk will describe an experiment to assess whether artificial intelligence (AI) could be used to assign quality scores to journal articles to partly or fully replace expert judges in research assessment exercises. It analyses human peer reviews from the UK Research Excellence Framework 2021 (1*, 2*, 3*, 4*) given to journal articles from all disciplines, but with a focus on physics. Although the results were not accurate enough for the satisfaction of the current peer reviewers, the results could be used to inform peer review, for example by identifying articles that are easy to score or by acting as a tie-breaker when the human judges disagree. An important useful property of the AI solution is its estimation of its own accuracy, so human judges can ignore computer predictions that are very uncertain. This talk will also discuss disciplinary differences in peer review and AI accuracy to give context to the physics AI results.