Reviewing Reviewers: An innovative tool for improving quality of local Diamond Open Access Journals

Serbian Citation Index – SCIndeks is a hybrid Open Access platform based on a business model of partnership between a non-profit web publisher and journal owners. Most journals hosted by SCIndeks are local scholar-led Diamond OA journals, relying on limited financial means, staff, and skills, but highly motivated by national regulatory and financing bodies to reach international status and level of excellence. The role of SCIndeks as a platform and its development team is to assist journals in their efforts by providing them with a journal management system, publishing expert guidance, techniques of promotion of journals on web, tools intended for quality control, and, finally, precise feedback on their advancement within all those processes. One of the most important forms of assistance is the comprehensive support for quality peer review.

SCIndeks development team uses various strategies and mechanisms to support the peer-review process:

• SCIndeks customized OJS-based journal management system (SCIndeks Assistant) helps editors keep track of the peer-review process;
• specialized, mostly custom-made tools and procedures make it easier for editors and reviewers to assess manuscripts: automated identification of mismatched/missing citations and references, references to retracted articles, and articles originating from disputable journals in the list of references;
• training for editors and reviewers;
• specialized software tool dubbed Reviewing Reviewers (RR) enabling the assessment of the usefulness of peer review reports for both (a) editorial decisions and (b) for authors’ improvement of submitted manuscripts; while the former is rated by editors, the latter is assessed by the authors themselves, which makes RR an innovative and (at the beginning) disputed tool;
• Statistics module of RR which is also integrated in the SCIndeks journal management system accumulating results (grades) of reviewers working for different journals; Statistic module is providing ranking of reviewers based on a composite score combining editors and authors grades.
• The RR composite score is used for selecting best reviewers for further engagement and at the same time for selecting candidates for annual financial award provided by SCIndeks publisher. 

In this presentation we describe RR in greater detail and analyze quantitatively the effects of its use in two previous years in comparison to the period preceding its implementation in SCIndeks Assistant. The analysis shows that the assessment of reviewers by the authors has resulted in some changes in reviewers’ behaviour. The change was found on the following indicators: (1) the share of reviews containing comments to the authors, (2) the share of such comments containing attachments (3) the average number of words in attachments, and (4) the relative number of new, first-time peer reviewers. All changes are statistically significant. Opposite to the expectations, no changes were observed in the average number of words in comments and the average rates given to the reviewers by editorial boards. However, what is most indicative, the structure of reviewers’ recommendations has slightly altered. Reviewers now tend to suggest accepting manuscripts conditionally (“accept after revision”) more frequently than before. The increase happened at the expense of the number of decisions to accept manuscripts immediately (“no revision”). This may be regarded a sort of proof that reviewers have started to take more helpful general attitude toward authors, which is exactly what we have sought to achieve.

To conclude, the results, supported by some anecdotal evidence, fully encourage the use of RR in an academic environment marked by an unsatisfactory reviewing process and the lack of motivation among eminent researchers to accept invitations of local journals editors. It is reasonable to assume that this will lead to the higher level of article quality in local Diamond OA journals, which is the ultimate goal of introducing RR.