Using bench protocol platforms to improve compliance with systematic review guidance documents and reporting checklists: proof of concept

Context: A significant number of guidance documents and reporting checklists have been published to support researchers in planning, doing, and writing up scientifically rigorous systematic reviews (SRs). However, compliance of researchers with SR guidance and reporting checklists remains a significant challenge, with the majority of published SRs lacking in one or more aspects of rigour of methods and transparency of reporting.

Objective: To explore how bench protocol development platforms might be repurposed for improving compliance of SRs with conduct guidance and reporting checklists.  

System design: We developed a proof-of-concept technology stack based around a general-purpose, guidance- and checklist-compliant SR protocol that was built in protocols.io. We used the protocols.io platform to create an integrated research planning and data collection process for planning guidance-compliant SRs. We used our own custom code and the mustache templating language to automatically create checklist-compliant first-draft SR protocol documents in Microsoft Word

Discussion: Creating the operational process for SR protocol planning and the technology stack for automated documentation allowed us to develop our theoretical understanding of how such a system may improve compliance with research conduct and reporting standards. This includes the potential value of algorithmic rather than heuristic approaches to conducting and reporting research studies, positioning of labelled data rather than a study manuscript as the primary product of the research process, and viewing the process of developing research standards as being analogous to development of open software. Our study also allowed us to identify a number of technological issues that will need to be addressed to enable further development and testing of our proposed approach. These include limitations in templating language, especially when working in Microsoft Word, and the need for more data labelling and export formats from protocols.io.