Reliability, usability and coverage of AMSTAR 2 assessing 544 systematic reviews and meta analysis reports - protocol for a descriptive analytic study
Description
Background: The AMSTAR 2 tool (second version of Assessing Methodological Quality in Systematic Reviews) is useful for critical appraisal of systematic reviews of clinical trials. In a methodological study of systematic reviews and meta-analysis reports of randomised clinical trials which used the sequential meta-analysis trial sequential analysis (the METSA study), we used the AMSTAR 2 to assess the overall quality of each included study. With the study outlined in this protocol, we aim to explore the interrater reliability of the AMSTAR 2, qualitatively describe our experiences using the tool, and discuss the tool’s coverage of critical domains.
Methods: In the METSA study, we investigated statistical methodology and transparency in 544 systematic reviews and meta-analysis reports of randomised clinical trials which used trial sequential analysis (TSA). All systematic reviews (with a protocol) were assessed with AMSTAR 2 by two independent authors (n=270). Meta-analysis reports – defined as not having a protocol – were automatically rated as ‘critically low confidence’ and did no undergo further AMSTAR 2 assessment. Disagreement on the AMSTAR 2 rating was resolved through discussion between the authors. Principal issues were discussed at weekly meetings. Thoughts on the usability and coverage of AMSTAR 2 was shared at these meetings and noted throughout and will be collected post-hoc for the current study. Here, we will analyse the level of agreement on the initial ratings by raw agreement rates and Cohen’s kappa and test for trends concerning the effect of the consensus process (rating up or down confidence) as well as the overall effect of assessor experience. We will compare the AMSTAR 2 rating with the assessments of TSA transparency performed during the METSA study.
Conclusion: This methodological study will provide insights in some of the characteristics of AMSTAR 2, including interrater reliability and usability in the context of assessing 270 systematic reviews of clinical trials. We will provide group consensus-based suggestions regarding usability and coverage.