Trial sequential analysis for assessing imprecision in GRADE evaluations – protocol for a methodological study
Description
Abstract
Background: Assessing statistical imprecision of summary estimates is an essential element in evaluating the strength of evidence. The GRADE framework recommends assessing imprecision by confidence intervals (CI) in relation to thresholds of interest and in selected cases to assess the relationship between the acquired information size and the calculated optimal information size (OIS).
Trial sequential analysis (TSA) calculates multiplicity-adjusted confidence intervals and can be used to calculate a required information size. In a recent methodological study of 544 systematic reviews and meta-analysis reports of clinical trials with TSA, we gathered data regarding the methods used for grading imprecision, specifically regarding the impact of TSA (the METSA project).
The questions regarding GRADE imprecision were initially superficially defined and were substantialized only during the project and in the preparations for this protocol. With this add-on study, we investigate the methods of grading imprecision in the GRADE framework by authors of systematic reviews and meta-analysis reports of clinical trials with TSA.
Methods: The outlined methodological study will be pre-planned but designed with knowledge about existing but not yet reviewed data on the study questions. The METSA project was not initially designed for the questions raised in the current study protocol, which warrants a critical review of the collected data. We aim to improve precision and accuracy of the collected data regarding imprecision methodology by a redesign of selected data fields, adding new data fields to the data extraction form and a subsequent revision of the existing data extraction accordingly. For each individual study, we will extract or review data regarding the specified methodology including methods for calculating CI and OIS, and thresholds of interest (definitions of important benefit and/or harm). For each topic, we will assess completeness in transparency of described methods and protocolisation, including coherence with the protocol (if relevant).
Results: We will report frequencies of observed methods, lack of transparency, and protocolisation. From data gathered in the METSA project, we will report the proportion of imprecision assessments that may have differed in their conclusions if the results of the TSA had been used. Informed by our findings, we will outline new suggestions on how to grade imprecision using TSA.
Conclusion: This protocol outlines a methodological study of methods and reporting characteristics imprecision assessment within the GRADE framework in recent systematic reviews and meta-analysis reports of clinical trials utilising trial sequential analysis.