How often are statistically significant results clinically relevant? Not often

Objectives. Statistical significance does not equal clinical significance. This study looked at how frequently statistically significant results in the nuclear medicine literature are clinically relevant. 

Methods. A Medline search was performed with results limited to clinical trials or randomized controlled trials published in one of the major nuclear medicine journals. Articles analyzed were limited to those reporting continuous variables where a mean (X) and standard deviation (SD) were reported and determined to be statistically significant (p < 0.05). A total of 32 test results were evaluated. Clinical relevance was determined in a two-step fashion. First, the crossover point between groups 1 (normal) and 2 (abnormal) was determined. At this point, a variable is just as likely to fall in the normal distribution as the abnormal distribution. Jacobson's test for clinically significant change was used: crossover point = (SD1 * X2 + SD2 * X1) / (SD1 + SD2). How many SDs from the mean this crossover point fell was then determined. For example, 13.9 +/- 4.5 compared to 9.2 +/- 2.1 was reported as statistically significant (p < 0.05). The crossover point is 10.7, which equals 0.71 std from the mean: 13.9 - (0.71*4.5) = 9.2 + (0.71*2.1). 

Results. The average crossover point was 0.66 SDs from the mean. The crossover point was within 1 SD from the mean in 26/32 cases and in these cases, averaged 0.45 SD. Thus, for 4 out of 5 statistically significant results, when applied to an individual patient, the cut-off between normal and abnormal was 0.45 SD from the mean. This results in a third of normal patients falling into an abnormal category. 

Conclusions. Statistically significant results frequently are not clinically significant. Statistical significance alone does not ensure clinical relevance. 

Citation: Heston TF, Wahl RL. "How Often Are Statistically Significant Results Clinically Relevant? Not Often." Journal of Nuclear Medicine 50, no. supplement 2 (2009): 1370–1370.