Fourth, the authors do not mention how other aspects of the scholarly communication system also contribute to the overreaching in the interpretation of findings. For example, high-rejection rates at journals make editors select articles with "hot" headlines and researchers' degrees of freedom stretch the evidence gathering process and potentially also change the scope of the analysis. Including these aspects in the (introduction of the) study would heighten the study's relevance.
Regarding the relevance and rationale of the study, the authors describe in their introduction two 'problems' that may result from interpreting correlational findings as causal. They (a) list several examples of newspapers and other media outlets, which include obvious (and amusing) erroneous interpretations and (b) note that also within research, errors are made in interpreting correlational evidence as causal. The issue here is that it is unclear which of these 'problems' (scenario a or b) the authors focus on, and therefore what the study results implicate.
Our impression is that the authors want to focus on scenario b, thus, describing potential causal explanations for correlational findings in research as 'wrong', because it suggests to other researchers that the study was actually of causal nature. This is of course a challenging statement that would give rise to an interesting discussion, regarding researchers' habits of providing potential causal explanations for their findings. There are however some issues in the manuscript that lead us to think that such a claim is not supported.
First, the introduction states that "inappropriate causal language" has been found in a few fields, such as nutritional and educational fields. However, to our anecdotalknowledge, most articles appropriately describe in their methods and discussion sections that an inferential approach has been used, and that one must be careful not to interpret the result as causal. In many research schools it is viewed as essential to provide some theoretical framework and possible (causal) interpretation of the current results, accompanied by a statement clarifying that the current data are correlational. If the authors argue that this is not happening correctly in scientific practice, more evidence for this argument should be brought forward in the introduction. Alternatively, the authors should specify to which specific scientific fields the results apply.
Second, the main audience for scientific articles consists of scientists or students with moderate knowledge on statistical methods. It seems inefficient to ask of researchers that they refrain from describing any theoretical framework that might suggest a causal relationship, because individuals without sufficient statistical and methodological knowledge might read the papers. It seems reasonable that scientific papers should mostly be comprehensible for other scientists and students with some knowledge on methods and statistics. Given that students following an introductory class in statistics were the study population, the claim that researchers in general are interpreting correlational inferential data erroneously, seems unsupported.
This brings us to the issue of generalizability. Do you view the students as representative of researchers, the press, or the general population? The study uses a causal design by using a randomized experiment, but has a non-random sample and does not allow for inferences. One could even consider the student "sample" as being the population and deem inferences beyond this not applicable.
Nevertheless, the authors bring up an important issue in the introduction regarding erroneous interpretations of the media (named scenario b above). We suggest that your paper has more clear-cut implications for the extent to which readers who are potentially not well-versed in research and statistics (given they are students), interpret research findings as causal versus correlational, than the researcher population at large. This point of view may better match your rationale in the introduction and the examples you have included. Moreover, the students you have tested may be more representative of people-who-are-not-(yet)-well-schooled-in-statistics, such as individuals working for newspapers, than of researchers.
Finally, regardless of which focus the authors choose, we note that the paper shows there is a problem of how strongly expressive language can affect student's interpretation of findings, but provides little framework for how to move forward. For example, it is implied throughout the paper that it is a bad thing to stretch the scope of an inferential analysis to causal (with which we agree), but does not show how we might prevent it. The manuscript seems to imply that researchers should not indicate the hypotheses underlying a certain correlation at all. If this is not the opinion of the authors, we recommend sharpening the writing to clarify what is and what is not permitted from your perspective and how to move forward.
Specific comments about experimental approaches and methods used in the study: