# Transparent Statistics Guiding Principles

**Version:** 1.0 

**Contributed to the writing:** Pierre Dragicevic, Chat Wacharamanotham, Matthew Kay

**Gave feedback:** 

**Endorsed:** 

**DOI:** [10.5281/zenodo.1344317](https://doi.org/10.5281/zenodo.1344317)

**Source:** [GitHub](https://github.com/transparentstats/guidelines/blob/master/guides/principles.Rmd)


## Introduction

Human-computer interaction (HCI) is a large, multidisciplinary field drawing on a variety of approaches for analyzing quantitative data. However, many of our existing practices have drawn increasing criticism, such as our overreliance on mechanical statistical testing procedures, our lack of replications and meta-analyses, and our unwillingness to share data and study materials. These issues have been discussed within HCI [@Wilson2011; @Kaptein2012; @Dragicevic2016; @Kay2016; @Cockburn2018] and (to a much larger extent) in other fields [@Cohen1994; @Gigerenzer2004; @Ioannidis2005; @Simmons2011; @Giner2012; @Cumming2014a; @Nosek2017]. Poor statistical practice and the lack of transparency in statistical reporting hamper the progress of knowledge and undermine the scientific credibility of affected disciplines, as witnessed by the growing number of press articles reporting a "crisis of confidence" in the most visible of these disciplines [@Earp2015].

The purpose of the transparent statistics guidelines is not to admonish an entire field of researchers for their existing practices nor to urge them to adopt a specific set of methods. There is no universal inference procedure that can act as a substitute for good statistical reasoning [@Gigerenzer2015]. The multifaceted nature of HCI also means we need to embrace a variety of practices. A fixed set of DOs and DON'Ts would be both too brittle to change over time and too restrictive in the face of the various ways of generating knowledge in our field. Instead, we propose to advance a general vision of *transparent statistics* that HCI researchers can draw inspiration from, and that is largely method-agnostic. We refer to transparent statistics simply as *"a philosophy of statistical reporting whose purpose is to advance scientific knowledge rather than to persuade"*. Regardless of the methods used, we aim to provide guidance that makes the communication of those methods more transparent, that makes reproduction and replication of work easier, and that makes evaluation of work (e.g., by peer reviewers) easier and more fair.

To that end, a "transparent statistics" initiative was started in 2016, whose purpose is to discuss ways of promoting transparent statistics at CHI and suggest a series of incremental changes within the community [@Transparent2017]. These include more specific author and reviewer guidelines, exemplars for authors, and "badges" [@Open2017]. The goal of the initiative is to address questions such as: what can an author do to improve the transparency of their communication? What can a reviewer do to encourage and reward transparency? What changes to the review process might encourage transparency and incentivize researchers? In this way we hope to avoid the time-honored tradition of admonishing researchers for doing statistics poorly, and instead encourage them—and guide them—to do better. The goal of this first chapter is to lay out the high-level principles on which other chapters will be based. Like other chapters, this chapter is not meant to be fixed in stone, but is meant to be constantly evolving and iteratively refined by the CHI community.


## Guiding Principles

Again, transparent statistics is *"a philosophy of statistical reporting whose purpose is to advance scientific knowledge rather than to persuade"*. This idea is not new. For example, the following quote from Ronald Fisher captures the essence of transparent statistics:

> “we have the duty of [...] communicating our conclusions in intelligible form, in recognition of the right of other free minds to utilize them in making their own decisions.” [@Fisher1955].

More recent writings have emphasized the importance of contributing useful and accurate knowledge over telling compelling and convincing stories [@Giner2012; @Dragicevic2016]. Based on these visions, we propose a set of nine guiding principles for writing transparent statistical reports: 1) faithfulness, 2) robustness, 3) resilience, 4) process transparency, 5) clarity, 6) simplicity, 7) non-contingency, 8) precision and economy, and 9) material availability.


### 1. Faithfulness {-}

At the most basic level, a transparent statistical report should strive to be faithful to the data and the phenomena studied. This means that it should strive to capture and convey the "truth" as accurately as possible, especially concerning the uncertainty within the data. Major sources of uncertainty need to be carefully assessed and propagated to the presentations and interpretations of the results, all the way up to the final conclusions. Conclusions should be nuanced and stress the uncertainty in the data and in the process.

* *Example:* It is evident that any major error in an analysis will result in findings that are likely not faithful to the data and the phenomena studied. This includes effect estimates that are very different from the true effect, but also measures of uncertainty that fail to capture the true uncertainty.

* *Example:* Exaggerating findings by presenting uncertain results as certain is unfaithful to the data.

* *Example:* A study report that analyzes all its data carefully but fails to acknowledge important issues with data validity (e.g., non-random condition assignment) is faithful to the data but unfaithful to the phenomena studied. The same goes with over-generalizing findings.


### 2. Robustness {-}

In order to minimize the likelihood of inaccurate (unfaithful) results, data analysis and reporting strategies that are robust to departures from statistical assumptions -- or that make few assumptions -- should ideally be preferred.

Given that statistical assumptions are never met perfectly, the question should not be "are the assumptions met?" but instead "what are the likely consequences of such and such departure?". Thus, it is hugely beneficial for researchers to know how their methods behave depending on the nature and degree of possible departures, so that they can explain it in their report when necessary. When uncertain, methods that are known for their robustness are safer choices.

* *Example:* ANOVA is robust to the normality assumption, and can in some cases give accurate results with unusual distributions and very small sample sizes [@Norman2010].
* *Example:* Bootstrapping makes no assumption about data distribution and is robust to departures from its own statistical assumptions, even though these assumptions are implausible [@Kirby2013].


### 3. Resilience {-}

Data analysis and reporting strategies should be resilient to statistical noise, i.e., they should yield similar outcomes across hypothetical replications of the same study. Researchers should ask themselves how their statistical report would change if they took another random sample from the same population, and should try to make claims that are as robust as possible to these changes.

In practice, the principle of resilience implies that researchers should avoid presenting statistical noise as signal, either by overfitting, or by overinterpreting patterns in results. It also implies that study reports should be smooth functions of the data. This means that data analysis and presentation strategies should be chosen so that similar experimental datasets yield similar results, interpretations and conclusions [@Dixon2003; @Dragicevic2016; @Dragicevic2017]. The principle of resilience is important and is directly relevant to the issue of study replicability.

* *Example:* Presenting a bar chart of means without error bars and commenting on the emerging patterns is akin to overfitting and is thus not resilient.
* *Example:* Computing and reporting 95% interval estimates is resilient, but drawing binary conclusions based on whether they contain zero is not, because two very similar datasets may yield seemingly very different scientific conclusions [@Cumming2013a].
* *Example:* For the same reasons, computing Bayes factors and interpreting them strictly based on conventional thresholds violates the principle of resilience [@Lakens2016].


### 4. Process Transparency {-}

A core aspect of transparent statistics is that data analysis and reporting strategies need to be explained rather than implied. The decisions made during the analysis and report writing should be communicated as explicitly as possible, as the results of an analysis cannot be fairly assessed and understood if many decisions remain concealed [@Giner2012; @Gelman2013].

At the most basic level, researchers should ideally state which portions of their data analysis were planned before the data was seen, and which portions were not. Analyses that are fully planned can be referred to as *prespecified*, while analyses that are largely unplanned should be referred to as *exploratory* [@Cumming2014a, p.10]. Both types of analysis are valid, although the former allows to support stronger claims than the latter.

Process transparency also implies faithfully reporting what were the research goals, the research questions, and (optionally) the researcher's expectations prior to seeing the data [@Kerr1998; @Gelman2013; @Cockburn2018]. Results from analyses need to be reported whether or not they meet the researcher's initial expectations. When this is not the case, the rationale for selecting results needs be explained. Finally, sharing data and analysis scripts greatly benefits process transparency.

* *Example:* Hypothesizing after the results are known (or "HARKing") strongly goes against process transparency [@Kerr1998]. Researchers who do not have clear expectations should state research questions instead of hypotheses [@Cumming2013a].
* *Example:* Cherry-picking "convenient" results (e.g., results that best support the hypotheses), or trying multiple alternative analyses and reporting only those that are convenient clearly violates process transparency [@Simmons2011].
* *Example:* Even when a researcher has no preference for a given hypothesis and no intention to p-hack, cherry-picking results to give the impression of a coherent story also goes against process transparency [@Giner2012; @Gelman2013].
* *Example:* Provided an analysis is presented as exploratory, trying multiple analyses and reporting the most interesting and informative results by taking a neutral stance is perfectly acceptable and does not go against process transparency [@Tukey1977]. Transparency can however be increased by explaining what has been tried.


### 5. Clarity {-}

Study reports should be as easy to understand as possible, because as explained by Ronald Fisher (quoted above), readers and reviewers cannot judge an analysis without understanding. There are two facets of clarity: *ease of processing* and *accessibility*.

Study reports should be easy to process, even when they target experts. When results can be communicated more effectively with visual representations than with numerals, visual representations should be preferred [@Loftus1993; @Gelman2002]. Although a study report should communicate as much relevant information as possible, information overload must be avoided by reporting non-essential information in appendices or in supplemental material.

Second, study reports should ideally be accessible to most members of the HCI community, instead of being comprehensible by only a handful of specialists. The more accessible an analysis is, the more the "free minds" who can judge it. Thus a study report should be more an exercise of pedagogy than an exercise of rhetoric. The goal of a statistical report is not to signal expertise, but to explain.

* *Example:* Using statistics for defensive purposes by generating p-cluttered reports rather than informative plots violates the principle of clarity.
* *Example:* Excessive numbers of significant digits are difficult to process thus they go against the principle of clarity [@Ehrenberg1977], in addition to giving a misleading impression of precision [@Taylor1997].


### 6. Simplicity {-}

When choosing between two data analysis procedures, the simplest procedure should ideally be preferred even if it is slightly inferior in other respects. A focus on simplicity follows from the principles of clarity and ease of processing, and it makes both researcher mistakes and reader misinterpretations less likely to occur. In other words, the KISS principle (Keep It Simple, Stupid) is as relevant in transparent statistics as in any other domain.


### 7. Non-contingency {-}

When possible and outside exploratory analyses, data analysis and reporting strategies should avoid decisions that are contingent on data, e.g., “if the data turns out like this, compute this, or report that”. This principle follows from the principles of process transparency, clarity, and simplicity, because data-contingent procedures are hard to explain and easy to leave unexplained [@Gelman2013]. It is also a corollary of the principle of resilience because any dichotomous decision decreases a procedure’s resilience to statistical noise.

Carefully planning an analysis is a good way to make sure that the principle of non-contingency is met [@Cumming2014a], especially if all the analysis code has been written ahead of time based on pilot data [@Dragicevic2016]. Pre-registering an analysis further increases transparency by allowing anyone to verify that the plan has been followed [@Nosek2017; @Cockburn2018]. In exploratory analyses and in complex modeling problems, which are often data-contingent by nature, the principle of non-contingency should be applied to the best effort.

* *Example:* Using a test of normality to decide whether to use parametric or non-parametric methods violates the principle of non-contingency, in addition to not being very useful [@Stewart1995; @Wierdsma2013]. If the test of normality is not mentioned in the report, it additionally violates the principle of process transparency.
* *Example:* Selective reporting of data (i.e., cherry-picking) clearly violates the non-contingency principle, and generally also the principles of faithfulness and of process transparency. It is only acceptable if the analysis is clearly presented as exploratory, and if the goal of the selection is to learn from the data rather than to support a convenient hypothesis or story.


### 8. Precision and economy {-}

Data quality [@Gelman2017] and high statistical power [@Cohen1994], which in the estimation world translates to high statistical precision [@Cumming2013a; @Kruschke2017], are important goals to pursue. This is because even if full transparency is achieved, a study report where nothing conclusive can be said would be a waste of readers' time, and may prompt them to seek inexistent patterns. Precision depends on experiment design, but also on the choice of analysis methods -- thus analysis methods that yield high precision should be prefered. However, researchers should strive to avoid false precision, e.g., reporting numerical results without information about their uncertainty and/or with way more significant digits than justified by their uncertainty [@Taylor1997].

Analysis and reporting strategies that waste statistical power and precision (e.g., by dichotomizing continuous variables) should also be ideally avoided [@Dragicevic2016]. Though the economy principle is not directly related to transparency, it is generally advisable not to waste data. It is a sensible goal for researchers to try to learn as much as possible from a study, provided that the principles of faithfulness and process transparency are carefully kept in mind. For similar reasons, while it is essential that researchers do not read too much in their data and do not fall for confirmation bias, exploratory analyses are often very informative and should thus be encouraged. The best study reports combine prespecified with exploratory analyses, while clearly distinguishing between the two.


### 9. Material availability {-}

Sharing as much study material as possible is a core part of transparent statistics, as it greatly facilitates peer scrutiny and replication. Being able to run the experimental software and examine what participants saw (the techniques, tasks, instructions, and questions asked) is essential in order for other researchers to understand the details of a study. In addition, sharing the source code of the experimental software greatly facilitates replication. Similarly, experimental data (all data files and if possible analysis scripts) is necessary for conducting re-analyses and meta-analyses. Although uploading supplementary material makes sense during the reviewing phase, to be really useful all material should be freely shared online upon paper acceptance, ideally on a website that can guarantee long-term accessibility.

