Non-Standard Errors

Albert J. Menkveld; Anna Dreber; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Michael Kirchler; Sebastian Neusüss; Michael Razen; Utz Weitzel

doi:10.2139/ssrn.3961574

Published November 24, 2021 | Version v1

Journal article Open

Non-Standard Errors

1. Vrije Universiteit Amsterdam, Tinbergen Institute
2. Stockholm School of Economics
3. University of Innsbruck
4. Aalto University
5. Radboud University, Vrije Universiteit Amsterdam

Contributors

Project members:

1. HEC Paris
2. Université d'Orléans
3. CNRS

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Online appendix available at: https://bit.ly/3DIQKrB.

Files