Journal article Open Access

What p -hacking really looks like: A comment on Masicampo and LaLande (2012)

Lakens, Daniël


DataCite XML Export

<?xml version='1.0' encoding='utf-8'?>
<resource xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://datacite.org/schema/kernel-4" xsi:schemaLocation="http://datacite.org/schema/kernel-4 http://schema.datacite.org/meta/kernel-4.1/metadata.xsd">
  <identifier identifierType="URL">https://zenodo.org/record/235811</identifier>
  <creators>
    <creator>
      <creatorName>Lakens, Daniël</creatorName>
      <givenName>Daniël</givenName>
      <familyName>Lakens</familyName>
      <nameIdentifier nameIdentifierScheme="ORCID" schemeURI="http://orcid.org/">0000-0002-0247-239X</nameIdentifier>
    </creator>
  </creators>
  <titles>
    <title>What p -hacking really looks like: A comment on Masicampo and LaLande (2012)</title>
  </titles>
  <publisher>Zenodo</publisher>
  <publicationYear>2014</publicationYear>
  <dates>
    <date dateType="Issued">2014-12-06</date>
  </dates>
  <resourceType resourceTypeGeneral="JournalArticle"/>
  <alternateIdentifiers>
    <alternateIdentifier alternateIdentifierType="url">https://zenodo.org/record/235811</alternateIdentifier>
  </alternateIdentifiers>
  <relatedIdentifiers>
    <relatedIdentifier relatedIdentifierType="DOI" relationType="IsIdenticalTo">10.1080/17470218.2014.982664</relatedIdentifier>
  </relatedIdentifiers>
  <rightsList>
    <rights rightsURI="info:eu-repo/semantics/openAccess">Open Access</rights>
  </rightsList>
  <descriptions>
    <description descriptionType="Abstract">Masicampo and Lalande (2012; M&amp;amp;L) assessed the distribution of 3627 exactly calculated p-values between 0.01 and 0.10 from 12 issues of three journals. The authors concluded that "The number of p-values in the psychology literature that barely meet the criterion for statistical significance (i.e., that fall just below .05) is unusually large". "Specifically, the number of p-values between .045 and .050 was higher than that predicted based on the overall distribution of p."
There are four factors that determine the distribution of p-values, namely the number of studies examining true effect and false effects, the power of the studies that examine true effects, the frequency of Type 1 error rates (and how they were inflated), and publication bias. Due to publication bias, we should expect a substantial drop in the frequency with which p-values above .05 appear in the literature. True effects yield a right-skewed p-curve (the higher the power, the steeper the curve, e.g., Sellke, Bayarri, &amp;amp; Berger, 2001). When the null-hypothesis is true the p-curve is uniformly distributed, but when the Type 1 error rate is inflated due to flexibility in the data-analysis, the p-curve could become left-skewed below pvalues of .05.
M&amp;amp;L (and others, e.g., Leggett, Thomas, Loetscher, &amp;amp; Nicholls, 2013) model pvalues based on a single exponential curve estimation procedure that provides the best fit of p-values between .01 and .10 (see Figure 3, right pane). This is not a valid approach because p-values above and below p=.05 do not lie on a continuous curve due to publication bias. It is therefore not surprising, nor indicative of a prevalence of p-values just below .05, that their single curve doesn't fit the data very well, nor that Chi-squared tests show the residuals (especially those just below .05) are not randomly distributed.</description>
  </descriptions>
</resource>
396
207
views
downloads
Views 396
Downloads 207
Data volume 38.5 MB
Unique views 382
Unique downloads 200

Share

Cite as