Dryad
Published July 21, 2020 | Version v1
Dataset Open

Data from: Are replication rates the same across academic fields? community forecasts from the DARPA SCORE program

Creators

  • 1. Massey University
  • 2. .

Description

The DARPA program "Systematizing Confidence in Open Research and Evidence" (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioral sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in Psychology and in Education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.

Notes

In prediction markets, we used the logarithmic scoring rule (Hanson 2005) with base 2 and a liquidity parameter of b = 100. Participants received an initial endowment of 100 points to trade with.

Within the demography survey, empty values are indicated with NA values. The fields of interest question asks about 'sociology and criminology separately however for the other surveys and prediction markets these fields are combined. Therefore combing these fields of interest will make them consistent with the rest of the project.

The user_id's provided in the prediction market and survey data are consistent across all datasets.

A codebook is provided for explanations of column names

 

Funding provided by: Defense Advanced Research Projects Agency
Crossref Funder Registry ID: http://dx.doi.org/10.13039/100000185
Award Number: N66001-19-C-4014

Files

final_demography_data.csv

Files (1.7 MB)

Name Size Download all
md5:e54fd15fe4a4d0917a345f22871e50b3
9.8 kB Download
md5:5dc03696d36e1601af195983906cb993
886.8 kB Preview Download
md5:31cd8ce73a75b136ca805f67c01be488
187.8 kB Preview Download
md5:857de2c1306ca95bd76158e70672df94
25.2 kB Download
md5:395f15704a7f9ed25dc2ead5e0f7af3a
403.8 kB Preview Download
md5:e7b814ae1b94478e786751ca6f32e846
220.4 kB Preview Download
md5:835aeb150041225f2b9060d0bacf4035
9.2 kB Download

This site uses cookies. Find out more on how we use cookies