Source title,Type of source,Thematic code,Coded Text
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Exploration and qualitative research are important and useful, but rarely praised","Data exploration is not fundamentally a bad thing. In fact when conducted transparently, it is very useful. It may allow you to discover something for which theory has not even been developed yet, or you may actually correctly identify a complex pattern ofinteractions for which theory is too simplistic."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Exploration and qualitative research are important and useful, but rarely praised",In some fields it is common practice to masquerade exploratory analyses as confirmatory hypothesis testing because exploratory work is often perceived as inferior or old-fashioned.
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"Exploration and qualitative research are important and useful, but rarely praised","There is also a divergence in interest between different parties in the overall research and development pipeline. Preclinical researchers need freedom to explore the borders of knowledge, while clinical researchers rely on replication to weed out false positives."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,The location and context of experimental research will affect results,"This is more easily said than done. The temptation of doing research in the easily available cohorts of undergraduate students at one’s own university is huge. I am far from criticising such research: it can be useful and valuable and I have been involved in several studies of this type myself (Bak, Vega-Mendoza, et al., 2014; Vega-Mendoza et al., 2015). But this should not be the only type of research being done. We need to go out and study populations across different countries, cultures and indeed continents (Bak & Alladi, 2015). If different environments influence physical observations (as in the example of boiling water above) as well as cognitive psychology findings (Henrich, Heine, & Norenzayan, 2010), why should we expect bilingualism to be an exception? I would not like to be misunderstood as an advocate of radical cultural relativism. It is very likely that many characteristics of human behaviour are universal. The seminal work of Paul Ekman on perception of emotions provides an impressive example of someone who set out to find differences and found similarities, leading to a truly universal theory (Ekman & Friesen, 1971; Ekman et al., 1987). But the only way we can find out whether something is universal is to do comparative research."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,The location and context of experimental research will affect results,"Until recently, most studies on bilingualism and cognition came from a relatively small number of countries, almost exclusively in the Western World. A good example of the problems connected with this lack of representativity is the intense debate about the role of immigration as a confounding variable in bilingualism research. Indeed, in almost all studies from the USA and several from Canada, bilingualism tended to be associated with an immigration background, making it difficult to disentangle the two phenomena (Fuller-Thomson & Kuh, 2014). To make things more complex, the association between bilingualism and immigration can also work the other way round, with the autochthonous population being originally bilingual and the immigrants (albeit often from different regions of the same country) being monolingual (as is the case in parts of the UK and Spain) (Bak, 2016). Given that both bilingualism and immigration might be associated with cognitive differences (Fuller-Thomson, Milaszewski, & Abdelmessih, 2013) both factors can either potentiate or cancel each other, explaining some superficially contradictory results (Bak, 2015). A good way forward in this case is to study societies in which bilingualism and immigration can occur independently of each other. Numerous studies taking this approach have shown beyond any reasonable doubt that the bilingual advantage is not always dependent on the immigration status (Alladi et al., 2013; Alladi et al., 2015; Bak, Nissan, et al., 2014; Ljungberg, Hansson, Andrés, Josefsson, & Nilsson, 2013; Woumans et al., 2015), although in other contexts immigration might constitute an important variable to be taken into account."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,The location and context of experimental research will affect results,"Another example, in which a comparison of results from different countries can be illuminating is the relationship between education and dementia, in which dementia is associated with lower educational level. This association has been reported not only from Europe and North America but also from Brazil (Cesar et al., 2015). However, in India bilingualism seems to have a stronger effect on dementia than education (Iyer et al., 2014) and the bilingual delay in the onset of dementia in illiterates is in fact larger than in the literate population (Alladi et al., 2013). In order to explain this discrepancy in findings, we need to examine the variables associated with education and literacy: in many countries, low education and/or illiteracy are associated with a whole range of negative variables, from unemployment and social deprivation to drug abuse and criminality. In India, in contrast, it is possible to find illiterates who are fully employed and well integrated into the society. We cannot simply expect to find the same associations of factors and causal relationships across countries and societies (Bak & Alladi, 2015). Moreover, it is important to bear in mind that in many societies bilingualism can be associated with higher as well as lower educational, professional and economic background, so its effects can be either potentiated or attenuated by these variables (Bak 2016)."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,The location and context of experimental research will affect results,"But experiencing it can lead to the realisation of the influence of altitude on basic physical processes. Doing bilingualism research in Toronto and San Francisco, Edinburgh and Barcelona, Hyderabad and Hong Kong might not be the same either."
Reproducible Research A Retrospective,"Report, policy document or website",Research is becoming more technically complex,"In much of the early literature on reproducible research, the focus is on “computationally- intensive” research which, because of its reliance on complex computer algorithms, was considered perhaps more impenetrable than other research. Fast forward 30 years and the use of computing in scienti?c research is ubiquitous. It is no longer the domain of niche geophysical scientists or mathematical statisticians using obscure computer packages. Now, all scienti?c research involves the use of powerful computers, whether it is for the data collection, the data analysis, or both. Furthermore, the increase in complexity of statistical techniques over this time period has resulted in the need for detailed descriptions of analytic approaches and data processing pipelines. We are all computational scientists now and as a result the concept of reproducibility is relevant to all scientists."
About ReproZip,"Report, policy document or website",Research is becoming more technically complex,"First, computational environments are complex, consisting of many layers of hardware and software, and the configuration of the OS is often hidden."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Research is becoming more technically complex,"Some technological advances involve increased complexity of methods and analysis. This can lead to problems unless matched by understanding of that complexity, such as the high risk of false positives when dealing with complex, multivariate datasets."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Statistical testing contributes to the 'addiction for oversimplisfication'.,"Statistical testing (like alcohol) often gives the wrong impression that complex decisions can be oversimplified without negative consequences, for example, by making decisions solely because p was above or below some cutoff like 0.05. And many researchers are addicted to such oversimplification. These addictions are worth breaking."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"The ultimate aim of research is to pursue new discoveries, not just to be reproducible","It should also be stressed that the goal of the entire bioscience research enterprise is increased knowledge and progress, not the highest achievable level of reproducibility, however de?ned. If the latter were to be taken as the dominant goal, it is very likely that the former would suffer. And a world in which highly reproducible papers combine to produce insights and discoveries of limited impact would not be a desirable one. So balance and judgment in addressing this issue are in the end critical, and these have not always be evident in much of the public discussion of this subject."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","The ultimate aim of research is to pursue new discoveries, not just to be reproducible","While the lack of reproducibility is a serious problem, it is not to the extent of a crisis. The problem is endemic to the research process and cannot be solved at once, as scientists constantly strive to find explanations that fit both old and new results. Policy-makers need to set expectations at the right level: today, researchers who adopt good practice in reproducibility are working a double-shift. From the perspective of scientists, a greater crisis in science is when citizens and funders stop"
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","The ultimate aim of research is to pursue new discoveries, not just to be reproducible",Believing in the capacity of science to address societal needs and in the need for the state to support it. The crisis narrative around reproducibility does not assist policy makers.
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,"Variation will persist despite rigorous, honest, valid research","Firstly, the fact that the same experiment does not replicate in every place does not have to be due to errors, incompetence, bad will, bias, let alone dishonesty of the researchers involved. It can reflect differences in the environment in which the study has been conducted. Secondly, and equally importantly, it does not matter how many measurements we can get from hundreds of coastal cities in the world, it does not invalidate one single measurement from La Paz."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,"Variation will persist despite rigorous, honest, valid research","Firstly, the fact that the same experiment does not replicate in every place does not have to be due to errors, incompetence, bad will, bias, let alone dishonesty of the researchers involved. It can reflect differences in the environment in which the study has been conducted."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Variation will persist despite rigorous, honest, valid research","When a replication fails to con?rm the original result, this is often interpreted as context dependence (e.g. ‘this was a wetter year’, ‘this location contained more conifers’, or ‘these animals were raised on a higher-protein diet’). After all, we know that ecology and behaviour are highly complex and we expect variability. However, in this situation context dependence is simply an untested post-hoc hypothesis. We cannot claim that divergent results stem from context dependence without explicit testing with new data. It may be that the difference in effect sizes observed in the two studies is no larger than what one would expect from chance alone (sampling noise). Meta-analysts (researchers who summarize effect sizes across numerous studies) are very familiar with this idea, and they quantify the extent of disagreement between studies as ‘heterogeneity’ in effect sizes."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Variation will persist despite rigorous, honest, valid research","If we are truly interested in context dependence and its sources, then we should design replication studies explicitly to investigate context dependence. This means systematically evaluating environmental variables that we hypothesize may be driving context dependence using ‘replication batteries’, in which conditions hypothesized to drive differences in results are manipulated while attempting to hold other variables constant (Kelly, 2006). Post-hoc hypotheses about context dependence are valid, but they remain nothing more than hypotheses before studies have been designed and implemented speci?cally to evaluate them."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Variation will persist despite rigorous, honest, valid research","In cases where a new result appears inconsistent with a prior report, if the new study employed experimental conditions and reagents distinct from the ?rst, it should not be taken as a failure to replicate it."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","Variation will persist despite rigorous, honest, valid research","Variations in the rates of reproducibility may be linked to differences across disciplines, for instance in the complexity of experimental design, in the statistical methods used, in the culture of transparency, and in the data sharing and replication practices. Therefore, lack of reproducibility may have both reasons that are endogenous to the research process: the complexity, specificities and constraints of specific research designs;10 and human-related reasons, malicious and accidental, in relation to specific truth claims."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,We cannot expect 100% reproducibility rates,"However, the assertion that a 50% level of reproducibility equates to a crisis, or that many of the original studies were really fruitless, has been disputed by some specialists in replication. “A 50% level of reproducibility is generally reported as being bad, but that is a complete misconstrual of what to expect”, commented Jeffrey Mogil, who holds the Canada Research Chair in Genetics of Pain at McGill University in Montreal. “There is no way you could expect 100% reproducibility, and if you did, then the studies could not have been very good in the first place. If people could replicate published studies all the time then they could not have been cutting edge and pushing the boundaries”."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,We cannot expect 100% reproducibility rates,"One reason not to expect 100% repro-ducibility in preclinical studies is that cutting edge or exploratory research deals with a lot of uncertainty and competing hypotheses of which only a few can be correct. After all, there would be no need to conduct experiments at all if the outcome were completely predictable. For that reason, initial preclinical study cannot be absolutely false or true, but must rely on weight of the evidence, usually using the P-test as a tiebreaker."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Conflicts of interest may bias the conclusions of research and affect reproducibility,"Researcher having links to commercial/pharmaceutical development. Though these con?icts of interest surely exist, and must be disclosed, their contribution to the problem of research irreproducibility writ large, especially as relates to basic and translational research, is likely quite small compared to the more prevalent incentives related to career considerations."
How to Make More Published Research True,Article,Conflicts of interest may bias the conclusions of research and affect reproducibility,"Dissociation of some research types from specific conflicted sponsors or authors has been proposed (not without debate) for designs as diverse as cost-effectiveness analyses, meta-analyses, and guidelines. For all of these types of research, involvement of sponsors with conflicts has been shown to spin more favorable conclusions."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Conflicts of interest may bias the conclusions of research and affect reproducibility,"Prevalent conflicts of interest can also affect the design, analysis, and interpretation of results. Problems in study design go beyond statistical analysis, and are shown by the poor reproducibility of research. Researchers at Bayer3 could not replicate 43 of 67 oncological and cardiovascular findings reported in academic publications. Researchers at Amgen could not reproduce 47 of 53 landmark oncological findings for potential drug targets. The scientific reward system places insufficient emphasis on investigators doing rigorous studies and obtaining reproducible results."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Conflicts of interest may bias the conclusions of research and affect reproducibility,"Industry-supported systematic reviews obtain favourable results more often than do other systematic reviews, although the difference lies more in the interpretation of the results rather than in the actual numbers."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Conflicts of interest may bias the conclusions of research and affect reproducibility,"Research is often done by stakeholders with conflicts of interest that favour specific results. These stakeholders could be academic clinicians, laboratory scientists, or corporate scientists, with declared or undeclared financial or other conflicts of interest."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Human error may lead to irreproducible research,"The confrontational nature of the debate offers a temptation to reduce the complexity of the matter to well-sounding but inadequate slogans. It pushes the adversaries towards a partisan myopia and “selective scepticism”, in which a detailed scrutiny of the other side is combined with an uncritical acceptance of everything that can be used in one’s favour. After all, one would not expect the prosecution to search for arguments that would strengthen the defence and vice versa. And once the arguments are exhausted, an adversarial debate risks shifting the emphasis from the relevant topics to the individuals representing them; taken to extremes it can lead to conspiracy theories."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Human error may lead to irreproducible research,This might direct our attention to different phenomena and influence our expectations as well as our interpretation of the existing data. But these differences could be more profound than the interpretation of the data – they could affect the data as well.
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Human error may lead to irreproducible research,"Expectations can influence performance. Participants who believe that the very fact of them being able to speak a second language is a disadvantage might perform differently from those perceiving their bilingualism as an asset. So far, there has been very little dialogue between cognitive and social research on bilingualism – and yet, this might be one of the most promising avenues to advance our understanding of this topic."
Biomarker development in the precision medicine era: Lung cancer as a case study,Review,Human error may lead to irreproducible research,"Statistical reproducibility issues arise owing to biases in the original sample or validation cohort, or owing to false discoveries."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Human error may lead to irreproducible research,"Hindsight bias is particularly dangerous because we overestimate the plausibility of our hypothesis (which in fact is a post hoc explanation, a hypothesis that was derived from the data, not one that we had a priori)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Human error may lead to irreproducible research,"Once a discovery has been made (P<0.05) and a plausible explanation has been found, it is very easy to deceive oneself into thinking that one actually had that hypothesis in mind before starting the exploration, and nothing seems wrong with writing up a publication saying ‘here we test the hypothesis that ... ’"
Detecting and avoiding likely false-positive findings – a practical guide,Article,Human error may lead to irreproducible research,"Since we often believe that an effect of interest exists (and we designed the experiment to reveal the effect), we tend to have greater trust in analyses that con?rm our belief. This powerful component of human nature is called con?rmation bias, and it has been documented in a wide array of settings (Nickerson, 1998). Obviously, con?rmation bias can render our science highly subjective unless we make all these arbitrary decisions a priori (ifpossible) or at least blind to the outcome."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Human error may lead to irreproducible research,"Somewhat surprisingly, it appears that the human brain has not evolved to maximize the objectivity of its judgements (Haselton, Nettle & Murray, 2005). Accordingly, psychologists have described a near-endless list of cognitive biases that in?uence our perception, reasoning and memory."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Human error may lead to irreproducible research,"Research publications will contain errors, despite procedures designed to avoid them. Fortunately, a fundamental attribute of science is its capacity for “self-correction”, through published ideas and claims being reviewed and tested by others."
A manifesto for reproducible science,Review,Human error may lead to irreproducible research,"The combination of apophenia (the tendency to see patterns in random data), confirmation bias (the tendency to focus on evidence that is in line with our expectations or favoured explanation) and hindsight bias (the tendency to see an event as having been predictable only after it has occurred) can easily lead us to false conclusions."
A manifesto for reproducible science,Review,Human error may lead to irreproducible research,"During data analysis it can be difficult for researchers to recognize or data dredging because confirmation and hindsight biases can encourage the acceptance of outcomes that fit expectations or desires as appropriate, and the rejection of outcomes that do not as the result of suboptimal designs or analyses. Hypotheses may emerge that fit the data and are then reported without indication or recognition of their post hoc origin. This, unfortunately, is not scientific discovery, but self-deception."
A manifesto for reproducible science,Review,Human error may lead to irreproducible research,We need measures to counter the natural tendency of enthusiastic scientists who are motivated by discovery to see patterns in noise.
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Human error may lead to irreproducible research,Errors favoring the publication of statistically significant results have been proposed as major contributing factors in the reproducibility crisis that is heavily debated in many scientific fields
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Human error may lead to irreproducible research,"These issues are often related to misuse of statistical methods, which is accentuated by inadequate training in methods."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Human error may lead to irreproducible research,"Even when errors are detected in published articles, detection is often a long time after publication, and refuted results might be cited for many years after they have been discredited."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Human error may lead to irreproducible research,"Errors in published research can be honest, such as typographical errors in data tables or broken links to source material."
Reproducible Research A Retrospective,"Report, policy document or website",Human error may lead to irreproducible research,"Because of the increasing complexity of data analyses, many choices and decisions must be made by analysts in the process of obtaining a result. With these increasing complexities we also increase the risk of human error and bias in data analysis. These choices and decisions often have an unknown impact on the ?nal estimates produced and therefore may or may not be recorded by the investigators. These “research degrees of freedom” allow investigators to unknowingly, or perhaps knowingly, steer data analyses in directions that may support speci?c hypotheses rather than represent all of the evidence in the data."
How we can make ecotoxicology more valuable to environmental protection,Note,Human error may lead to irreproducible research,"Relying on expert judgment alone can be problematic, as it is often inconsistently applied between reviewers (e.g., depends on the reviewers' expertise and availability, as well as their own biases (Mahoney, 1977))."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Human error may lead to irreproducible research,Researchers compute the exact p-value as Fisher did but also mechanistically reject H0 and accept the unde?ned H1 if p ? (? = 0.05) without ?exibility following the behavioral decision rule of Neyman and Pearson.
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Human error may lead to irreproducible research,"Scientists are trained to avoid or reduce the probability of error; however, they may mistakenly draw false positive and false negative conclusions from their experiments. In addition to errors of interpretation, scientists may draw inaccurate conclusions from experimental data in a way that reflects bias (Mahoney, 1977). Such contexts make the process ofscientific knowledge accumulation difficult, and may lead to inflated confidence about phenomena of interest (e.g., Doyen, Klein, Pichon, & Cleeremans, 2012). Scientific claims to knowledge are, therefore, considered tentative, and confidence in claims should correspond with the amount and quality ofevidence."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Human error may lead to irreproducible research,"There are many factors that bias the decision-making of authors, reviewers and editors throughout the publication process to the detriment of a reliable evidence base. In the absence of external pressures, the simple human desire for seeking information that supports one’s beliefs, and ignoring that which does not, means authors are more likely to find, and reviewers to believe, evidence that confirms accepted theories."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Full data sharing adds burdens to the research process,"The most common ways of sharing data that were reported tend to be suboptimal, with email being the most common method for private data sharing (Allin 2018) and journal supplementary materials being most common for public data sharing (email is not secure enough for private data sharing; data repositories are preferred over supplementary materials for public data sharing) (Michener 2015)"
Promises and pitfalls of data sharing in qualitative research,Note,Full data sharing adds burdens to the research process,"First, the burden of organizing qualitative data for inspection or use by external investigators could easily exceed the work of writing the manuscript itself. How should the interests of research transparency be weighed against the potential costs of documentation burden? Redacting the hundreds of pages of transcripts collected during the course of a small qualitative study would require months of work."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Journals are expected to detect problems, but may not have the resources to do so","Another contextual factor related to the pressure to publish in “A-journals” is the increase in the number of manuscript submissions, causing an ever-growing workload on editors and reviewers. Many journals receive more than 1,000 submissions a year, making it necessary for many action editors to produce a decision letter every three days or so—365 days a year (Cortina et al., 2017a). But, the research performance of editors and reviewers is still contingent on their own publications in “A-journals” (Aguinis, de Bruin, Cunningham, Hall, Culpepper, & Gottfredson, 2010a). So, the increased workload associated with the large number of submissions, along with other obligations (e.g., teaching, administrative duties), suggests that our current system places enormous, and arguably unrealistic, pressure on editors and reviewers to scrutinize manuscripts closely and identify areas where researchers need to be more transparent (Butler et al., 2017)"
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Making sense of the large datasets produced by these technologies requires sophisticated statistical and computational methods, aswell as substantial computational power. This has led to an acute crisis in life sciences, as researcherswithout informatics training attempt to perform computation-dependent analyses."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques",Generally we feel that there is insuf?cient recognition of the extent to which incorrect P-values resulting from pseudoreplication have contributed to the current reliability crisis.
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","De?cient oversight, training and mentorship In most modern biomedical labs, the responsible leader is the principal investigator, but most or all of the work is carried out by students, post docs or technicians. One of the responsibilities of the PI is to assure that their trainees receive proper education in experimental conduct and data analysis. Although provision of formal curricula is necessary, education must go beyond lectures on research methodologies and responsible conduct, the impact of which, even if well designed, may be limited [31]. There must also be increased attention to behaviors and attitudes in group and individual meetings that reinforce correct approaches. The “hidden curriculum”, e.g. how a lab leader approaches issues of appropriate research conduct in real time, may be more important than the formal curricula in these areas."
Practical Computational Reproducibility in the Life Sciences,Note,"Lack of training and mentoring, including in computing analysis, coding and novel techniques",There are also substantial cultural differences among research ?elds in the degree of software openness that will need to be tackled.
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","And even in cases when the code is published, it is sometimes incomplete and, for instance, does not include the code for data preprocessing, parameter tuning, or the exact evaluation procedures."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","The code of the core algorithms seems to be more often shared by researchers than in the past, probably also due to the fact that reproducibility has become an evaluation criterion for conferences. However, in many cases, the code that is used for hyper-parameter optimization, evaluation, data pre-processing, and for the baselines is not shared. This makes it difcult for others to validate the reported fndings."
On the Reproducibility of Psychological Science,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Another unintended consequence of the reproducibility challenge (that was not one of the original goals) is that the code repository serves as a useful teaching resource. In our experience, students new to information retrieval often struggle with basic tasks such as indexing and performing baseline runs. Our resource serves as an introductory tutorial that can teach students about the basics of working with IR test collections: indexing, retrieval, and evaluation."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","As the complexity of a software program increases, the likelihood of undiscovered bugs quickly reaches certainty. This implies that the software that is used
for fMRI analysis is likely to contain bugs. Most fMRI researchers use one of several open-source analysis packages for pre-processing and statistical analyses; many additional analyses require custom programs. Because most researchers writing custom code are not trained in software engineering, there is insufficient attention to good software-development practices that could help to catch and prevent errors. This issue came to the fore recently, when a 15-year-old bug was discovered in the AFNI program 3dClustSim (and the older AlphaSim), which resulted in slightly inflated type I error rates (the bug was fixed in May 2015)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Finally, researchers need to acquire sufficient training on the implemented analysis methods, particularly so that they understand the default parameter values of the software (such as cluster-forming thresholds and filtering cut-offs), as well as the assumptions on the data and how to verify those assumptions"
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Finally, researchers need to acquire sufficient training on the implemented analysis methods, particularly so that they understand the default parameter values of the software (such as cluster-forming thresholds and filtering cut-offs), as well as the assumptions on the data and how to verify those assumptions"
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Second, data analysis is highly technical, can be very flexible, and many analytical choices have to be made on how exactly to analyze the results; and a large number ofexploratory tests can be run on the vast amount ofdata collected in each brain imaging study. This allows for running a very high number ofundocumented and sometimes poorly understood and difficult to replicate idiosyncratic analyses influenced by a large number ofarbitrary ad hoc decisions. These, in their entirety, may be able to generate statistically significant false positive results with high frequency [27,33–35], especially when participant numbers are low."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Standards for radiomic features have not yet been universally adopted; therefore, software should be described because differences due to feature extraction are likely to in?uence the apparent stability of features."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","The total number of published predictive modeling studies using image-based quantitative features has been rapidly rising, but global consensus about features that are repeatable and reproducible has not yet emerged. Lack of uni?ed synthesis could potentially undermine future discussions about clinical applicability and prospective multi-institutional external-validation trials."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","MISEV2014 reflected a field in flux, with a need for standard development tempered by remaining uncertainties. Indeed, this is still the case."
How to Make More Published Research True,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques",Better training of scientific workforce in methods and statistical literacy
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","These issues are often related to misuse of statistical methods, which is accentuated by inadequate training in methods."
Our path to better science in less time using open data science tools,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Environmental scientists are expected to work effectively with ever­increasing quantities of highly heterogeneous data even though they are seldom formally trained to do so. This was recently highlighted by a survey of 704 US National Science Foundation principal investigators in the biological sciences, which found training in data skills to be the largest unmet need. Without training, scientists tend to develop their own bespoke workarounds to keep pace, but with this comes wasted time struggling to create their own conventions for managing, wrangling and versioning data. If done hap hazardly or without a clear protocol, these efforts are likely to result in work that is not reproducible—by the scientist’s own ‘future self’ or by anyone else."
Our path to better science in less time using open data science tools,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Open data science tools helped us upgrade our approach to reproducible, collaborative and transparent science, but they did require a substantial investment to learn, which we did incrementally over time."
Our path to better science in less time using open data science tools,Article,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","From our own experience and from teaching other academic, we find that the main barriers to engagement boil down non­profit, and government groups through the Ocean Health Index project to exposure and confidence: first knowing which tools exist that can be directly useful to one’s research, and then having the confidence to develop the skills to use them. These two points are simple but critical. We are among the many environmental scientists who were never formally trained to work deliberately with data. Thus, we were unaware of how significantly open data science tools could directly benefit our research."
On the issue of transparency and reproducibility in nanomedicine,Letter,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","For example, lack of sufficient understanding of emerging materials does not allow fast evaluation and standardization."
On the issue of transparency and reproducibility in nanomedicine,Letter,"Lack of training and mentoring, including in computing analysis, coding and novel techniques",Inconsistencies in nanomaterials reporting standards have major roots in inadequate training and lack of familiarity with relevant biological and analytical methodologies and their limitations when applied to the bio–nano arena.
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Better Training and Better Use of More Statistical Methods: from Believers to Thinkers A core problem seems to be that the statistical subject knowledge of many researchers in biomedical and social science has been shown to be poor (Oakes, 1986; Gliner et al., 2002; Castro Sotos et al., 2007, 2009; Wilkerson and Olson, 2010; Hoekstra et al., 2014). NHST perfectly ?ts with poor understanding because of the perceived simplicity of interpreting its outcome: is p ? 0.05 (Cohen, 1994)?"
What you see is what you get? Enhancing methodological transparency in management research,Review,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Because of the financial constraints placed on business and other schools (e.g., psychology, industrial and labor relations), many researchers and doctoral students are not receiving state-of-the-science methodological training. Because doctoral students receive tuition waivers and stipends, many schools view doctoral programs as cost centers when compared with undergraduate and master’s programs. The financial pressures faced by schools often result in less resources being allocated to training doctoral students, particularly in themethodsdomain (Byington&Felps,2017; Schwab &Starbuck, 2017;Wright, 2016)."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","The sheer volume of submissions requires expanding editorial boards to include junior researchers, even at “A-journals.” Unfortunately, these junior researchers themselves may not have received rigorous and comprehensive methodological training because of the financial constraints on schools and departments. The lack of broad and state-of-the-science meth-
odological training, the rapid developments in research methodology (Aguinis, Pierce, Bosco, & Muslin, 2009; Cortina et al., 2017a), and the sheer volume and variety of types of manuscript submissions mean that even the gatekeepers can be considered novices and, by their own admission, often do not have the requisite KSAs to adequately and thoroughly evaluate all the papers they review (Corley & Schinoff, 2017)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","Eliminating HARKing requires an orchestrated effort to seriously change deeply embedded practices in the scholarly community (Ioannidis, 2005, 2012). What we can do, for now, is ?rstly to reduce the focus on single test statistics when assessing results in favor of comprehensive assessments, and thereby to reduce the incentives to engage in HARKing (hence our guidelines 1–9), and secondly to mentor and train a new generation of scholars to intrinsically dislike HARKing practices. Here, key is that established scholars lead by example."
Reproducibility and Research Integrity,Note,"Lack of training and mentoring, including in computing analysis, coding and novel techniques","However, it is important to note that scientists may not always know all the factors that could impact research outcomes. For example, Sorge et al (2014) found that exposure to human male odors, but not female odors, induces pain inhibition and stress in mice and rats. Failure to record the sexes of the researchers conducting experiments on rodents involving measurements of pain responses could therefore undermine reproducibility. Prior to the publication of this finding, many researchers would have not recorded or reported the sexes of the experimenters, or taken this information into account in studies or pain or stress in rodents. If researchers are having difficulty reproducing the results of an experiment, they may need to reexamine methods, materials, and procedures to determine whether they have overlooked an important detail."
Practical Computational Reproducibility in the Life Sciences,Note,Lack of familiarity with computing and software for reproducibility,"We are reaching the point where not performing data analyses reproducibly becomes unjusti?able and inexcusable. Aside from hardening the software, the main challenges ahead are in education and outreach that will be critical for fostering the next generation of researchers."
Recommendations for open data science,Note,Lack of familiarity with computing and software for reproducibility,"Train scientists in data science A critical element of a cultural revolution toward high standards for data analysis will be engaging young scientists in data science training. Valuable new tools are emerging for facilitating open data science such as Github for sharing code, and Project Jupyter notebooks for publishing reproducible analyses. However, students rarely receive formal training in current data science tools and techniques. Moreover, they often face pressure to produce results as quickly as possible. As a result, scientists may produce hastily and poorly written code that they are reluctant to share and is unusable by others."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Replication studies are unattractive (lower reputation, excitement and originality)","However, a perceived lack of prestige, excitement, and originality of replication plagues L2 research (Porte, 2012), as it does other disciplines (Berez-Kroeker et al., 2017; Branco, Cohen, Vossen, Ide, & Calzolari, 2017; Chambers, 2017; Schmidt, 2009), and these perceptions are thought to have caused, at least in part (directly or indirectly), alleged low rates and a poor quality of published replication studies."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Replication studies are unattractive (lower reputation, excitement and originality)","On a ?nal note about nomenclature, in the current reviews, we have used the term initial study rather than original study when referring to studies that were replicated. This is because studies are rarely ifever truly original in the sense of being a completely novel idea. Also, original carries negative connotations for its replication because it could imply that anything that is not original cannot share other characteristics broadly associated with originality, such as being innovative, fundamental, or agenda setting."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Replication studies are unattractive (lower reputation, excitement and originality)","The low rate of replication is likely due in part to a lack of willingness to self-label as replication (Neulip & Crandall, 1993; Polio, 2012b). This reticence is complex. Anecdotally, we observed during colloquia discussing this study and the research by Morgan-Short et al. (2018) that some researchers reported actively undertaking and promoting replication with students and in their own work, yet they were less enthusiastic about labeling these studies as replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Replication studies are unattractive (lower reputation, excitement and originality)","Our data demonstrate that the perceived low prestige of replication research is unfounded in at least two respects: perceived ease and perceived low impact. Carrying out well-justi?ed, carefully administered replications that are rigorously analyzed in relation to their initial study is no trivial task and very rare in self-labeled replications to date. Our data also show that replications have been relatively highly cited and have been published in some ofthe highest impact journals."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Replication studies are unattractive (lower reputation, excitement and originality)","Recommendation: When the initial study is cited, also cite (at least any direct and partial) replication studies of it."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"Replication studies are unattractive (lower reputation, excitement and originality)","First, although technological developments (such as platforms to support preregistration, open materials, data, and software) facilitate multisite replication projects that gather large data sets (e.g., Morgan-Short, Marsden, Heil, et al., 2018), the perceived extra effort these approaches require can deter researchers, especially given the lack of assurance ofeventual publication."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"Replication studies are unattractive (lower reputation, excitement and originality)","A ?nal impediment to replication research is a perception, at many levels, that replication research has low impact and prestige, although Marsden et al. illustrate that, in fact, self-labeled replications have been relatively very well cited and published by journals with high impact factors. In sum, despite multiple calls for increased replication research (e.g., Polio&Gass, 1997; Porte, 2012), cultural and structural issues such as these may have systematically hindered replication efforts."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Replication studies are unattractive (lower reputation, excitement and originality)","There are two interrelated explanations for this. First, many researchers have not yet come to appreciate the important role of replication in developing robust scienti?c inference, and second, the institutions that in?uence scientists’ choices do not reward close replication (Nosek, Spies & Motyl, 2012)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Replication studies are unattractive (lower reputation, excitement and originality)","Seeking novelty and discovery may be emotionally rewarding, but in light of the currently low thresholds for reaching nominal signi?cance, isolated, unreplicated reports of?ndings should be regarded as preliminary until con?rmed by rigorous replication studies. The term ‘evidence’ should be used more cautiously (when there is consensus from con?rmatory tests) and the expression ‘as predicted’ should maybe be limited to predictions that have been documented or to those that strictly follow from theory. The crucial second step from exploratory to con?rmatory research should be encouraged by funding bodies supporting rigorous replication studies and by citation practices of researchers who might want to prefer citing the rigorous con?rmatory over the initial exploratory study."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website","Replication studies are unattractive (lower reputation, excitement and originality)","Despite the importance of replications, there are a number of barriers and challenges to conducting and disseminating replication research, including a real or perceived bias by funding agencies, grant reviewers, and journal editors toward research that is novel, innovative, and groundbreaking (Travers, Cook, Therrien, and Coyne, 2016)."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,"Some researchers, especially younger generations, see replication and reproducibility as worthy endeavours","Importantly – and perhaps surprisingly to those who are less familiar with our highly collaborative field – 55% of respondents to the MISEV survey said that they or their laboratories could “devote reasonable resources” to evaluations, if these were to become necessary for MISEV updates. The language was kept intentionally vague, but “evaluations” could mean anything from literature reviews to technical/experimental comparisons. It is encouraging that such a large percentage of participants indicated a willingness to engage in coordinated activities. As needed, a Standardization Committee of ISEV could be formed to establish working groups on topics that have not been fully addressed and thus focus the substantial energies of the community."
On Replication in Communication Science,Editorial,"Some researchers, especially younger generations, see replication and reproducibility as worthy endeavours",Our (admittedly nonscientific) read of the field is that most social scientific researchers want to see a greater emphasis on replication and view replications as a worthy goal.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","Policies have historically worked with data, but may not work with reproducibility","Journal publishers have also been active on reproducibility; they have produced guidelines and policies for proposers, both in large-scale, collaborative efforts, and independently of each other. Journals, currently 63 of them, have introduced ‘badges’ for good open data and reproducibility value of a paper. Specific reproducibility policies of scientific journals have been tested and found necessary but wanting in effectiveness; policies may increase data sharing but not necessarily reproducibility."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Changing practices and cultures takes time and collective action,"Although scientists must take ultimate responsibility for their standards, ethics, training environments, validation of reagents, experimental results, and reproducibility of their published data, the complex web of individuals contributing to the current reproducibility crisis is worth mentioning. Journal editors attend many meetings, socialize and network with hundreds of scientists, and compete for the most exciting papers from the best labs. Editors in turn are aware that the most tantalizing papers will garner the greatest media visibility and, ultimately, will indirectly attract more page views, advertising revenue, and boost the journal’s reputation, impact factor, and pro?tability."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Changing practices and cultures takes time and collective action,"Changes to the culture of scienti?c research It is well known that efforts to change culture, apart from changing rules, are exceptionally dif?cult and often unsuccessful."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Changing practices and cultures takes time and collective action,"Whatever the measures to improve reproducibility of biomedical research, it will have to involve all actors from researchers to funding agencies to journals and eventually commercial players as the main customers of academic research. In addition, it may also require new ways to conduct research and validate experimental data."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Changing practices and cultures takes time and collective action,Laboratories that were unaware of the MISEV effort before its publication may simply not have had time to incorporate the guidelines by the time our survey took place.
Raising research quality will require collective action,"Report, policy document or website",Changing practices and cultures takes time and collective action,"But one institution will make little difference on its own. For better practices to become the norm, many universities need to act collectively. Changes to incentives at a single institution will not make new behaviours stick, not least because practices required in only one place can act as a career tax on its scientists. Only if changes occur across many institutions will the impacts permeate scientific culture."
Raising research quality will require collective action,"Report, policy document or website",Changing practices and cultures takes time and collective action,"Strategy for breakfast — grand plans founder on the rocks of implicit values, beliefs and ways of working. Top-down initiatives from funders and publishers will fizzle out if they are not implemented by researchers, who review papers and grant proposals. Grass-roots efforts will flourish only if institutions recognize and reward researchers’ efforts. Funders, publishers and bottom-up networks of researchers have all made strides. Institutions are, in many ways, the final piece of the jigsaw. Universities are already investing in cutting-edge technology and embarking on ambitious infrastructure programmes. Cultural change is just as essential to long-term success."
"Most computational hydrology is not reproducible, so is it really science?",Note,Changing practices and cultures takes time and collective action,"As has guided our recommendations we make above, there is wide recognition that gradual steps are required to change a deeply engrained research culture that does not currently require reproducibility [Bailey et al., 2016; Peng,2011; Koutsoyiannis et al., 2016]."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Changing practices and cultures takes time and collective action,"A ‘one size fits all’ approach is unlikely to be effective and in most cases, no single measure is likely to work in isolation. It will take time to identify and implement the most effective solutions."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Changing practices and cultures takes time and collective action,"As it emerges from the literature, there are cultural and economic drivers for the lack of reproducibility, and it will take time and resources to address them."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Even if recommendations exist, they are rarely taken up by journals and labs","These recommendations include calls for random assignment of animals, blinding of preclinical treatment groups, more rigorous sample and effect size calculations, and formal rules for handling of data involving outliers, pre-speci?ed primary and secondary endpoints, and replication of key experimental ?ndings (Landis et al., 2012). While laudable, these recommendations have not been formally adopted by journals, and reading the metabolism literature suggests that the majority of laboratories do not strictly adhere to these ‘‘best practices."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Even if recommendations exist, they are rarely taken up by journals and labs","Surprisingly, however, although considerable effort has been devoted to recognition of the problems associated with cell line identi?cation and veri?cation (Freedman et al., 2015), there is scant evidence that scientists have routinely adopted these guidelines to ensure the ?delity and rigor of their own cell line research."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Even if recommendations exist, they are rarely taken up by journals and labs","Despite widespread endorsement of the guidelines by funding agencies and multiple journals, the impact to date of the guidelines on improvement of reporting of animal experiments has been modest (Baker et al., 2014)."
"Data management plans, the missing perspective",Note,"Even if recommendations exist, they are rarely taken up by journals and labs","Thirty-eight (58%) of the organizations from which we received DMP requirements either required or suggested a written DMP. This is 42% of the contacted funders and 37% of the identi?ed funders. Thus, outside the therapeutic development industry, written DMPs are not required a majority of research funders. Yet, data sharing plans or a justi?cation of why data sharing was not possible were required by 81% of the funders from which we received DMP requirements, highlighting the disparity of emphasis on pre versus post-publication management of data."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,"Even if recommendations exist, they are rarely taken up by journals and labs","Although most of the proposed solutions are uncontroversial in principle, their implementation is often challenging for the research community, and best practices are not necessarily followed."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,"Even if recommendations exist, they are rarely taken up by journals and labs","The main lesson learnt (from internal journal audit) that we may now share is that the guidelines that have been journal requirements since 2015 are not being routinely followed by authors and this is being missed during the peer review process. This ‘non-compliance’ is not unique to British Journal of Pharmacology (BJP) and is a phenomenon experienced by many other journals. Indeed, Nature recently reported that when guidelines are introduced, ‘author compliance can be an issue’ (Anonymous, 2017)"
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Even if recommendations exist, they are rarely taken up by journals and labs","However, as of 2019 fewer than ten journals have implemented this policy of formal data peer review as a mandatory requirement, and journal policies on data sharing and reproducibility tend to focus on transparent reporting, such as including links to data sources. This enables a motivated peer reviewer to assess aspects of a study, such as data and code, more deeply, but this is not routinely expected."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many think the system works well as it is and do not consider reproducibility as a significant issue,"However, an academic culture in which there is little chance of replication happening or being published reduces the perceived need to make research replicable through materials and data availability and transparent reporting because researchers might very reasonably ask themselves, “Is anyone really going to attempt to replicate this?” This no doubt partially accounts for a history of inadequate reporting practices (e.g., as noted by Derrick, 2016; Han, 2016; Larson-Hall & Plonsky, 2015; Plonsky & Derrick, 2016), poor transparency of materials (Marsden & Mackey, 2014; Marsden et al., 2016; Marsden, Thompson, & Plonsky, in press), and very scarce availability of data (Larson-Hall & Plonsky, 2015; Larson-Hall, 2017; Plonsky, Egbert, & LaFlair, 2015)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many think the system works well as it is and do not consider reproducibility as a significant issue,"This is, we stress, not to criticize these studies (and a good proportion of our own research certainly has aligned with this practice). Rather, we aim (a) to acknowledge that our synthesis is not a fully comprehensive re?ection of the amount and nature of replication effort in the ?eld and (b) to recognize the complexities that our arguments and recommendations about nomenclature entail."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Many think the system works well as it is and do not consider reproducibility as a significant issue,"The ?rst, and a necessary step in addressing this issue, is to accept that it is real and important, even if its prevalence is uncertain, and requires concerted responses by and on behalf of all elements of the bioscience research community. Despite its importance, we should recognize that there are many countervailing forces working against this. First, many scientists believe that the problem is exaggerated, either in its prevalence or impact. These people see the system as working well, and/or they remain unconvinced of the problems’ salience based on the literature or their personal experience."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Many think the system works well as it is and do not consider reproducibility as a significant issue,"So many things can go wrong in an experiment done by someone for the first time. Instead I think we should let the scientific process run its course. Findings that are not correct will disappear because others can’t reproduce them or publish divergent results, after an adequate try and hopefully also explaining why the results are different”. Ruoslahti has received support from Tim Errington, manager of the Centre for Open Science’s cancer reproducibility project, who agreed that a single failure to replicate should not invalidate a paper."
Towards reproducibility in recommender-systems research,Article,Some are hopeless about the issue of reproducibility and consider it to be unavoidable,"The recommender-system community, and in particular the research-paper recommender-system community, widely seems to accept that research results are dif?cult to reproduce, and that publications provide little guidance for recommender-system developers and researchers. When we submitted a paper to the ACM RecSys conference pointing out the inconsistent results of the above-mentioned evaluations, which are not contributing to identifying effective recommendation approaches, one reviewer commented: I think that it is widely agreed in the community that this [dif?culty to reproduce results] is just the way things are—if you want a recsys for a speci?c application, there is no better way than just test and optimize a number of alternatives."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",There is room for enhanced collaboration and communication between research repositories and between repositories and journals,"Additionally, this collaboration may be articulated via repositories of scientific results, broadly defined; there is a need for greater collaboration among repositories (there is no level playing field today, compounded by the lack of data sharing and just the beginning of work on FAIR) and between repositories and journals."
A manifesto for reproducible science,Review,The lack of opportunities for open scholarship has kept the research process largely closed,"Very little of the research process (for example, study protocols, analysis workflows, peer review) is accessible because, historically, there have been few opportunities to make it accessible even if one wanted to do so. This has motivated calls for open access, open data and open workflows (including analysis pipelines), but there are substantial barriers to meeting these ideals, including vested financial interests (particularly in scholarly publishing) and few incentives for researchers to pursue open practices."
Replicability and replication in the humanities,"Report, policy document or website",Current incentives are not conducive to reproducible science,partly due to unhealthy research systems with perverse publication incentives
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Current incentives are not conducive to reproducible science,"How did we arrive at this state of affairs? Although it is admittedly more dif?cult to become an elite ?ghter pilot, astronaut, or head of state, competition for faculty positions and resources in the best academic institutions is ?erce, and the most valuable currency continues to be a mixture of publications in ‘‘the best journals,’’ ideally coupled with already secured independent funding. To obtain these valuable prestigious publications, one must meet the standards and expectations of journal editors, who similarly prize research that is spectacular, highly novel, and ideally accompanied by wellde?ned reductionist mechanisms and immediate obvious translational relevance"
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Current incentives are not conducive to reproducible science,"Reporting of negative results (conditions and models tried where an expected result was not obtained) is likely to be extremely valuable to the research community, yet at present there is no uniformly accepted or promoted mechanism enabling or requiring reporting of negative results."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Current incentives are not conducive to reproducible science,"directly related to the conduct of science Scienti?c research typically begins with a scientist who desires to understand the workings of the world. Objectivity and honesty are essential tools and values throughout the process, as is the ability to generally trust the work of others. But scientists are human, and, of course, fallible. They live in a world ?lled with incentives that con?ict with behaviors required to ?nd the truth. When such incentives collide with a diverse community of scientists with different life situations and values, it should not be surprising that some will exhibit behaviors inconsistent with the dispassionate pursuit of the truth."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Current incentives are not conducive to reproducible science,"Science is a career as well as a calling. So concerns about academic appointments, grant funding, compensation, and fame are ever-present during the conduct of research and, as choices are made, about its publication and dissemination. The highest professional standards for the ethical conduct of research require that such exogenous incentives not affect the integrity of research conduct, analysis, and reporting, and I believe this is most often the case. But this line is too often crossed, perhaps more so today when researchers, having been drawn into research during a period of relatively plentiful funding, are experiencing increased dif?culty obtaining funding to support their research and compensation."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Current incentives are not conducive to reproducible science,"Unfortunately, these desirable characteristics do not always align with a scientists’ academic rank, fame, recognition, level of funding, number of publications, or in which journals they are published. There is no simple approach to achieving this, but awareness and articulation of its importance by scienti?c leaders, and the broader research community, are necessary ?rst steps"
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Current incentives are not conducive to reproducible science,"“The real challenge is with individual institutions and their policy and practices around academic promotion, tenure, and so on, which are still largely wedded to outdated measures such as Impact Factor and author position”."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Current incentives are not conducive to reproducible science,misaligned incentives for authors that might stimulate certain types of research.
Increasing the Impact of Medical Image Computing Using Community-Based Open-Access Hackathons: the NA-MIC and 3D Slicer Experience,"Report, policy document or website",Current incentives are not conducive to reproducible science,"Traditionally and understandably, reasons for the gap between theory and practice is ascribed to the academic reward system and the mechanisms for research funding where novelty is valued over robustness and reproducibility."
A manifesto for reproducible science,Review,Current incentives are not conducive to reproducible science,"For example, current incentive structures promote the publication of ‘clean’ narratives, which may require the incomplete reporting of study procedures or results."
A manifesto for reproducible science,Review,Current incentives are not conducive to reproducible science,"Publication is the currency of academic science and increases the likelihood of employment, funding, promotion and tenure. However, not all research is equally publishable. Positive, novel and clean results are more likely to be published than negative results, replications and results with loose ends; as a consequence, researchers are incentivized to produce the former, even at the cost of accuracy positives in the published literature CPS (N = 104) DP (N = 634) JEPLMC (N = 483) JPSP (N = 419) PSCI (N = 838) PSCI introduces badge for data sharing. These incentives ultimately increase the likelihood of false . Shifting the incentives therefore offers an opportunity to increase the credibility and reproducibility of published results."
A manifesto for reproducible science,Review,Current incentives are not conducive to reproducible science,"This showed that, for parameter values derived from the scientific literature, researchers acting to maximize their ‘fitness’ should spend most of their effort seeking novel results and conduct small studies that have a statistical power of only 10–40%. Critically, their model suggests that altering incentive structures, by considering more of a researcher’s output and giving less weight to strikingly novel findings when making appointment and promotion decisions, would encourage a change in researcher behaviour that would ultimately improve the scientific value of research."
How to Make More Published Research True,Article,Current incentives are not conducive to reproducible science,"This status quo can easily select for those who excel at gaming the system, producing prolifically mediocre and/or irreproducible research; controlling peer review at journals and study sections; enjoying sterile bureaucracy, lobbying, and maneuvering; and promoting those who think and act in the same way."
How to Make More Published Research True,Article,Current incentives are not conducive to reproducible science,"The current system does not reward replication—it often even penalizes people who want to rigorously replicate previous work, and it pushes investigators to claim that their work is highly novel and significant."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Current incentives are not conducive to reproducible science,"In most research specialties, great credit is given to the person who first claims a new discovery, with few accolades given to those who endeavour to replicate findings to assess their scientific validity."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Current incentives are not conducive to reproducible science,"Reward mechanisms (eg, prestigious publications, funding, and promotion) often focus on the statistical significance and newsworthiness of results rather than the quality of the design, conduct, analysis, documentation, and reproducibility of a study. Similarly, statistically significant results, prestigious authors or journals, and well connected research groups attract more citations than do studies without these factors, creating citation bias."
"Most computational hydrology is not reproducible, so is it really science?",Note,Current incentives are not conducive to reproducible science,"An oft-cited underlying reason for such failures is the present reward system in scienti?c publication, which prioritizes the publication of innovative, and seemingly statistically signi?cant results over the publication of both null results [Franco et al., 2014; Jennions and Møller, 2002; cf Freer et al., 2003], and reproduced experiments. Such a system provides few incentives to adopt open science practices that support and enable veri?cation [Nosek et al., 2015]"
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Current incentives are not conducive to reproducible science,"Although the replication crisis has been blamed, in part, on questionable research practices involving manipulation and selective reporting of data collected during group experimental research (e.g., Kerr, 1998; Simmons, Nelson, & Simonsohn, 2011), such practices likely are motivated and reinforced by publication bias, or the tendency of journals to exclusively publish studies that find statistically significant effects (Franco, Malhotra, & Simonovits, 2014; Lilienfeld, 2017;Rosenthal, 1979)."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Current incentives are not conducive to reproducible science,"Under circumstances in which these contingencies are less influential, publishing in certain niche outlets—high-status journals that are affiliated with an organization attached to one’s discipline—may nonetheless confer particular social prestige and recognition (e.g., Dixon, Reed, Smith, Belisle, & Jackson, 2015), attract desirable attention to one’s organization, or result in more tangible reinforcers (e.g., increased invitations for speaking engagements, awards and honors, paid consulting opportunities, promotion, tenure, and enhanced prospects for future employment). Such contingencies motivate and reinforce biases in ways that can contribute to erroneous findings."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Current incentives are not conducive to reproducible science,"The bias for positive results is further exacerbated by the external influence of a competitive research culture. Publications are the prime currency for advancing academic careers, and where editorial decisions are seen to favour positive results,"
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Current incentives are not conducive to reproducible science,"At a systemic level, career pressures to publish offer a sharp incentive to authors to favour writing up papers with the greatest chance of success, and under the current system of publication this will inevitably favour positive results."
What you see is what you get? Enhancing methodological transparency in management research,Review,Current incentives are not conducive to reproducible science,"In conclusion, the present study provides prospective authors with detailed information regarding what the gatekeepers say about research methods and analysis in the peer review process.” Transparencywas not mentioned once in the entire article. These results provide evidence that greater transparency is not necessarily rewarded and many ofthe issues described in our article may be “under the radar screen” in the review process. In short, the focus on publishing in “A-journals” as the arbiter of rewards is compounded by the lack of obvious benefits associated with methodological transparency and the lack of negative consequences for those who are not transparent, thus further reducing the motivation to provide full and honest methodological disclosure"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Current incentives are not conducive to reproducible science,"It has been argued that this development is due to a counterproductive academic reward structure, arising from the combination of top-tier journals’ preference for ‘statistically signi?cant results’ and a highly competitive tenure-track system in many universities that relies disproportionately on top-tier journal publications (Bedeian, Taylor, & Miller, 2010; Pashler & Wagenmakers, 2012)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Current incentives are not conducive to reproducible science,"This reward structure encourages practices inconsistent with statistical best practice (Wasserstein & Lazar, 2016), speci?cally ex post writing of hypotheses supposedly ex ante tested, also referred to as HARKing (Kerr, 1998), and of manipulating of empirical results to achieve threshold values, varyingly referred to as p-hacking (Head et al., 2015; Simmons et al., 2011) star wars (Brodeur et al., 2016), and searching for asterisks (Bettis, 2012)"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Current incentives are not conducive to reproducible science,"The heavy focus on signi?cant effects opens the door to a variety of questionable (and occasionally plain bad) practices, some of which we discuss below in greater detail. Most fundamentally, such approaches are inconsistent with Popper’s (1959) falsi?cation criterion, which is the philosophical foundation for conducting hypothesis tests in the ?rst place (van Witteloostuijn, 2016). As a consequence, the reliability and validity of cumulative work are not as high as they could be without biases in the publication process."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Current incentives are not conducive to reproducible science,"Of course, this also requires broader institutional change to remove some of the incentives that disproportionately reward scholars ?nding statistically signi?cant results (Ioannidis, 2012; van Witteloostuijn, 2016)."
On Replication in Communication Science,Editorial,Current incentives are not conducive to reproducible science,"Yet the structure of the publishing system in the field of Communication, as well as incentives related to employment, retention, and promotion, often discourages the replication of communication findings."
On Replication in Communication Science,Editorial,Current incentives are not conducive to reproducible science,"Unfortunately, the structure of the publishing system in the field of Communication (and beyond) discourages the replication of important communication findings. Scholars are
rewarded within their field and discipline by greater numbers of published findings."
Reproducibility and Research Integrity,Note,Current incentives are not conducive to reproducible science,"Pressures to produce results and publish may be important factors in science’s reproducibility problems (Horton 2015, Shamoo and Resnik 2015). Researchers who are trying to publish results to advance their careers or meet deadlines imposed by supervisors or sponsors may cut corners when designing and implementing experiments. For example, if a researcher has obtained a result that is marginally statistically significant, he or she may decide to go ahead and publish the result without replicating the result and carefully considering whether it is a false positive."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website",Current incentives are not conducive to reproducible science,"In education, as in other research fields, a wide range of factors (e.g., publication bias; reputation and career advancement norms; emphases on novel, potentially transformative lines of inquiry) may dis-incentivize reproducibility and replication studies"
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Current incentives are not conducive to reproducible science,The incentives for researchers are important. The structure of the scientific environment can lead to grant chasing and questionable research practices.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Current incentives are not conducive to reproducible science,"Second, there is a perceived deliberateness, or at least carelessness, in scientific production due to competitive pressures. A growing proportion of scientists are perceived as – willingly or unwittingly – bending some of the basic premises of the scientific method to produce ‘fast science’ or even ‘make believe science’ – facts and theories that are declared true but are dubious or even false. This rests more on the structure of incentives of science-making, embedded in culture and practice, than on deliberate attempts to ‘cheat’. The need for results to be reproducible, and the tangible steps needed to make them so, may help results be trustworthy and keep scientists honest."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Current incentives are not conducive to reproducible science,"There is wide agreement in the literature that a good part of these pitfalls, underlined above, are rooted in current practices governing the career advancement of researchers, the publication of scientific results, the allocation of grants and recognitions, and overall a culture of research that primes competition over collaboration. As it emerges from the literature, there are cultural and economic drivers for the lack of reproducibility"
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Measurement of reproducibility is currently missing from most metrics and equations used to evaluate scientists. Although all scientists are data driven, we currently have no accepted way to track and quantify our own track record of scienti?c reproducibility. This reproducibility knowledge gap extends to our funding agencies and journals, which collectively have little understanding of whether the science they regularly fund and publish, respectively, turns out to be reproducible."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Rather than only quantifying citations, papers in top journals, and research funding, why not quantify reproducibility? Imagine if each scientist was associated with a Rs (ReproducibilityScientist) index, re?ecting the number of times the key scienti?c ?ndings in a paper had been reproduced by at least one other independent research group. Of course, one would have to carefully debate and re?ne the meaning of ‘‘reproducible’’ (Goodman et al., 2016), but perhaps one could start simply by requiring that (a) the key ?ndings and (b) at least 50% of the experimental data, from a single paper, were independently reproduced by at least one other research group. So a senior scientist with a Rs index of 40 would have published 40 research papers with ?ndings found to be independently reproduced by others."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Place a greater emphasis on the reproducibility and importance of published research by faculty and a reduced emphasis on the number of publications and journal in which they are published In an age when quantitative metrics are widely employed, including measures of article citations [41] and impact factor of speci?cjournals [42], we should be reminded that such metrics can hide major defects in the quality and reproducibility of published research. Though academic organizations seek to go beyond these metrics in their judgment of scienti?c quality and impact, and often succeed in this endeavor, my experience as dean suggests that this approach not infrequently fails. As dif?cult as it might be to accomplish, there should be a major effort to identify better metrics of research reproducibility and durable scienti?c impact. If developed, such metrics could support academic appointments and promotions and the response to cases where poor reproducibility is an issue. It would also be important for faculty review processes to accord greater credit to well-conducted studies that con?rm or contradict published work. Although quanti?able metricsareagoal, it wouldbehardtohave a better approach than one based on a deeply informed assessment by objective experts in the ?eld."
Reboot undergraduate courses for reproducibility,"Report, policy document or website",Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Most undergraduate dissertations turn into exercises tallying the limitations of the research design — frustrating for both student and supervisor. However, each year a few students get lucky and publish, securing a huge CV advantage. I wondered what lesson this was teaching. Were we embedding a culture that rewards chance results over robust methods?"
Reproducibility literature analysis - a federal information professional perspective,Article,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"‘Publish or perish’ is a phrase coined to describe the pressure in academia to rapidly and continuously publish academic work to sustain or further one’s career. Frequent publication is one of the few methods at scholars’ disposal to demonstrate academic talent. The desire or need to publish work at a near-constant rate can lead to problems in reproducibility, and can lead to issues concerning selective reporting, also known as ‘cherry picking’ (Baker, 2016a). [Related terms: incentives; publication policy (solution); culture shift (solution)]"
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,the unquestioning and inept reliance on metrics in evaluation;
How we can make ecotoxicology more valuable to environmental protection,Note,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,coupled with the needs of a journal to ?ll issues and increase impact factors
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"In recent years, the academic environment has increasingly reinforced the quantity of research studies published by academics, creating a competitive environment during an era when resources were less available due to increasing costs of coupled with dwindling public funding ofhigher education (Lilienfeld, 2017;Mitchell, Leachman, and Masterson, 2016). The increasing emphasis on publication metrics, at both the journal and researcher level (e.g., journal impact factor, researcher h-index, citation count) likely functions as reinforcement for publishing in particular journals, publishing at a high rate, and for attaining a large number of citations for one’s publications (Lane, 2010)."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"The competition for faculty positions is therefore fierce. Once secured, retaining a faculty position can be dependent on meeting key performance targets, and the main indicators of academic success are number of publications, journal impact factors and number of citations."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"As has been discussed, in the current publication system, positive results are more easily published, especially those studies reporting large effects which, despite methodological limitations, are often published in highimpact journals."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Scientists themselves are often judged on the frequency and impact of their publications, and their ability to acquire funding, which are key criteria for hiring and promotion at many universities and research institutes."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Use of evaluative metrics focusing on quantity and impact rather than on quality and reproducibility,"Given the current fetishisation of publication in ‘top journals’, there is a need for broader action than to address the impact factors of the biggest journals, especially for young scholars."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Incentives from the media and from public outreach encourage simplistic and promising reports,"The same scientist will, at many institutions, receive regular email requests, periodic visits, and exhortations from public relations or media communications of?cers, anxious for new exciting stories to tell that highlight the wonderful work being done within the institution. There are monthly newsletters to ?ll, websites to update, fundraising pitches and portfolios to embellish, and local media contacts always need a new story. The media itself has an extraordinary appetite for scienti?c and medical information, especially stories with a hint of therapeutic relevance. The media beast is insatiable, although even my mother has now learned that most ‘‘medical breakthrough stories’’ featured on the television, radio, in print, or disseminated via the internet and social media are almost always exaggerated and often frankly incorrect."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Incentives from the media and from public outreach encourage simplistic and promising reports,"Even more than in academic publishing, publication bias affects which stories are featured in the media. The need for newsworthiness means replications and null results are almost never published in the popular press."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Incentives from the media and from public outreach encourage simplistic and promising reports,"It is the joint responsibility of researchers, journalists, science writers and press officers to ensure science is accurately reported in the media."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"The format of research proposals focuses on 'Expected findings', which may discourage the reporting of negative findings","Incentives produced by the major funders of bioscience research NIH and other funding agencies place great emphasis on, and often have requirement for, research proposals with well delineated hypotheses and “expected ?ndings”, some of which are expected to have been demonstrated at the time of grant submission. It has been persuasively argued that this approach to both conducting and funding science has major conceptual ?aws [36]. As relates to the issue of reproducibility, a key ?aw of this requirement is its’ placing the scientist in a position where data is ?ltered through the lens of the stated hypothesis, in a way that promotes expectation of a particular result, and biases against, or promotes rejection of, contradictory evidence. It is easy to see how this construct puts great pressure on a scientist to avoid falsifying the hypothesis upon which their grant was funded, even when evidence suggests this is the most rational approach."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,The publish or perish culture and need for high impact publication discourages reproducibility,"In some environments, accomplishment is judged on the basis of funding rather than publications, but funding is a means, not an end product. Researchers are tempted to promise and publish exaggerated results to continue their funding for what they think of as innovative work."
Replicability and replication in the humanities,"Report, policy document or website","There is resistance against exposing some issues of science, e.g. because of the potential negative impact on one's career","If too many results turn out not to be replicable, upon attempting to replicate them, that will gradually erode public trust in science."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,"There is resistance against exposing some issues of science, e.g. because of the potential negative impact on one's career","One implication of the crisis is that psychological phenomena purportedly based on evidence does not hold up when subjected to further scientific scrutiny (i.e., replication studies). More broadly, a lack of sound replication studies foments public distrust in science, scientists, and scientific evidence."
"Test, Model, and Method Validation: The Role of Experimental Stone Artifact Replication in Hypothesis-driven Archaeology",Scopus item - Unclassified,Criticism can introduce defensive responses,"One might have thought that a set of comments that critical, especially appearing as they did in a leading anthropological journal, would have led to changes in how flintkapping experiments were designed and carried out (such as those outlined in Clarke 1968; Dunnell 1971; Eren et al. 2014a; Lett 1997; Lycett and Chauhan 2010; O’Brien 2010; Surovell 2009), but with a few exceptions, archaeologists, especially those who were expert flintknappers, continued to use the craft to make authoritative, intuitive arguments about lithic technology."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Criticism can introduce defensive responses,"Another factor potentially indicating importance, and thus a need for replication, are “surprising” ?ndings (see Makel et al., 2012, p. 540; Porte, 2012, p. 7). Surprising could be, for example, large effect sizes when a meta-analysis would predict them to be smaller (or vice versa)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Criticism can introduce defensive responses,"This reviewer will often be predisposed to be negative, sometimes trying to save his or her own results by questioning the quality ofthe replication (e.g.making the case that an incompetent person will often fail to get the correct result; Bissell, 2013). Thus it may often be dif?cult to publish a replication, especially when that replication contradicts earlier work, and this is a strong disincentive to replicate."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Irreproducibility efforts raise a fear of loss of trust in science,"Third, some are concerned that public discussions of these issues would provide ammunition to forces opposed to science and science funding. While this is indeed a risk, inaction seems to be a greater risk to the reputation of the ?eld over the long term. Finally, some individuals and institutions fear diminished support from donors and organizations inclined to support research in response to public airing of these problems."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,The replicability crisis debate incites fields to compare one another rather than to work towards a solution,Maybe it is time to move beyond questioning each other’s data to work on a broader theoretical model that could accommodate both types of results?
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"There is a resistance from authors and journals, as ""irreproducible research"" carries a negative connotation and retractions are difficult in practice","Many scientists might scoff with indignation if someone questions reproducibility of their own research; after all, this ‘‘reproducibility issue’’ is usually someone else’s problem."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"There is a resistance from authors and journals, as ""irreproducible research"" carries a negative connotation and retractions are difficult in practice","In some instances where retractions appear to be called for because data are no longer considered valid, authors resist this, perhaps out of fear they will be misconstrued as being guilty of misconduct rather than simply being wrong. So it should be clear that retractions do not imply research misconduct. Perhaps different naming conventions should be employed to clarify speci?c causes."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"There is a resistance from authors and journals, as ""irreproducible research"" carries a negative connotation and retractions are difficult in practice","Another important question is whether journals create unnecessary barriers to publishing retractions, perhaps to limit unwelcome damage to their reputations."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"There is a resistance from authors and journals, as ""irreproducible research"" carries a negative connotation and retractions are difficult in practice","Clarify use of retractions Develop an agreed upon and consistent approach for journals to manage retractions. This would include clear criteria for retractions versus corrections, how they are linked to published papers, and how the reason for the retraction or correction is explained, whether due to misconduct or a wide array of more innocent explanations."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website","There is a resistance from authors and journals, as ""irreproducible research"" carries a negative connotation and retractions are difficult in practice","Some have told Nature privately that they are worried because of the way in which even unintended errors in papers are flagged publicly online. It can be easy to make a mistake when handling massive and complex biological data sets, they say — and they fear their papers might be publicly picked apart, derailing their careers before they get started."
Reproducible Research A Retrospective,"Report, policy document or website","Code shared alongside papers needs to be regularly maintained, but this is not currently funded","Code that was once highly readable can become unreadable as newer languages come to the fore and practitioners of older languages decrease in number. Maintenance of data and code is not a question of paying for computer hardware or services. Rather, it is about paying for people to periodically update and ?x problems that may be introduced by the constantly changing computing environment. Unfortunately, funding models for scienti?c research are aligned with the mechanism of paper publication, where one can de?nitively mark the end of a project (and also the end of the funding). However, with data and code, there is often no speci?c end point because other investigators may re-use the data or code for years into the future. Termbased project funding, which is the structure of almost all research funding, is simply not designed to provide support for maintaining materials on an uncertain timeline."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Journals in particular have a responsibility and could help by changing outdated methods of reward and insisting on more detailed method descriptions, according to Matt Hodgkinson, Head of Research Integrity at Hindawi, one of the largest open-access journal publishers. “Incentive structures now reward publication volume and being first”, he said. “Citations are currently counted and more is considered better for the authors and journal, which can perversely reward controversial findings that fail to replicate. Instead funders and institutions should reward quality of reporting, replicability, and collaborations”."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"It is difficult to get funding to conduct replication studies, and it can also be challenging to publish replication studies. As a result, academics interested in promotion or tenure may understandably believe that conducting replication studies is a risky endeavor."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Similar to our previous evaluation, approximately halfof eligible articles clearly claimed to present novel discoveries. Despite claiming novelty on various aspects, it is improbable that the vast majority of articles have truly innovative findings. These results likely reflect the culture ofadoring novelty and significance. Investigators are incentivized to say that they do something different and innovative, even when the differences with prior research are subtle or unimportant."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Related to that, publication bias occurs when papers are more likely to be published if they report significant results (Bishop & Thompson, 2016)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"So far, academic reward mechanisms often focus on statistical significance and newsworthiness of results rather than on reproducibility (Ioannidis et al., 2014). Also journalists and media consumers and, therefore, all ofus ask for the novel, unexpected and surprising. Thus the average truth often does not make it to the paper and the public, and much of our attention is attracted by exaggerated results."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"we argue that contingencies of publication bias that led to the “replication crisis” also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"A majority ofresearchers reported that they were more likely to recommend publication of a submitted manuscript when a dataset showed positive effects, and a minority of researchers indicated a willingness to drop datasets showing weaker experimental effect when submitting a manuscript for publication."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Perhaps because of the differences in interpretation, reviewers have been shown to be highly influenced by the direction and strength of effects. On average, null papers take several months longer from the time of submission to eventual publication than positive papers (median, 1.1 vs 0.8 years; P = .04), suggesting that null results receive more criticism during the peer-review process. This delay may stem from the increased difficulties of trying to persuade reviewers of the merits of null findings."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Peer reviewed publication remains the primary vehicle for research dissemination. There are a number of weaknesses in the dissemination-bypublication model which may serve to undermine replicability and verifiability. For example, publication bias encourages selective reporting, and in particular the reporting of novel and surprising findings, over the publication of replications, protocol optimisations, and null results."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Journal dependence on novelty and citations for reputation and recognition further strenghtens publication biases,"Funders typically reward novelty, to the exclusion of replication. Journals preferentially publish findings that are statistically significant, novel or surprising. These biases are amplified in journals with higher impact factors which are associated with greater scientific prestige. Null findings may be considered to be difficult to interpret, studies, either supporting or refuting previous findings, has been documented to disincentivise the publication of null findings and efforts at replication, leading to a body of scientific research that is characterised by surprising, novel and significant findings."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Journals often encourage authors to simplify their work to look translational, leading to oversimplifications","Corresponding authors will not infrequently receive a small editorial nudge if they have not suf?ciently framed the biomedical translational importance of their basic science with suf?cient positivity, panache, and verve"
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Poor documentation due to limitations on word/page counts,"Another barrier to replication is that it is dif?cult to include both a replication and an extension study within one published article given normal space limitations, yet this study structure may alleviate the stigma attached to doing replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Poor documentation due to limitations on word/page counts,Recommendation: Encourage publishers to lift word limits or provide online capacity to encourage more replication work within individual study reports.
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Poor documentation due to limitations on word/page counts,"Indeed, the trend in many journals is toward minimizing the length of methods sections. The regular use of simple quality control techniques to verify cell line identity and potential contamination would greatly enhance the validity of cell line data, yet many institutions, granting agencies, and journals do not regularly insist on obligatory detailed cell line reporting (Freedman et al., 2015). With the emerging use of stem cell-derived cells for the study of islet biology, it seems likely that very precise detailed disclosure of the exact composition of cell culture media and all essential exogenous additives and growth factors, coupled with speci?cation of the gender and age of origin as well as extensive molecular characterization and footprinting criteria, will be needed to ensure replication of stem cell-derived cells within and across laboratories."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Poor documentation due to limitations on word/page counts,"Although some journals cite space limitations, precluding provision of more extensive information, the widespread use of online supplemental information should facilitate, not hinder, precise research communication."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Poor documentation due to limitations on word/page counts,"Hodgkinson added that journals should also abandon space constraints on the methods sections to allow authors to describe the experimental procedures and conditions in much more detail. In fact, a growing number of journals and publishers encourage authors to provide more details on experiments and to format their methods section so as to make it easier for other researchers to reproduce their results."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Poor documentation due to limitations on word/page counts,Where limitations of word count preclude reporting all items in the body of a study
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Poor documentation due to limitations on word/page counts,"In the last 10 years, more journals, and types of journal article, have emerged that publish articles that describe speci?c parts of a research project. The print-biased format of traditional research articles does not always provide suf?cient space to communicate all aspects of a research project. These new publications include journals that specialise in publishing articles that describe datasets or software (code), methods or protocols. Established journals have also introduced new article types that describe data, software, methods or protocols."
What you see is what you get? Enhancing methodological transparency in management research,Review,Poor documentation due to limitations on word/page counts,"To some extent, the implementation of many of our recommendations is now possible because of the availability of online supplemental files, which removes the important page limitation constraint. For example, because ofpage limitations, the Journal of Applied Psychology (JAP) had a smaller font for the Method section (same smaller size as footnotes) from 1954 to 2007 (Cortina et al., 2017a). In addition, the page limitation constraint may have motivated editors and reviewers to ask authors to omit material from their manuscript, resulting in low transparency for consumers of the research."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Poor documentation due to limitations on word/page counts,"For example, editors and reviewers may have asked that authors remove information on some aspects of the study to comply with a journal's word or page limit. Also, information directly relevant to some of the 12 transparency criteria may have been included in the authors' responses to the review team, but not in the actual manuscript. So, the overall low level of transparency uncovered by our study is likely the result of a complex review process involving not only authors but also a journal's review team."
Reproducibility and Research Integrity,Note,Poor documentation due to limitations on word/page counts,"Information may be disclosed in the materials and methods section of the paper or in appendices. Because many journals have space constraints that limit the length of articles published in print, some disclosures may need to occur online in supporting documents. Many journals require authors to make supporting data available on public websites (Shamoo and Resnik 2015)"
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Poor documentation due to limitations on word/page counts,"Replication and verification of scientific findings is hindered by abridged or partial reporting. These are caused in part by factors such as word limits imposed by journals, although the availability of online supplementary information mitigates this in some cases. However, the scientists themselves bear the ultimate responsibility for complete, reproducible and verifiable reporting."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Peer-review secrecy, which is thought to help honest review, may cause reviews of lower quality","The major argument against reviewer identi?cation is a concern that critical but honest reviewers could be subjected to consequential retribution by unhappy but in?uential authors. This is a real concern. But to the extent it is true, this brings dishonor to the scienti?c community and would need to be actively resisted, rather than be deployed as an argument for"
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Predatory journals may contribute to irreproducible science in cases where peer review is not scrupulous or missing,"Some journals, referred to as “predatory journals”, are a small subset of open access journals that charge authors publication fees and claim to have rigorous peer review, but publish papers with minimal or shoddy peer review."
Reproducibility and Research Integrity,Note,Predatory journals may contribute to irreproducible science in cases where peer review is not scrupulous or missing,"The growing number of for-profit, open access scientific journals which charge high publication fees, i.e. “predatory journals,” may also exacerbate reproducibility problems (Clark and Smith 2015). These journals often promise rapid publication and have negligible peer review. While it is not known how many articles published in these journals report
irreproducible results, the poor peer review standards found in these journals present a significant threat to the quality and integrity of published research (Beall 2016)."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Some consider peer review to be a flawed or incomplete process,"Finally, another general problem might lie in today’s research practice in applied machine learning in general. Several “troubling trends” are discussed in, including the thinness of reviewer pools."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Some consider peer review to be a flawed or incomplete process,"Journal policies often encourage reviewers to consider authors’ compliance with data sharing policies, but formal peer review of data tends to occur only in a small number of specialist journals, such as data journals (see later in this chapter) and journals with the strictest data sharing policies."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Some consider peer review to be a flawed or incomplete process,"Perhaps because of the differences in interpretation, reviewers have been shown to be highly influenced by the direction and strength of effects. On average, null papers take several months longer from the time of submission to eventual publication than positive papers (median, 1.1 vs 0.8 years; P = .04), suggesting that null results receive more criticism during the peer-review process. This delay may stem from the increased difficulties of trying to persuade reviewers of the merits of null findings."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,The impact on non-peer-reviewer research (e.g. preprints) on reproducibility is unknown,"The ?rst is the practice of placing manuscripts on “pre-print servers” for public comment prior to peer review and publication in traditional journals. There are potential advantages and some risks of this practice, which has existed for quite a while in physics and mathematics. What impact its growing use might have on reproducibility of published research is currently unknown."
The challenges of replication,Editorial,Broad replication studies may lack the expertise that informed the original studies,"A potential strength of the approach is that the experiments are performed by disinterested third parties with no vested interest in whether the experiments reproduce or not. However, this is also a potential disadvantage because the contract research laboratories performing the replications may not have the same level of expertise or motivation as the original laboratories."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","The extent to which reproducible ?ndings are deemed to be a desirable ambition can vary according to different ontological, epistemological, and methodological perspectives (Markee, 2017; Polio, 2012a; Porte, 2012; Porte & Richards, 2012)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Thus, the number of citations may be a warrant for replication."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","However, citation counts alone are unlikely to offer reliable or suf?cient motivation for replication. Importance also stems from the research community’s views on what research needs to be replicated to inform theory, method, or practice."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","A ?nal possibility is that researchers themselves provide theoretical and methodological justi?cations in the rationales sections of their replication studies, and these arguments are evaluated via current peer-review mechanisms. All these approaches may help to establish which research merits replication, but data are needed to ascertain the extent to which they are effective mechanisms for improving the amount, quality, or perceived prestige of replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Also, using the surprising-?ndings rationale alone as a warrant for replication could introduce a type of reverse publication bias, whereby ?nding no effect in a replication (where an effect or statistical signi?cance was found in the initial study) is considered the more publishable and citable outcome (Ioannidis, 2005; Luijendijk & Koolman, 2012)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Finally, the statistical signi?cance of a study’s results may have (undue) in?uence on its perceived importance for replication."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Yet it is of course useful to carry out replications of studies with null or borderline ?ndings. For instance, for the three null studies replicated by the Open Science Collaboration, the replications con?rmed two as null but produced statistically signi?cant ?ndings for the other one (see also Morgan-Short et al., 2018)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","The need to replicate studies with null ?ndings is particularly important in L2 research, where sample sizes are often too underpowered to reject the null hypothesis with an average post hoc power of.57 (Plonsky, 2013), the statistical equivalent of “tossing a coin in the air and hoping for heads” (Plonsky, 2015, p. 29). In sum, the absence of statistical signi?cance in an initial study may: (a) not validly indicate the absence of an effect but rather be an artefact of other issues, such as small sample size or chance ?ndings; (b) be a theoretically or practically useful ?nding that does merit corroboration via replication; and (c) lead to dichotomous rather than nuanced interpretations. Thus, statistical signi?cance alone serves as a dubious warrant for replication."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Beyond the signi?cance of an initial study, a warrant for replication must also consider research design."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"It is not clear how studies that need replication can be selected (citations, methods, impact on the field, etc.)","Recommendation: Provide warrants for replication studies and have them peer reviewed on a case-by-case basis with rationales including, but not restricted to, one or more of the following characteristics of the initial study: surprising ?ndings; one or more troubling methodological features; and/or high (potential) impact, such as theoretical or practical signi?cance."
MAKING REPLICATION MAINSTREAM,Article in Press,Not all researchers have the expertise to conduct replication studies,"A controversial issue surrounding the de?nition of direct and conceptual replications concerns who actually plans and conducts the replication research; some distinguish replications conducted by the original authors from those conducted by an independent group (e.g., Hüffmeier et al. 2016). The rationale for these distinctions often rests on concerns about expertise or unidenti?ed moderators that may vary across research laboratories. The basic idea is that some people have theexpertise to carry out the replication (usually the original authors), whereas others do not have such skills (usually researchers who fail to replicate an effect). Likewise, original authors are often working in settings similar to those of the original study so manypotentialmoderators are held constant."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,"To understand the state of replication research in a particular ?eld, one must ?rst determine the quantity of replication research that has been undertaken. To do this, those studies which should be counted as replication research must be identi?ed. This is not a trivial matter. Given a broad de?nition (e.g., studies investigating related questions using similar designs and materials), a very large number of studies could be called replications (see Plonsky, 2012, for discussion of the extent to which studies included in a meta-analysis could be considered replications, and VanPatten, 2002a and 2002b, for narrower conceptualizations). On the other hand, even studies that fall into a narrower de?nition of replication (e.g., investigating the same research questions with a design and materials that are as similar as possible to an earlier study) may not label themselves as replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,"The low rate of replication is likely due in part to a lack of willingness to self-label as replication (Neulip & Crandall, 1993; Polio, 2012b). This reticence is complex. Anecdotally, we observed during colloquia discussing this study and the research by Morgan-Short et al. (2018) that some researchers reported actively undertaking and promoting replication with students and in their own work, yet they were less enthusiastic about labeling these studies as replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,"First, it hinders the general tracking of intellectual connections and hides theoretical and methodological precedents under an invisibility cloak or a “cloaking device” (Makel et al., 2012, p. 541)"
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,"Second, without replication labels, heterogeneity from one study to the next can pass largely unchecked. We found that despite many suggestions in the limitations/further research sections of articles regarding necessary replications with different language combinations, participant demographics, or design features, very little such speci?c variation is undertaken systematically in self-labeled replications, and variation was often accompanied by other, potentially confounding, changes"
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,The lack of self-labeling adversely affects efforts to synthesize and meta-analyze research.
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Many replication studies are not labeled clearly,Recommendation: Use more self-labeling with the term replication wherever appropriate.
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Many replication studies are not labeled clearly,"Perhaps more worryingly for a synthetic ethic to research is that many studies do not self-label as replications, making it dif?cult to ascertain methodological and analytic similarities between studies that address similar questions."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Replications are not always possible (e.g. exploratory research),"There is a high degree of consensus that replication, particularly when narrowly de?ned as direct or close replication, is not appropriate or useful for all types and stages of research (e.g., ideological or interpretative approaches, exploratory or grounded research, or case studies)."
Making sense of replications,Article,Replications are not always possible (e.g. exploratory research),"There is no such thing as exact replication because there are always differences between the original study and the replication. These differences could be obvious (like the date, the location of the experiment, or the experimenters) or they could be more subtle (like small differences in reagents or the execution of experimental protocols). As a consequence, repeating the methodology does not mean an exact replication, but rather the repetition of what is presumed to matter for obtaining the original result."
The challenges of replication,Editorial,Replications are not always possible (e.g. exploratory research),"If a replication reproduces some of the key experiments in the original study, and sees effects that are similar to those seen in the original in other experiments, we need to conclude that it has substantially reproduced the original study."
Making sense of replications,Article,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","There is no straightforward answer to the question ""what counts as a successful replication of an original result?"" (Open Science Collaboration, 2015; Valentine et al., 2011)."
Making sense of replications,Article,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","However, asking the following questions will provide some insight: Does the replication produce a statistically significant effect in the same direction as the original? Is the effect size in the replication similar to the effect size in the original? Does the original effect size fall within the confidence or prediction interval of the replication (and vice versa)? Does a meta-analytic combination of results from the original experiment and the replication yield a statistically significant effect? And do the results of the original experiment and the replication appear to be consistent?"
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","Problems arise when new research or a new analysis of prior research is, or at least seems to be, inconsistent with previously published ?ndings. This may have many explanations, related to differences in design or execution of either the initial or the subsequent work. One often cannot conclude whether one of the studies is at fault, or whether they are just different, and the answer often requires additional experimentation."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","It is possible that the experiment would have been successfully replicated had its precise conditions been employed. On the other hand, a failure to replicate under slightly different conditions may suggest that the initial ?nding is at least less generalizable than initially claimed."
MAKING REPLICATION MAINSTREAM,Article in Press,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","Nevertheless, the post hoc reliance on context sensitivity as an explanation for all failed replication attempts is problematic for science. A tacit assumption behind the contextual sensitivity argument is that the original study is a ?awless, expertly performed piece of research that reported a true effect. The onus is then on the replicator to create an exact copy of the original context to produce the same exact result (i.e., the replicator must conduct an exact replication). The fact that contextual factors inevitably vary from study to study means that post hoc, context-based explanations are always possible to generate, regardless of the theory being tested, the quality of the original study, or the expertise of and effort made by researchers to conduct a high-?delity replication of an original effect."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","Traditionally, the success versus failure ofa replication is defined in terms ofwhether an effect in the same direction as in the original study has reached statistical significance again (Miller, 2009; Open Science Collaboration, 2015; Simonsohn, 2015; Fabrigar & Wegener)"
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","There is little doubt that behavior scientists and behavior analysts as a research community recognize the intrinsic importance of replication in our science. A pillar of evidence-based practice in ABA is the concept that the best available research evidence should always inform clinical decisions (Slocum et al., 2014). However, there is little consensus about how much replication is necessary to establish an intervention as having sufficient empirical support for broad application. Likewise, there is little consensus about the type of replication activities needed to demonstrate satisfactory or compelling empirical support."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,"There is no definitive answer on what is a successful vs unsuccessful replication, and non-replication does not mean that the original research is wrong","Despite recent attempts to quantify the number and kind of successful replication and nonreplication studies required to reach a conclusion about a particular intervention, there is currently little consensus on the minimum threshold of credible evidence beyond the notion that more evidence is better (e.g., Council for Exceptional Children, 2014; Chambless & Hollon, 1998; Kratochwill et al., 2013; Lonigan, Elbert, & Johnson, 1998)."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Unsuccessful replications also suffer from 'file drawer problem',"Although high impact papers whose results are rapidly retracted garner major attention, we should perhaps be more concerned about another situation that appears to be more common: papers of potential importance where many in the community have dif?culty building on or reproducing the results, but despite much informal discussion of these failures, whether at meetings or at the water cooler, papers documenting this fact do not get published."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Unsuccessful replications also suffer from 'file drawer problem',"In particular, their findings suggested that researchers were less likely to submit PRT studies for publication when effects were not robust, that reviewers and journal editors were less likely to accept studies for publication given lack of robust effects, or of both."
On Replication in Communication Science,Editorial,Unsuccessful replications also suffer from 'file drawer problem',"The upshot is that many studies that attempt but fail to replicate findings are not published. This is known as the “file drawer problem”, because they sit in the researchers’ file drawers rather than being part ofthe public research record (Rosenthal, 1979). The result is that if an early finding showed a statistical relationship because of sampling error (Type I error), and that finding is popular, few if any of the follow-up studies showing the true null result will be published."
On Replication in Communication Science,Editorial,Unsuccessful replications also suffer from 'file drawer problem',"One concern is that if the replication results are nonsignificant, it may be difficult to publish the results (Levine, Asada, & Carpenter, 2009)."
MAKING REPLICATION MAINSTREAM,Article in Press,Replication studies have limited added value,"Effect size estimates can be in?ated by sampling error alone. Thus, at a fairly abstract level, there are good reasons why replication is necessary in science. Nevertheless, there are ongoing debates about nearly all aspects of replications, from terminology to purpose to their inherent value."
Making sense of replications,Article,"Even in replication studies there may be errors or misinterpretation of statistical findings - if so, failure to replicate is not meaningful","Errors can also be caused by the improper execution of an experimental technique or by problems with samples and materials (such as the contamination of cell lines; Peterson, 2008). Discrepancies due to error are less interesting than those due to previously unidentified differences in methodology: unfortunately, results rarely provide clear evidence for whether it is one or the other."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Even in replication studies there may be errors or misinterpretation of statistical findings - if so, failure to replicate is not meaningful","The reports of such replication projects are often misinterpreted as showing that the original “significant” studies were mostly or entirely false positives. To see the error in such interpretations, consider that theOpen Science Collaboration (2015) observed 97-35 = 62 replications with p > 0.05 for which the original study had p ? 0.05, which is 64% of the 97 replications. This emphatically does not mean that 64% of the 97 original null hypotheses with “significant” P-values were correct, or that 62 of the 97 “significant” results were false positives. Why not? If as suggested (and indicated by the replications) theoriginal reported effect sizes were largely inflated due to selective reporting based on p ? 0.05, then the actual effect sizes (both in the original and replication studies) could easily be too small to provide high power for replication (Camerer et al. 2018)."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Even in replication studies there may be errors or misinterpretation of statistical findings - if so, failure to replicate is not meaningful","Thus, with selective reporting in the original studies, it may be unsurprising to get “nonsignificant” results in about twothirds of replications. And even with no selective reporting and only random variation present, replication studies remain subject to what may be severe false-negative errors. Consequently, “non-significant” replications (those with p > 0.05) do not reliably flag original studies as being false positives."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Even in replication studies there may be errors or misinterpretation of statistical findings - if so, failure to replicate is not meaningful","This means that, unless statistical power of the replication is nearly 100%, interpretations of replication attempts must allow for false-negative errors as well as false-positive errors, and that
“significance” and “non-significance” cannot be used to reliably judge success or failure of replication (Goodman 1992;Senn 2001, 2002)."
MAKING REPLICATION MAINSTREAM,Article in Press,Excessive replication can be wasteful too,"Do we really need attempts to replicate these studies to demonstrate that they lack value? We could do with much less of this research.” Moreover, just as original studies can be unreliable, so can replications, which means that one can be skeptical about the value ofany individual replication study (Smaldino & McElreath 2016)."
How to Make More Published Research True,Article,Excessive replication can be wasteful too,We should also consider the possibility that interventions aimed at improving scientific efficiency may cause collateral damage or themselves wastefully consume resources.
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Excessive replication can be wasteful too,"Studies are often designed without proper consideration of the value or usefulness of the information that they will produce. Although replication of previous research is a core principle of science, at some point, duplicative investigations contribute little additional value."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Excessive replication can be wasteful too,Most research designs do not take account of similar studies being done at the same time. The total sample size of clinical trials that are in progress might exceed the total sample size of all completed trials. The launch of yet another trial might be unnecessary. The need for replication of biomedical research should be balanced with the avoidance of mere repetition.
Reproducible Research A Retrospective,"Report, policy document or website",Excessive replication can be wasteful too,"In hindsight, another lesson learned from the HEI re-analysis is that the importance of reproducibility of a given study can fade with time. Over 25 years later, there have been scores of follow-up studies and replications that have largely come to similar conclusions as the Six Cities and ACS studies. Although both studies remain seminal in the ?eld of air pollution epidemiology, they could be deleted from the literature at this point and have little impact on our understanding of the science. This is not to say that the data and ongoing analyses do not have value, but rather the original results have been subsumed by later studies. Reproducibility was only critical when the studies were ?rst published because of the paucity of large studies at the time."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Replications alone won't get rid of false results (they show the problem but do not solve it),"There have been various projects to reproduce results, but these merely helped to define the scale of the problem rather than provide solutions, according to Mogil."
A manifesto for reproducible science,Review,Replications alone won't get rid of false results (they show the problem but do not solve it),"Proposed solutions may also give rise to other challenges; for example, while replication is a hallmark for reinforcing trust in scientific results, there is uncertainty about which studies deserve to be replicated and what would be the most efficient replication strategies. Moreover, a recent simulation suggests that replication alone may not suffice to rid us of false results."
Reproducible Research A Retrospective,"Report, policy document or website",Replications alone won't get rid of false results (they show the problem but do not solve it),"What do we ultimately learn from merely reproducing the results of an analysis? For example, it may be possible to execute code on a dataset without ever looking at the code or the data. In that case, the original goal of reproducibility—to learn about the details of an investigation—has been thwarted. We have simply learned that the code produces what the authors claim the code produces. In general, executing a process and seeing that process produce the results exactly as they were expected, produces very little new information."
Making sense of replications,Article,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","But, on its own, reproducibility does not guarantee validity. For example, the methodology that was repeated could have confounds, or the theoretical explanation for why the finding occurred could simply be wrong. By providing convergent evidence across methodologies, conceptual replication can foster confidence in the explanation for a finding but such evidence, on its own, does not guarantee the reproducibility of any individual piece of evidence. For example, individual findings could have occurred by chance. Together, direct and conceptual replication provides confidence in the reproducibility of a finding and the explanation for the finding."
Making sense of replications,Article,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","Scientific claims gain credibility by accumulating evidence frommultiple experiments, and a single study cannot provide conclusive evidence for or against a claim. Equally, a single replication cannot make a definitive statement about the original finding. However, the new evidence provided by a replication can increase or decrease confidence in the reproducibility of the original finding."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","Ruoslahti has been hotly disputing the results of that replication, arguing that it was a limited study comprising just a single experiment and that the associated metaanalysis ignored previous reproduction of his results by three generations of post docs. “I do disagree with the idea of reproducibility studies”, he said. “If only one experiment is done without any troubleshooting, the result is a tossup. Anything more extensive would be prohibitively costly. So many things can go wrong in an experiment done by someone for the first time. Instead I think we should let the scientific process run its course. Findings that are not correct will disappear because others can’t reproduce them or publish divergent results, after an adequate try and hopefully also explaining why the results are different”. Ruoslahti has received support from Tim Errington, manager of the Centre for Open Science’s cancer reproducibility project, who agreed that a single failure to replicate should not invalidate a paper"
The challenges of replication,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","It is also important to note that even if all the original studies were reproducible, not all of them would be found to be reproducible, just based on chance."
The challenges of replication,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","In particular, the experiments reported in the Replication Studies provide one indication of how readily reproducible previously published results are, but they cannot be considered conclusive evidence of the reproducibility, or lack of reproducibility, of any one study. For that, it will be necessary for the scientific community to aggregate results from multiple attempts by multiple groups"""
The challenges of replication,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","While we wait for this, it is important not to overinterpret the results. Already it is clear that nuanced interpretations are necessary, not black and white conclusions about which studies reproduced and which did not. It is also clear that this approach to testing reproducibility remains an experiment, with advantages and disadvantages, including the fact that it sometimes yields results that cannot be interpreted."
MAKING REPLICATION MAINSTREAM,Article in Press,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","The theoretical value of direct replications is limited Several arguments against replication converge on a general claim that direct replications are unnecessary because they either have limited informational value (at best) or are misleading (at worse). Crandall and Sherman (2016, p. 95) argue that direct replications only help to “uphold or upend speci?c ?ndings” which, in their view, makes direct replications uninformative and uninteresting from a theoretical perspective.For instance,dif?culties reproducing a speci?ceffect can only suggest a problem with a speci?c method used to test a theoretical idea. Likewise, a successful direct replication has little implication for theory because “[a] ?nding may be eminently replicable and yet constitute a poor test ofa theory” (Stroebe & Strack 2014). If the dependent measures of an original study are poorly chosen, a ?nding might replicate consistently, yet its replicability is problematic because it reinforces the wrong interpretation (Rotello et al. 2015). The concern is that the direct replications provided a false sense of certainty about the robustness of the underlying idea."
Replication in strategic management,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","That is, a replication that does not support the original results neither nullifies nor falsifies the original results. Instead, it alters the balance of the evidence. To support strong conclusions about the original results, multiple replications may be needed."
On Replication in Communication Science,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","We do not have to throw away a theory after one disconfirmation, especially if we have built up a reservoir of evidence for that theory (Lakatos, 1978), but differences in findings across samples can force the field to examine and learn from these discrepancies"
On Replication in Communication Science,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge","Therefore, many messages must be tested across many replication studies to determine the effect of a proposed message variable. While the articles presented within this special issue generally represent direct replications (see Simons, 2014), we also call for greater respect for studies that serve as conceptual replications by altering message characteristics or other features to probe effects. Indeed, we echo Hunter’s (2001) call for replications of conceptual replications."
On Replication in Communication Science,Editorial,"Successful replications do not ensure validity of results, and multiple replications are needed to infer knowledge",Many replications are required before researchers should confidently make recommendations about message strategies.
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Even if a replication is warranted, other design characteristics of studies may affect the feasibility of carrying out a replication. Practicalities of time and resources may impede the replication of certain studies, meaning that studies termed cheap and easy by Laws (2016) are replicated while replication in some subdomains is “likely to remain castles in the air” (p. 3). One likely manifestation of these practical constraints was the Many Labs Replication Project (Klein et al., 2014), which delivered a single 15-minute questionnaire (combining 13 earlier experiments) to 6,344 participants across 12 countries via 36 research groups. In L2 research, designs that are usually more costly involve longitudinal designs (e.g., experiments with pre-, post-, and delayed posttests as opposed to one-shot or cross-sectional designs), one-to-one measures (e.g., oral production tests versus group-delivered pen-and-paper or computer-based tests), equipment that is expensive to purchase or utilize (e.g., eye-tracking or neuroimaging hardware), and participant populations that are dif?cult to reach (e.g., rarer language combinations, schools, heritage speakers, or participants linked to a speci?c history or culture). Replications with such designs may be underrepresented compared to more easily administered designs."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Certainly, the academic and ?nancial incentives to routinely replicate the work of others do not exist. Rather than being replicated, published results are typically built upon and extended by subsequent research, perhaps carried out under somewhat different conditions from the original work, or with different reagents, but with results that are seen as consistent with and supportive of the earlier claims."
The challenges of replication,Editorial,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"The original plan was to conduct 50 replications but some had to be dropped for budget reasons, and a small number of Registered Reports did not make it through peer review as reviewers decided that it would not be possible to draw meaningful conclusions from the proposed experiments. The first Registered Reports were published in December 2014 and a total of 29 have been published to date."
On the Reproducibility of Psychological Science,Article,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Relying on the volunteer e?orts of many teams around the world, which meant that coordinating schedules was di?cult to begin with. Nevertheless, the implementations generally took longer than expected."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"It is difficult to get funding to conduct replication studies, and it can also be challenging to publish replication studies. As a result, academics interested in promotion or tenure may understandably believe that conducting replication studies is a risky endeavor"
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"It is worth noting that, although the cost of conducting a new fMRI experiment is a factor limiting the feasibility of replication studies, there are many findings that can be replicated using publicly available data."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Careful comparisons to resolve these questions are conducted and published less often than might be optimal. This is because ofthe resources they require, resources that could otherwise be channelled into evaluating biological hypotheses that are more likely to secure funding."
How to Make More Published Research True,Article,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"For some clinical research, replication is difficult, especially for very large, long-term, expensive studies."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Initiatives, like registered multi-lab replication studies should also be prioritized when the validity of important proposals is at stake. Funders are currently often reluctant to fund such studies. However, they should realize that the continuous seeking of new results and theories may just waste most of their resources (Ioannidis et al., 2014; Kaplan and Irvin, 2015)."
On Replication in Communication Science,Editorial,Conducting replication studies costs money but is not likely to bring grants or contribute to career progression,"Thus, although replication is an important component of communication research, it can be costly for scholars to invest time"
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Replication studies can even be more costly than initial research,There is clearly a downside to focusing on higher precision and direct replications. Perhaps the biggest one is that carrying out experiments towards the aim of increasing precision would take much longer.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Replication studies can even be more costly than initial research,"Funding agencies and journal editors focus on novelty. This is particularly hard to justify on the part of funders since failing to invest in replication means failing to seek robust answers to questions they already have made a commitment to answering. If the answer truly was worth paying for, then the replication should also be worth paying for (Nakagawa & Parker, 2015). Promoting the funding of replication studies would be relatively straightforward. Most obviously, agencies could set aside funds for important and well-justi?ed replications. Agencies could also incentivize replication by preferentially funding novel studies when those studies rest on well-replicated foundations (Parker, 2013). They could also preferentially fund researchers whose prior work has often been successfully replicated."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Replication studies can even be more costly than initial research,"For this reason, replication studies need even greater statistical power than the original, Macleod argued, given that the reason for doing them is to confirm or refute previous results. They need to have “higher n’s” than the original studies, otherwise the replication study is no more likely to be correct than the original"
Towards reproducibility in recommender-systems research,Article,Different implementation or use of codes,"Different implementations might explain some of the discrepancies observed in the evaluations presented in Sect. 1. In some evaluations, the implementations, and even approaches, differed slightly."
Towards reproducibility in recommender-systems research,Article,Different implementation or use of codes,"Different users tend to have different preferences and needs, and these needs are each uniquely satis?ed by a particular product (Burns and Bush 2013). Hence, results from user studies should not be generalized to scenarios where user populations differ strongly from the original population—this is common knowledge in many research disciplines (Burns and Bush 2013). We would assume that the same holds true for recommender-system research."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Improper data analysis,By exaggerating or misinterpreting research results they naturally invite well-justified criticism.
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Improper data analysis,"For practical reasons, many if not most studies of bilingualism compare two groups: “monolinguals” and “bilinguals”. However, bilingualism is neither a dichotomous, “categorical variable” (Luk & Bialystok, 2013) nor a “unitary phenomenon” (Bak & Alladi, 2014). There can be different degrees of proficiency in bilingualism, from some knowledge of an additional language to a perfect mastery of both of them (Bialystok, in this volume). Even a relatively short exposure to a new language can lead to cognitive effects (Linck, Kroll, & Sunderman, 2009)."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Improper data analysis,"Finally, different ways of analysing the same data can lead to different conclusions. Let us turn again to the already mentioned study of active and non-active bilingualis in the Hebrides (de Bruin et al., 2015): if we analyse the performance of active bilinguals versus monolinguals separately for switch and non-switch trials we find no significant difference between them, so we could simply conclude that nothing was found and end there. If, however, we compare the difference between switch and non-switch trials in each of the groups, the difference becomes significant"
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Improper data analysis,"Indeed, it has been argued that this routine attribution of certainty to noisy data is a major contributor to the current replication crisis in psychology and other sciences (Amrhein, Korner-Nievergelt, & Roth, 2017; Open Science Collaboration, 2015)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Improper data analysis,There are many ways of carrying out statistical tests incorrectly which often will yield highly signi?cant P-values that are misleading and incorrect to an extent that cannot be adjusted for.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Improper data analysis,"As promised earlier, we now return to the issue of‘researcher degrees of freedom’ (Simmons et al., 2011), which refers to researchers’ ?exibility in how to collect and how to analyse their data. One striking issue regards stopping rules for data collection. How do you decide that you have enough data?"
Detecting and avoiding likely false-positive findings – a practical guide,Article,Improper data analysis,"Thus, when decisions about sample size are not made a priori, and data sets are subject to iterative tests for signi?cance as data accumulate, you must correct for multiple testing. The more often you stop data collection to check for signi?cance, the greater your risk of a false positive. It is important to remember here that your decision to collect data on another 10 females was conditional on the ?rst outcome."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Improper data analysis,"When analysing data, we face a wide variety of rather arbitrary decisions that we have to make, such as: (i) should I include covariate x in the model as a possible confounding factor, and should x be log-transformed or should I subdivide it into categories (and how many)? (ii) Should I include or exclude a particular outlier or an in?uential data point (high leverage)? (iii) Should I transform the dependent variable to approximate normality better, and which transformation should I choose? (iv) Should I add baseline measures taken before the start of the experiment as a covariate into the model in order to remove some noise in the data? (v) Should I control for sex as a ?xed effect or also model a sex by treatment interaction term? (vi) Should I exclude individuals from the analysis for which the number of observations is low? (vii) Should I remove a third treatment category that seems unaffected by the treatment or should I lump it with the control group? With all these decisions to make, there is again a risk of trying several versions (multiple testing) and of favouring the version that renders the more interesting story (selective reporting). Often, we may subconsciously favour the version."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Improper data analysis,"Flexibility in data collection, analysis and reporting — termed ‘researcher degrees of freedom’ by Simmons et al."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Improper data analysis,"The overwhelming majority of NHST studies relies on nil–null hypothesis testing where the null hypothesis assumes an exact value. In such cases, the null hypothesis almost always assumes exactly zero difference between groups and/or conditions. For these applications ofNHST, FRP can be computed as Oa FRP ¼ Oa þ Power where O stands for prestudy H0:H1 odds and ? denotes the statistical significance level, which is nearly always ? = 0.05. So, for given values ofO and ?, FRP is higher ifpower is low. As, in practice, O is very difficult to ascertain, high power provides the most straightforward “protection” against excessive FRP in the nil–null hypothesis testing NHST framework [5–7] (see further discussion ofour model in the Materials and Methods section)."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Improper data analysis,"Newer studies, however, point to bias in reporting results and improper data analysis as key factors that limit reproducibility and validity of preclinical mouse research."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Improper data analysis,"There are many explanations for poor replication, including subject selection bias, poor experimental control, inconsistent measurement, demand characteristics, post-hoc cherry picking of signi?cant results, partial reporting and inadequate consideration of statistical power."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Improper data analysis,"Many definitions and most analyses involve subjective judgments that leave much room for the so-called vibration of effects during statistical analysis.43 Vibration of effects means that results can differ (they vibrate over a wide possible range), dependent on how the analysis is done."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Improper data analysis,"Statistical methods can be complex, and continue to evolve in many specialties, particularly novel ones such as omics. However, statisticians and methodologists are only sporadically involved, often leading to flawed designs and analyses.72 Much flawed and irreproducible work has been published, even when only simple statistical tests are involved."
Reproducibility literature analysis - a federal information professional perspective,Article,Improper data analysis,"Incorrect or incomplete data used to influence decision-making minimizes accuracy and strategic advantage (Redman, 2008)."
Reproducibility of Published Research,"Report, policy document or website",Improper data analysis,"The second case refers to proper data analysis and the proper use of statistical methods. If an identical data set for the same question yields different results by different researchers, it is worthwhile looking into the statistical methods used, as some methods might have been incorrectly applied."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website",Improper data analysis,"Most of these issues were to do with the use of statistics — things like undersampling or use of not fully suited statistical procedures,”"
Original article experimental design in ocean acidification research: Problems and solutions,Article,Improper data analysis,"17%ofstudies otherwise segregated all the replicates for one treatment in one space,15%ofstudies replicatedCO2 treatments in away thatmade replicates moreinterdependent within treatments than between treatments"
Our path to better science in less time using open data science tools,Article,Improper data analysis,"This comes wasted time struggling to create their own conventions for managing, wrangling and versioning data. If done hap hazardly or without a clear protocol, these efforts are likely to result in work that is not reproducible—by the scientist’s own ‘future self’ or by anyone else."
Reproducibility and Research Integrity,Note,Improper data analysis,"Irreproducibility, by contrast, may indicate a problem with any of the steps involved in the research such as, but not limited to, the experimental design, variability of biological materials (such as cells, tissues or animal or human subjects), data quality or integrity, statistical analysis, or study description (Landis et al 2012, Shamoo and Resnik 2015)."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Improper data analysis,"The statement ‘a lot of what is published is incorrect’ was made in a meeting on replicability and reliability of biomedical research held at the Wellcome Trust in London in 2015. Here it was argued that science is plagued by poor practice, small samples, small effect and invalid exploratory analyses, and that much of the scientific literature might be untrue. These risks are not specific to biomedical research but have also been raised as a concern in the environmental sciences."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Incorrect interpretation of statistical findings,"However, this advantage comes with a new methodological problem: can it be that people who grow up monolingually but later learn other languages are in some way different from those who remain monolingual? In other words, does learning languages lead to cognitive benefits or do better cognitive functions lead to language learning. This question, referred to in science as “reverse causality” and in every-day life as the “chicken and egg problem” is one of the greatest challenges in the field."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incorrect interpretation of statistical findings,"Yet, no matter whether you are running a simple t-test or a restricted maximum likelihood animal model, there is always a risk of getting it wrong [for examples of mistakes that lead to over-con?dence see Had?eld et al. (2010) and Valcu & Valcu (2011)]. Hence, our ?rst point is that there are some common mistakes in the use ofstatistical tools and that these mistakes often lead to nominal signi?cance (P<0.05), yet the P-value is often incorrect and (frequently) too small, thereby contributing to false-positive claims in the literature."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incorrect interpretation of statistical findings,"False-positive conclusions can also arise from over-interpretation of differences or from misinterpretation of measurement error, which we address in Section II.4."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incorrect interpretation of statistical findings,"Although the rationale behind such a sampling design seems perfectly understandable, the risk of making a Type I error has just risen from 5% to approximately 7.7 (Simmons et al., 2011). This is because you gave the data two chances of reaching signi?cance. Since the ?rst data set is included in the second, these are not two fully independent chances (which would yield 9.75% false positives; 1?(0.95×0.95)=0.0975), so the combined risk of drawing a false-positive lies somewhere between 5 and 10% (this risk can be estimated from simulations)."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Incorrect interpretation of statistical findings,"Because a small P-value could result from random variation alone, Fisher (1937) wrote that “no isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon.” And Boring (1919) said a century ago, “scientific generalization is a broader question than mathematical description.” Yet today we still indoctrinate students with methods that claim to produce scientific generalizations from mathematical descriptions of isolated studies."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Incorrect interpretation of statistical findings,"But any selection criterion will introduce bias. If there is a tendency to publish results because the estimates are yellow, because interval estimates are short, and because P-values are small or the point estimates are far from null, then the published literature will become biased toward yellow results with underestimated variances and overestimated effect sizes. The latter effect is the “winner’s curse,” or inflation ofeffect sizes, that is reflected in the findings of the Open Science Collaboration (2015): the average effect size in the original studies was about twice as large as in the replication studies that reported all results and thus did not suffer from selection bias"
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Incorrect interpretation of statistical findings,"Keep in mind that these statistics do not “measure” uncertainty. At best, the interval estimate may give a rough idea of uncertainty, given that all the assumptions used to create it are correct. And even then, we should remember the “dance of the confidence intervals” (Cumming 2014) shows a valid interval will bounce around from sample to sample due to random variation."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Incorrect interpretation of statistical findings,"Claims in the neuroimaging literature are often asserted without corresponding statistical support. In particular, failures to observe a statistically significant effect can lead researchers to proclaim the absence of an effect — a dangerous one."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Incorrect interpretation of statistical findings,"‘Reverse inference’ claims, in which the presence of a given pattern of brain activity is taken to imply a specific cognitive process (for example, “the anterior insula was activated, suggesting that subjects experienced empathy”), are rarely grounded in quantitative evidence. Furthermore, claims of ‘selective’ activation in one brain region or experimental condition are often made when activation is statistically significant in one region or condition but not in others. This false assertion ignores the fact that “the difference between ‘significant’ and ‘not significant’ is not itself statistically significant”; such claims require appropriate tests for statistical interactions."
MAKING REPLICATION MAINSTREAM,Article in Press,Incorrect interpretation of statistical findings,"Yet there are additional forces and practices that can increase the rates of false positives. For example, there is a growing body of meta-scienti?c research showing the effects of excessive researcher degrees of freedom (John et al. 2012; Simmons et al. 2011) or latitude in the way research is conducted, analyzed, and reported. If researchers experience pressure to publish statistically signi?cant ?ndings, then the existence of researcher degrees of freedom allows investigators to try multiple analytic options until they ?nd a combination that provides a signi?cant result."
MAKING REPLICATION MAINSTREAM,Article in Press,Incorrect interpretation of statistical findings,"Importantly, con?rmation bias alone can convince investigators that the procedures that led to this signi?cant result were the “best” or “most justi?able” approach in the ?rst place. Thus, capitalizing on researcher degrees of freedom need not feel like an intentional decision to try multiple options until a set of procedures “works” (Gelman & Loken 2014). It can seem like a reasonable approach for extracting the most information from a data set that was dif?cult to collect."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Incorrect interpretation of statistical findings,"Quite often, significance testing appears like a sort of gambling. Even a study with minimized investment into sample sizes will yield significant results, if only enough variables are measured and the right buttons in the statistical software are pushed. Small sample sizes further have it that significant effects will probably be inflated, so large and surprising effects are almost guaranteed. And we may have become somewhat addicted to this game—it is satisfying to feed data into the machine and find out whether the right variables have turned significant."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,Incorrect interpretation of statistical findings,We additionally encourage authors to be aware that the calculated P value is almost always bigger than it seems (‘less signi?cant’)owing to the false discovery rate which is one reason why some investigators argue that most (rather than just some) research ?ndings are false (Colquhoun 2014; Begley 2013; Begley & Ioannidis 2015).
Original article experimental design in ocean acidification research: Problems and solutions,Article,Incorrect interpretation of statistical findings,"Some studies did not report degrees of freedom adequately. If the degrees of freedom reported are larger than the total number of experimental units, then samples have been pooled in an inappropriate way."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Incorrect interpretation of statistical findings,"A study may, for example, be considered less reliable because of an inadequate experimental design (e.g., too few replicates), poor performance (e.g., mortality is too high in the controls), or insuf?cient data analysis (e.g., inadequate statistics)."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Incorrect interpretation of statistical findings,"Generations of scientists encouraged by incorrect editorial interpretations (Bakan, 1966) started to exclusively rely on the p-value in their decisions even if this meant neglecting their substantive knowledge: scienti?c conclusions merged with reading the pvalue (Goodman, 1999)."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Incorrect interpretation of statistical findings,There are also differences in interpretability of positive and null findings (compounded by common design flaws such as having low statistical power) which mean that positive results can be misguidedly seen to overcome methodological weakness that would be critical for a null finding.
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Low statistical power (small n),"Whenever an effect in an underpowered study comes out significant, it is necessarily an overestimate. In fields where power tends to be low, these overestimates will fill the literature. If we base the power analysis on the published literature, we would conclude that the effects are large. A formal power analysis based on such exaggerated estimates is bound to yield an overestimate of power, and we can incorrectly convince ourselves that we have an appropriately powered study."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Low statistical power (small n),"A central problem is that underpowered studies can yield a statistically significant result due to Type M error, and these significant results will be overestimates."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Low statistical power (small n),"The smaller estimate with narrower credible intervals may reflect reality better. Thus, when power is low, using significance to decide whether to publish a result leads to a proliferation of exaggerated estimates in the literature."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Low statistical power (small n),"We brie?y discuss two approaches to addressing this combined problem of small data sets and lack of replication, both of which set the scene for the introduction of Registered Reports at Language Learning."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Low statistical power (small n),"In many studies, sample sizes are low, resulting in statistical power that is often as low as 20% (Møller & Jennions, 2002; Smith, Hardy & Gammell, 2011; Button et al., 2013; Parker et al., 2016)."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Low statistical power (small n),"Median power to detect small, medium, and large effects was 0.12, 0.44, and 0.73, reflecting no improvement through the past half-century. This is so because sample sizes have remained small."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Low statistical power (small n),We established that the statistical power to discover existing relationships has not improved during the past halfcentury. A consequence oflow statistical power is that research studies are likely to report many false positive findings.
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Low statistical power (small n),"Low power is usually only associated with failing to detect existing (true) effects, and therefore, with wasting research funding on studies which a priori have a low chance to achieve their objective. However, low power also has two other serious negative consequences: it results in the exaggeration ofmeasured effect sizes and it also boosts FRP, the probability that statistically significant findings are false."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Low statistical power (small n),"First, if we use Null Hypothesis Significance Testing (NHST), then published effect sizes are likely to be, on average, substantially exaggerated when most published studies in a given scientific field have low power. This is because even ifwe assume that there is a fixed true effect size, actual effect sizes measured in studies will have some variability due to sampling error. Underpowered studies will be able to classify as statistically significant only the occasional large deviations from real effect sizes. Conversely, most measured effects will remain under the statistical significance threshold even if they reflect true relationships. Effect size inflation is greater when studies are even more underpowered. Consequently, while meta-analyses may provide the illusion ofprecisely estimating real effects, they may, in fact, estimate exaggerated effects detected by underpowered studies while at the same time not considering unpublished negative findings."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Low statistical power (small n),"The trustworthiness of statistically significant findings depends on power, prestudy H0:H1 odds, and experimenter bias."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Low statistical power (small n),There are also differences in interpretability of positive and null findings (compounded by common design flaws such as having low statistical power) which mean that positive results can be misguidedly seen to overcome methodological weakness that would be critical for a null finding.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Overconfidence in statistical results affects reproducibility,"The key problem is that we are expecting too much from every single empirical study (despite knowing that most effect sizes are small and hence power is very limited), meaning that we set up the unrealistic expectation that each study should yield a clear-cut conclusion by itself. This leads to a situation where junior scientists complain when their laborious efforts of data collection have not yielded a signi?cant ?nding, meaning that their work cannot be published. In response senior scientists help with advice on alternative data analyses designed to squeeze out something signi?cant. And there are more consequences of our unrealistic expectations."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Overconfidence in statistical results affects reproducibility,"In the following, we argue that the crisis of unreplicable research is mainly a crisis of overconfidence in statistical results. We recommend that we should use, communicate, and teach inferential statistical methods as describing logical relations between assumptions and data, rather than as providing generalizable inferences about universal populations."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Overconfidence in statistical results affects reproducibility,"Further assumptions that are often explicitly addressed in research reports are that sampling was random or that residuals are independent and identically distributed. Other assumptions may not even be recognized or mentioned in research reports, such as that there was no selection of particular results for presentation, or that the population from which we drew our random sample is equivalent to the population we have targeted for inference. Whether it is assumptions that are reviewed by inspecting residuals, or further assumptions that link statistics to reality, the validity of statistical inferences depends on the entire set of assumptions."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Overconfidence in statistical results affects reproducibility,"A 95% confidence interval cannot realistically be claimed to have as much as 95% coverage of the true effect when study imperfections exist. Consequently, having confidence in generalizations from single studies means having overconfidence in most cases. Inference that could be called trustworthy would require merging information from multiple studies and lines of evidence."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Materials and reagents used are not always accurately detailed by providers nor validated,"Equally vexatious is the ongoing crisis promulgated by use of antibodies that have not been properly validated and, as a result, generate irreproducible or incorrect data due to lack of sensitivity and/or problems with speci?city."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Materials and reagents used are not always accurately detailed by providers nor validated,"To produce experimental results that are both true and reproducible, key reagents must be well described and they must be capable of producing the claimed measurement outcomes. Problems with reagents, including antibodies, cell lines, chemical agents, and experimental animals, are the source of many issues in irreproducible studies. Antibodies that claim to have speci?city for antigens, but lack such, commonly continue to be employed even after the demonstrated lack of speci?city has been published. There are well documented examples of widely employed cell lines being misidenti?ed as having a speci?c lineage or properties. There are also many complexities in the use of animal models, especially mouse models. How these may lead to poor reproducibility as well as failure to predict outcomes when applied to humans has recently been reviewed."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Poor experimental design,"The question of the selection of participants, based on the definitions of bilingualism, has entered the debate more recently. As already mentioned in previous sections, where we put the “bilingualism threshold” might well decide whether we find significant results or not. In epidemiological studies, such as those exploring the potential influence of bilingualism on the age of onset of dementia, the sampling frame and the choice of the outcome measures can play a crucial role."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Poor experimental design,"This bias can be minimized by ‘blinding’ observers to the hypotheses being tested or to the treatment categories of the individuals being measured. However, blinding is rare."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Poor experimental design,"Poor experimental methodology Most instances of irreproducibility likely result from the failure to properly design, execute, and evaluate experimental data e activities at the core of what is required to conduct excellent research."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Poor experimental design,"We “don’t look for a magic alternative to NHST [null hypothesis significance testing], some other objective mechanical ritual to replace it. It doesn’t exist” (Cohen 1994). And if it existed, we would probably not recommend it for scientific inference. What needs to change is not necessarily the statistical methods we use, but how we select our results for interpretation and publication, and what conclusions we draw. Why would we want a mechanical decision procedure for single studies, if not for selecting results for publication or interpretation? As we described above, every selection criterion would introduce bias. Therefore, we join others who have advised that we should, to the extent feasible."
Towards reproducibility in recommender-systems research,Article,Poor experimental design,"Since minor variations in approaches and scenarios can lead to signi?cant changes in a recommendation approach’s performance, ensuring reproducibility of experimental results is dif?cult."
Towards reproducibility in recommender-systems research,Article,Poor experimental design,"In the recommender-systems community, we found that reproducibility is rarely given, particularly in the research-paper recommender-system community (Beel et al. 2013b, 2015a). In a reviewof 89 evaluations of research-paper recommender-systems, we found several cases in which very slight variations in the experimental set-up led to surprisingly different outcomes"
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Poor experimental design,"Poor experimental design combined with a lack of rigor in reporting and reviewing has contributed to irreproducibility of findings, which is particularly rife in work that uses preclinical models."
Reproducibility Crisis: Are We Ignoring Reaction Norms?,Letter,Poor experimental design,"Poor experimental design and conduct, including small sample sizes, risks of bias, selective reporting, and publication bias. In a recent review in this journal, Jarvis and Williams drew a more nuanced picture, challenging the increasingly common view that irreproducibility re?ects poor research conduct. While we applaud their effort to reset the frame and their suggestions for improving research practice, we feel that they have missed perhaps the most important point: ignorance of phenotypic plasticity in experimental design and analysis."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Poor experimental design,"There are many explanations for poor replication, including subject selection bias, poor experimental control, inconsistent measurement, demand characteristics, post-hoc cherry picking of signi?cant results, partial reporting and inadequate consideration of statistical power."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Poor experimental design,"When effects and biases are potentially of similar magnitude, the validity of any signal is questionable. Design choices can increase the signal, decrease the noise, or both."
Reproducibility literature analysis - a federal information professional perspective,Article,Poor experimental design,"Experimental design aims to describe or explain the variation of information under conditions that are hypothesized to reflect the variation, and use this knowledge to collect more accurate data (AcevesBueno et al., 2017). A framework for a systematic process to guide researchers and reviewers in assessing, documenting, and mitigating the sources of uncertainty in a study enhance comparability and reproducibility (Plant et al., 2018). Experimental design features should enhance, or facilitate inference about, the reproducibility and generalizability of the expected results (Würbel, 2017). [Related terms: pre-registration of results; case study]"
Original article experimental design in ocean acidification research: Problems and solutions,Article,Poor experimental design,"Among the 30% of studies (i.e. 180) where we could determine if treatment replicates were interspersed randomly or not, 36 studies employed a randomized design, whereas 130 were non-random and 13 employed no replication oftreatments."
Original article experimental design in ocean acidification research: Problems and solutions,Article,Poor experimental design,The tendency for the use of inappropriate experimental design also undermines our con?dence in accurately predicting the effects of ocean acidi?cation on the biological responses ofmarine organisms.
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Poor experimental design,"A study may, for example, be considered less reliable because of an inadequate experimental design (e.g., too few replicates), poor performance (e.g., mortality is too high in the controls), or insuf?cient data analysis (e.g., inadequate statistics)."
How we can make ecotoxicology more valuable to environmental protection,Note,Poor experimental design,"There is widespread and growing concern that the quality, usability, and reporting of published peer-reviewed research is not as good as it could, and should, be. This can undermine the credibility and functioning of the scienti?c endeavor (Alberts et al., 2014; Forbes et al., 2016) and is a conversation that has spread beyond just the scienti?c community (e.g., The Economist, 2013). Poor science and reporting also come with steep economic costs. For example, it has been estimated that irreproducible results in the biomedical literature cost 28 billion USD in America alone, each year (Freedman et al., 2015)."
How we can make ecotoxicology more valuable to environmental protection,Note,Poor experimental design,"In addition to the direct economic costs from repeating poorly conducted studies that report spurious results, poor quality research delays and hinders protection of the environment, which is an underlying reason for conducting ecotoxicology research."
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences,Article,Poor experimental design,"How many trials should be administered? We found that the literature on these seven tasks also lacks information to guide researchers on how many trials to run, and different studies can choose very different numbers without any explicit discussion or justification."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Poor experimental design,"We argue that contingencies of publication bias that led to the “replication crisis” also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication ofSCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Poor experimental design,"Thus a third asymmetry arises in study quality; design limitations are seen to weaken the case for publishing a null result, while passing the 5% significance criterion can be seen as a golden ticket for dismissing away methodological concerns."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"At present, positive findings are more attractive to researchers, funders and publishers alike","Attempts to improve rates of replication have met many challenges (Porte, 2012), including some imposed by publishing venues themselves. The quantity ofreplication is, perhaps, in?uenced by the extent to which journals encourage or discourage replication. To investigate how psychology journals approach this issue, Martin and Clarke (2017) reviewed the scope sections of author guidelines of 1,151 journals and found that 63% did not state that they accepted replications, but neither did they discourage them; 33% implicitly discouraged them by emphasizing originality, novelty, or innovation of submissions; 3% of journals stated that they accepted them; and 1% actively discouraged replications by stating that they did not publish them. The fact that only 3%ofjournals stated that they accepted replications may partly be due to the perceived impact, and hence prestige, of replication."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike","Currently, due to the unreasonable pressure to publish fast and to report novel results in top journals, crucial data-analysis decisions are often made after examining the data."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"At present, positive findings are more attractive to researchers, funders and publishers alike","First, one likely reason for the low rates of replication research is a concern (warranted or not) that, due to unfavorable reviews, a replication study will not be published if it does not reproduce the initial study’s ?ndings. As a consequence, many replication studies are probably consigned by the researchers to the ?le drawer."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike","Statistically signi?cant ?ndings typically seem more interesting than non-signi?cant ?ndings and are thus easier to publish. This has created our current scienti?c culture of actively seeking statistical signi?cance, often with practices that lead to misleading results. We hence try to raise the general awareness of psychological biases that we need to keep in check in order to ensure an objective reporting of research outcomes."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike","Hence, if we started with the aim of objectively quantifying something (rather than discovering something) we should face less of a risk of misleading ourselves and our colleagues and of having wasted efforts for the short-term bene?t of possibly publishing in a higher-ranking journal."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike",Funding agencies and journal editors focus on novelty. This is particularly hard to justify on the part of funders since failing to invest in replication means failing to seek robust answers to questions they already have made a commitment to answering.
Detecting and avoiding likely false-positive findings – a practical guide,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike","In the case of journals, pursuit of novelty may be harder to curb, but there are paths to reducing the tyranny of this pursuit. Journals seek novelty in part because of the competition for impact factors. Studies which report surprising (i.e. unlikely) ?ndings are often highly cited and thus contribute to the stature of the journal. Thus, ‘the more surprising, the better’. This effect may be exacerbated by the for-pro?t publishing industry. Fortunately, replications can also be heavily cited."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"At present, positive findings are more attractive to researchers, funders and publishers alike",A research system in which results are much more likely to get reported if they reach statistical signi?cance violates scienti?c objectivity and is highly inef?cient if our interest lies in the quanti?cation of effect sizes because unbiased effect-size estimates are dif?cult to obtain in such a system.
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"At present, positive findings are more attractive to researchers, funders and publishers alike","How did we arrive at this state of affairs? Although it is admittedly more dif?cult to become an elite ?ghter pilot, astronaut, or head of state, competition for faculty positions and resources in the best academic institutions is ?erce, and the most valuable currency continues to be a mixture of publications in ‘‘the best journals,’’ ideally coupled with already secured independent funding. To obtain these valuable prestigious publications, one must meet the standards and expectations of journal editors, who similarly prize research that is spectacular, highly novel, and ideally accompanied by wellde?ned reductionist mechanisms and immediate obvious translational relevance."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"At present, positive findings are more attractive to researchers, funders and publishers alike","On the other end of the spectrum, many highly prestigious and sought-after journals seek to publish papers that make exciting claims that will generate buzz in the scienti?c community and garner press attention, goals that are easy to understand. To accomplish this goal, their decisions to accept papers are sometimes contingent on authors producing speci?c results suggested by reviewers or editors. Such an approach is distinct from criteria for acceptance that primarily demand well conducted studies where data support the conclusions and ?ndings advance the ?eld. The highest impact journals can easily afford to pass on such papers. Sometimes these editorial requests occur late in what is too often a lengthy review process, causing authors to fear loss of priority, or inability to use the acceptance in support of grant submissions or promotions. The power of such journals, re?ecting in part the ability of their decisions to in?uence funding and promotions, creates a strong, if misplaced, incentive to ful?ll the editorial requests. We may surmise that some investigators make decisions in data selection that they would not make absent the high stakes editorial exchanges. To describe this sequence is in no way to justify it."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"At present, positive findings are more attractive to researchers, funders and publishers alike","This leads to a fundamental problem for the life sciences, especially preclinical research: a huge vested interest in positive results has mitigated against replication. Authors have grants and careers at stake, journals need strong stories to generate headlines, and pharmaceutical companies have invested large amounts of money in positive results and patients yearn for new therapies"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,"At present, positive findings are more attractive to researchers, funders and publishers alike","The root of the problem is the publication bias, caused by journals seeking theoretical novelty with empirical con?rmation, in combination with counterproductive universitylevel career incentives focused on publications in a limited number of journals (for a recent summary, see van Witteloostuijn, 2016)."
On Replication in Communication Science,Editorial,"At present, positive findings are more attractive to researchers, funders and publishers alike","As Hunter (2001) notes, such a study is likely to be rejected from most journals, precisely because it is not novel and does not add new knowledge. Yet these studies are of great importance. Without replication the original studies also fail to add knowledge to our field."
On Replication in Communication Science,Editorial,"At present, positive findings are more attractive to researchers, funders and publishers alike","Another is that reviewers may deem the replication results insufficiently “novel” for publication. However, when articles are rejected for not being novel enough, when journals worry that publishing replications will harm their impact factor, when new scholars are steered away from doing replication work for fear it might harm their job prospects or tenure process, then we lose this critical part of the scientific process. Without replication, we are no longer doing good science; we are simply playing with numbers."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Incomplete or biased reporting,"It would be tempting but inaccurate to put all the blame on the media; the tendency to exaggerate the potential relevance of research findings can be traced back to academic press releases (Sumner et al, 2014), if not to the statements by scientists themselves. The current emphasis on “impact” in an environment shaped by increasing competition means that many scientists and their institutions feel under pressure to produce constantly novel and socially relevant results: an atmosphere unlikely to encourage a balanced and self-critical attitude, which characterises the best of scientific endeavour. So it might be worth to start our considerations by discussing some common myths and misconceptions about bilingualism and cognition, wherever they might have originated."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Incomplete or biased reporting,"Indeed, the problems facing bilingualism research today are a reflection of a much broader debate in modern science, centred on issues such as “publication bias”, “decline effect” and “replication crisis”. Most of them are not new. Conflicting evidence and theoretical debates have accompanied science since its very infancy: indeed, it would be hardly possible to imagine science without them. The question of publication bias has been discussed for over 50 years, since Stirling’s seminal observation that out of 298 papers published in leading psychological journals, 286 confirmed the original hypothesis (Sterling, 1959). He argued that this is likely to have resulted from a selective publication practice with a bias towards confirmatory results and observed a similar trend across natural and social sciences of his time. Indeed, subsequent large studies have confirmed the presence of a pronounced publication bias in medical sciences (Easterbrook, Gopalan, Berlin, & Matthews, 1991)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Incomplete or biased reporting,"Publication bias—a tendency for journals to publish and/or researchers to submit only statistically signi?cant ?ndings—is a widely acknowledged problem, and null ?ndings are con?ned to the “?le drawer,” a term coined by Rosenthal (1979) and a phenomenon documented by many scholars (e.g., Bakker, van Dijk, & Wicherts, 2012; Schmidt & Oh, 2016; Sterling, Rosenbaum, & Weinkam, 1995; Sutton, 2009). Though the extent of ?eld-wide publication bias in L2 research has not yet been systematically studied, it likely exists (Fanelli, 2012; Plonsky, 2013), and several meta-analysts have found some evidence of it in speci?c domains (Lee & Huang, 2008; Lee, Jang, & Plonsky, 2015; Plonsky, 2011). This means that even unintentionally, anyone choosing a study to replicate is likely, due to chance alone, to select one with statistically signi?cant ?ndings."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Incomplete or biased reporting,"In sum, issues such as the initial reporting ofmethods, results, and analysis; the availability of the initial materials and data; and the resources needed may all reduce the likelihood, quality, or usefulness of replication (even when a replication is clearly warranted)."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Incomplete or biased reporting,"Given that significant results are favored by journals and reviewers, effects reported in the literature are guaranteed to be overestimates when power is low."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Incomplete or biased reporting,"There is in principle no harm in publishing low-powered studies in top journals, as long as strong claims are avoided. This is what statisticians mean when they suggest that researchers “accept uncertainty and embrace variation” (McShane et al., 2017). Currently, in psycholinguistics and other areas, we are taught to have the expectation that every experiment be a “win.” Under this prior belief in routine success, even null results from low-powered studies start to look informative."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Incomplete or biased reporting,"Crucial data-analysis decisions are often made after examining the data. For example, the same researcher will often include or exclude data on different criteria, so that it eventually passes the statistical significance filter. Sometimes, excluding or including a few data points can make the difference between significance and non-significance (Vasishth et al., 2013). Another example is region-of-interest selection in reading studies: researchers often change the region of interest from study to study or even within a study, driven exclusively by the search for significance (an example is discussed in Vasishth & Nicenboim, 2016). Another common approach is to run the study, check for significance, then either run more participants if significance is desired but not reached, or stop collecting data if a null result is desired. These decisions are often not reported in the published paper. Pre-registration would remove these degrees of freedom and thereby ensure a clear separation between confirmatory and exploratory analyses (De Groot, 1956/2014)."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Incomplete or biased reporting,"Too often, published empirical results are treated as a novel contribution simply because of the application of the statistical significance filter."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Incomplete or biased reporting,"Finally, there are signi?cant concerns about publication practices themselves. Of key signi?cance is the well-attested phenomenon of publication bias, whereby authors tend to submit, and journals tend to publish, ?ndings that show statistical signi?cance or align with the outcome that is perceived as being more exciting."
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,Incomplete or biased reporting,We are still concerned to see that major conclusions in some articles are not sufficiently supported by the experiments performed or the information reported.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,We argue that this problem may originate from a culture of ‘you can publish if you found a signi?cant effect’. This culture creates a systematic bias against the null hypothesis which renders meta-analyses questionable and may even lead to a situation where hypotheses become dif?cult to falsify.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"Flexibility in de?ning and testing our hypotheses, combined with selective reporting of apparent cases of success hence leads to a high risk of publishing false-positive ?ndings."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"However, as soon as reporting becomes conditional on the outcome (typically: positive ?ndings being more likely to get reported) or when we focus our attention on the promising outcomes (ignoring or forgetting about negative outcomes), the risk of a false-positive conclusion is much higher than 5% (e.g. 53%)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"When, for instance, the authors highlight a single signi?cant ?nding from a pool of 10 tests they report, this inspires much less con?dence in that ?nding than if it had arisen from a single planned test. This is a serious dilemma."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"There is compelling evidence that many tests do, in fact, go unreported."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"As mentioned above, across scienti?c disciplines, 84% of all studies present positive support for their key hypothesis (Fanelli, 2010). Such a high success rate is impossible to obtain without selective reporting or biased attention that de-emphasizes non-signi?cant ?ndings or likely a combination of both."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"Hence, this means that most disciplines presumably sit on a huge pile of ‘failed’ experiments and unpublished null results that are inaccessible because they are hidden in the ?le-drawers of the experimenters [known as the ‘?le-drawer problem’ (Rosenthal, 1979)]."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"Although any meta-analytic summary would certainly reveal a strong effect of ornaments on mating success, it is unclear whether or to what extent this is evidence for a theory as opposed to evidence of selective reporting driven by a theory."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"We end up with a large amount of wasted effort because non-signi?cant parameter estimates end up unpublished in the so-called ‘?le-drawer’. And, the studies that make it to the publication stage often yield parameter estimates that are biased upwards or are simply false positive."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Incomplete or biased reporting,"Estimates therefore paint a distorted picture of the reality that we set out to study. How absurd is a system in which we measure an effect of interest with meticulous accuracy, but then subject our measure to a self-imposed censorship by only reporting it if it exceeded a certain strength?"
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Incomplete or biased reporting,"These unpublished experiments are not restricted to the academic sector, as the pharmaceutical industry also undertakes a great deal of preclinical research that may never be publicly disclosed."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Incomplete or biased reporting,"Incentives produced by the major funders of bioscience research NIH and other funding agencies place great emphasis on, and often have requirement for, research proposals with well delineated hypotheses and “expected ?ndings”, some of which are expected to have been demonstrated at the time of grant submission. It has been persuasively argued that this approach to both conducting and funding science has major conceptual ?aws. As relates to the issue of reproducibility, a key ?aw of this requirement is its’ placing the scientist in a position where data is ?ltered through the lens of the stated hypothesis, in a way that promotes expectation of a particular result, and biases against, or promotes rejection of, contradictory evidence. It is easy to see how this construct puts great pressure on a scientist to avoid falsifying the hypothesis upon which their grant was funded, even when evidence suggests this is the most rational approach."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Incomplete or biased reporting,"There is also a more prevalent problem of publication bias, wherein new and positive ?ndings are far more easily published than are con?rmatory or negative ?ndings. Very often, the latter go unpublished, to the detriment of scienti?c knowledge."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Incomplete or biased reporting,"This leads to a fundamental problem for the life sciences, especially preclinical research: a huge vested interest in positive results has mitigated against replication. Authors have grants and careers at stake, journals need strong stories to generate headlines, and pharmaceutical companies have invested large amounts of money in positive results and patients yearn for new therapies."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Incomplete or biased reporting,"However, perhaps the biggest elephant in the room is related to publication bias towards positive results and away from the null hypothesis. This affects replication in so far that not just negative but also confirmation studies tend not to get reported."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Incomplete or biased reporting,Earlier work also discusses the community’s focus on abstract accuracy measures or the narrow focus of machine learning research in terms of what is “publishable” at top publication outlets.
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Incomplete or biased reporting,"The “crisis of unreplicable research” is not only about alleged replication failures. It is also about perceived nonreplication of scientific results being interpreted as a sign ofbad science (Baker 2016). Yes, there is an epidemic ofmisinterpretation ofstatistics and what amounts to scientific misconduct, even though it is common practice (such as selectively reporting studies that “worked” or that were “significant”; Martinson, Anderson, and de Vries 2005; John, Loewenstein, and Prelec 2012)."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Incomplete or biased reporting,"Even if authors report all study outcomes, but then select what to discuss and to highlight based on P-value thresholds or other aids to judgment, their conclusions andwhat is reported in subsequent news and reviews will be biased (Amrhein, KornerNievergelt, and Roth 2017). Such selective attention based on study outcomeswill, therefore, not only distort the literature but will slant published descriptions of study results—biasing the summary descriptions reported to practicing professionals and the general public."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Incomplete or biased reporting,"Unfortunately, reporting standards in the fMRI literature remain poor. Carp and Guo et al. analysed 241 and 100 fMRI papers, respectively, for the reporting of methodological details, and both found that some important analysis details (such as interpolation methods and smoothness estimates) were rarely described. Consistent with this, in 22 of the 66 papers that we discussed above, it was impossible to identify exactly which multiple-comparison correction technique was used (beyond generic terms such as ‘cluster-based correction’), because no specific method or citation was provided."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Incomplete or biased reporting,"The trustworthiness of statistically significant findings depends on power, prestudy H0:H1 odds, and experimenter bias."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Incomplete or biased reporting,"Hence, sticking to low participant numbers may facilitate finding statistically significant publishable (false positive) results. It is also important to consider that complicated instrumentation and (black box) analysis software is now more available, but training may not have caught up with this wider availability"
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Incomplete or biased reporting,"Related concern is the negative correlation between power and journal impact factors. This suggests that high impact factor journals should implement higher standards for prestudy power (optimally coupled with preregistration ofstudies) to assure the credibility of reported results. Speculatively, it is worth noting that the high FRP allowed by low power also allows for the easier production ofsomehow extraordinary results, which may have higher chances to be published in high impact factor journals."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Incomplete or biased reporting,"For example, researchers may neglect multiple testing correction; post hoc select grouping variables; use machine-learning techniques to explore a vast range ofpost hoc models, thereby effectively p-hacking their data by overfitting models (http://dx.doi.org/10.1101/078816); and/ or liberally reject data not supporting their favored hypotheses. Some ofthese techniques can easily generate 50% or more false positive results on their own while outputting some legitimate looking statistics. In addition, it is also well documented that a large number of p-values are misreported, indicating statistically significant results when results are, in fact, nonsignificant."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Incomplete or biased reporting,"Trial descriptions of exercise interventions are often suboptimal, leaving readers unclear about the content of effective programmes."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Incomplete or biased reporting,"Complete and explicit reporting of the components of complex interventions, including contextual factors, in clinical trials evaluating their effects is essential to the interpretation, translation and implementation of research ?ndings into clinical practice. Yet, these interventions are often incompletely described in study reports."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Incomplete or biased reporting,"In general, methodologic aspects were adequately documented. However, only 7 of 35 studies reported detailed information in every one of the aforementioned quality domains. In 3 quality aspects, the overall standard of reporting was lower: (1) providing details of the software implementation to extract radiomic features, (2) providing details pertaining to image preprocessing before extracting"
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Incomplete or biased reporting,"One study did not provide detailed information about the disease groups used in the analysis, apart from stating that different types of solid cancers were included."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Incomplete or biased reporting,"Software details (application framework used for analysis, programming language, and version) were not reported in detail in 16 studies."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Incomplete or biased reporting,Fifteen studies did not document the cutoff value used in their statistical metrics to discriminate between reproducible and irreproducible features.
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Incomplete or biased reporting,"Nearly one-third (46, 30.9% [23.7% to 39.0%]) ofthe 149 biomedical articles did not include information on funding. There were 78 articles (52.3% [44.0% to 60.5%]) that were publicly funded, either alone or in combination with other funding sources. Ofthese, three received National Science Foundation (NSF) support and 25 had NIH funding, either alone or in combination with other funding sources. Among the 149 articles, there were 52 (34.9% [27.4% to 43.2%]) that did not include a conflicts of interest statement. However, there were 87 (58.4% [50.0% to 66.3%]) that specifically reported no conflicts ofinterests and 10 (6.7% [3.4% to 12.3%]) that included a clear statement of conflict."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Incomplete or biased reporting,"Newer studies, however, point to bias in reporting results and improper data analysis as key factors that limit reproducibility and validity of preclinical mouse research."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Incomplete or biased reporting,"Poor experimental design combined with a lack of rigor in reporting and reviewing has contributed to irreproducibility of findings, which is particularly rife in work that uses preclinical models."
MAKING REPLICATION MAINSTREAM,Article in Press,Incomplete or biased reporting,"Problems with replicability can emerge for a variety of reasons. For example, publication bias, the process by which research ?ndings are selected based on the extent to which they provide support for a hypothesis (as opposed to failing to ?nd support), can on its own lead to high rates of false positives (Greenwald 1975; Ioannides 2005; Kühberger et al. 2014; Smart, 1964; Sterling 1959; Sterling et al. 1995)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Incomplete or biased reporting,Current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings.
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Incomplete or biased reporting,"However, when nonsignificant results on a particular hypothesis remain unpublished, any significant support for the same hypothesis is rendered essentially uninterpretable (ASA statement; Wasserstein & Lazar, 2016). Ifwhite swans remain unpublished, reports ofblack swans cannot be used to infer on general swan color. In the worst case, publication bias means according to Rosenthal (1979) that the 95% of studies that correctly yield nonsignificant results may be vanishing in file drawers, while journals may be filled with the 5% of studies committing the alpha error by claiming to have found a significant effect when in reality the null hypothesis is true."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Incomplete or biased reporting,"There are many explanations for poor replication, including subject selection bias, poor experimental control, inconsistent measurement, demand characteristics, post-hoc cherry picking of signi?cant results, partial reporting and inadequate consideration of statistical power."
How to Make More Published Research True,Article,Incomplete or biased reporting,"We must diminish biases, conflicts of interest, and fragmentation of efforts in favor of unbiased, transparent, collaborative research with greater standardization."
How to Make More Published Research True,Article,Incomplete or biased reporting,"Currently, many published research findings are false or exaggerated, and an estimated 85% of research resources are wasted."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Incomplete or biased reporting,Cross-validation of a single dataset might yield inflated results because of biases.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Incomplete or biased reporting,"Publication bias, also known as reporting bias, is the phenomenon in which only some of the results of research are published and therefore made available to inform evidence-based decision-making. Papers that report “positive results”, such as positive effects of drugs on a condition or disease, are more likely to be published, are more likely to be published quickly and are likely to be viewed more favourably by peer reviewers (McGauran et al. 2010; Emerson et al. 2010)."
Reproducibility literature analysis - a federal information professional perspective,Article,Incomplete or biased reporting,"Bias includes prejudice in favor of or against one thing, person, or group compared with another, usually in a way considered unfair. Two common types of bias in research studies are confirmation bias and hindsight bias. Confirmation bias promotes the tendency to focus on evidence that is in line with our expectations or favored explanation. Hindsight bias is the tendency to see an event as having been predictable only after it has occurred (Munafò et al., 2017). [Related terms: cherry picking; replication studies (solution)]"
Reproducible Research A Retrospective,"Report, policy document or website",Incomplete or biased reporting,"Because of the increasing complexity of data analyses, many choices and decisions must be made by analysts in the process of obtaining a result. With these increasing complexities we also increase the risk of human error and bias in data analysis. These choices and decisions often have an unknown impact on the ?nal estimates produced and therefore may or may not be recorded by the investigators. These “research degrees of freedom” allow investigators to unknowingly, or perhaps knowingly, steer data analyses in directions that may support speci?c hypotheses rather than represent all of the evidence in the data."
The Resource Identification Initiative A cultural shift in publishing,"Report, policy document or website",Incomplete or biased reporting,"Because current practices for reporting research resources within the literature are inadequate, nonstandardized, and not optimized for machine-readable access, it is currently very difficult to answer very basic questions about published studies."
How we can make ecotoxicology more valuable to environmental protection,Note,Incomplete or biased reporting,"There is increasing awareness that the value of peer-reviewed scienti?c literature is not consistent, resulting in a growing desire to improve the practice and reporting of studies."
How we can make ecotoxicology more valuable to environmental protection,Note,Incomplete or biased reporting,"There is widespread and growing concern that the quality, usability, and reporting of published peer-reviewed research is not as good as it could, and should, be. This can undermine the credibility and functioning of the scienti?c endeavor (Alberts et al., 2014; Forbes et al., 2016) and is a conversation that has spread beyond just the scienti?c community (e.g., The Economist, 2013). Poor science and reporting also come with steep economic costs. For example, it has been estimated that irreproducible results in the biomedical literature cost 28 billion USD in America alone, each year (Freedman et al., 2015)."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Incomplete or biased reporting,"In conferences we may have also heard about 9 highly powered but failed replication attempts very similar to the original study. In this case we may assume that the odds of H0:H1 are 9:1, that is, pr(H1) is 1/10. Of course, these pre-study odds are usually hard to judge unless we demand to see our colleagues’ “null results” hidden in their drawers because of the practice of not publishing negative ?ndings. Current scienti?c practices appreciate the single published “positive” study more than the 9 unpublished negative ones."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Incomplete or biased reporting,"The currently dominant, NHST in?uenced approach is that instead of understanding raw data researchers often just focus on the all or nothing rejection of a vaguely de?ned H0 and shift their attention to interpreting brain “activations” revealed by potentially highly misleading statistical parameter maps. Based on these maps then strong (qualitative) claims may be made about alternative theories whose support may in fact never be tested. So, current approaches seem to reward exuberant theory building based on small and underpowered studies (Szucs and Ioannidis, 2017)"
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Incomplete or biased reporting,"Central to the problem is the peer-review system, and the role it plays in perpetuating biases in the published record; generally, authors, reviewers, and editors prefer results which show support for tested hypotheses and are prejudiced against submitting or publishing inconclusive or null findings."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Incomplete or biased reporting,"All of this combines to create a powerful incentive structure for authors to find certain results, and powerful incentives lead to biased decision making."
What you see is what you get? Enhancing methodological transparency in management research,Review,Incomplete or biased reporting,"Concerns about lack of reproducibility are not entirely surprising considering the relative lack of methodological transparency about the process of conducting empirical research that eventually leads to a published article (Banks et al., 2016a; Bedeian, Taylor, & Miller, 2010; John, Loewenstein, & Prelec, 2012; O’Boyle, Banks, & Gonzalez-Mul´e, 2017; Schwab & Starbuck, 2017; Simmons, Nelson, & Simonsohn, 2011; Wicherts et al., 2011; Wigboldus & Dotsch, 2016)."
What you see is what you get? Enhancing methodological transparency in management research,Review,Incomplete or biased reporting,"For example, consider the possible requirement that authors state whether they tested for outliers, how outliers were handled, and implications of these decisions for a study’s results (Aguinis et al., 2013). This actionable and rather easy to implement manuscript submission requirement can switch an author’s expected outcome from “dropping outliers without mentioning it will make my results look better, which likely enhances my chances of publishing” to “explaining how I dealt with outliers is required if I am to publish my paper—not doingso will result inmypaper beingdesk-rejected.”"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"What is known as the ‘‘?le-drawer problem’’ is very common: scienti?c studies with negative or nil-results often remain unpublished (Rosenthal, 1979; Rothstein, Sutton, & Borenstein, 2005)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"The root of the problem is the publication bias, caused by journals seeking theoretical novelty with empirical con?rmation, in combination with counterproductive university level career incentives focused on publications in a limited number of journals (for a recent summary, see van Witteloostuijn, 2016)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"Particularly, the focus onp-values leads to publication bias. It has always been the case that journals have an interest in publishing interesting results – i.e., signi?cant estimates – and not noise (to paraphrase Fisher), but the introduction of the publish-or-perish culture appears to have increased the publication bias."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"The publication bias arises from two practices. First, papers reporting signi?cant relationships are more likely to be selected for publication in journals, leading to a bias towards tests rejecting the null hypothesis."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"Second, authors ‘?ne-tune’ their regression analysis to turn marginally nonsigni?cant relations (those just above p= .01, p = .05 or p= .10) to signi?cant relations (i.e., just below these thresholds), which causes an in?ation of signi?cance levels in (published and unpublished) empirical tests."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"Scholars A and B ?nd no signi?cant results; they quickly move to other topics because ‘not statistically signi?cant’ will not be published in top management journals. Scholar C ?nds a result signi?cant at p\.05 level, which gets published in a high-impact outlet on the basis of which s/he receives tenure. The published result is treated as scienti?cally proven, and not challenged. Yet the actual evidence is that two out of three studies did not ?nd a signi?cant effect – and no one knows how many regressions scholar C ran in addition to the one with the signi?cant effect. This problem is not unique for nonexperimental ?eld work; experimental study designs are not immune to p-hacking either, as researchers may well stop their experiments once analysis yields a signi?cant p-value."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"In practice, scholars often conduct many tests, and develop their theory ex post but present it as if the theory had been developed ?rst."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Incomplete or biased reporting,"The issue with HARKing is that we have no knowledge of the many nulls and negatives that were found but not reported along the way, and therefore readers cannot be sure as to the true power of the statistical evidence."
On Replication in Communication Science,Editorial,Incomplete or biased reporting,"Nosek and Lakens (2014) noted that part of the problem with the current system of academic publishing is that editors and reviewers tend to be biased in favor offindings that reach traditional norms ofstatistical significance levels (p < .05). Ifthe analysis ofthe data from a study produces a null effect at the normative .05 level, many editors and reviewers conclude that the result is not interesting or worse, that the researchers must have made some error that resulted in the null finding, even though, as Rosnowand Rosenthal (1989) suggest “….surely, God loves the .06 nearly as much as the .05” (p. 1277)."
Reproducibility and Research Integrity,Note,Incomplete or biased reporting,"Irreproducibility, by contrast, may indicate a problem with any of the steps involved in the research such as, but not limited to, the experimental design, variability of biological materials (such as cells, tissues or animal or human subjects), data quality or integrity, statistical analysis, or study description (Landis et al 2012, Shamoo and Resnik 2015)."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Incomplete or biased reporting,"Third, where biases operate in the selection of outcome measures and data analysis practices, this increases the rates of spurious findings. These biases distort
the research record, reduce both replicability and verifiability, and can lead to bias in systematic reviews and meta-analyses."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Lack of detail in methods,"The origin and sourcing of cell lines is often inadequately described in publications, further challenging efforts to reproduce published data."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Lack of detail in methods,"Nonetheless, the cancer reproducibility project highlighted a wider problem: that experimental methods or environmental conditions are often not reported in sufficient detail to recreate the original set up accurately. Indeed, the most obvious conclusion was that many papers provide too little detail about their methods, according to Errington. As a result, replication teams have to devote many hours to chase down protocols and reagents, which often had been developed by students or post docs no longer with the team. The exposure of such discrepancies is
itself a positive result from the replication study, Errington asserted, and it has sparked efforts to make experiments more repeatable. “"
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Lack of detail in methods,"Our approach to reproducibility is to rely as much as possible on the artifacts provided by the authors themselves, i.e., their source code and the data used in the experiments. In theory, it should be possible to reproduce published results using only the technical descriptions in the papers. In reality, there are, however many tiny details regarding the implementation of the algorithms and the evaluation procedure, e.g., regarding data splitting, that can have an impact on the experiment outcomes."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Lack of detail in methods,"Excluding case studies or case series and models/modeling studies, in which a protocol would not be relevant, one (1.0% [0.1% to 6.0%]) ofthe 104 articles with empirical data included a link to a full study protocol."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Lack of detail in methods,"There were 31 (29.8% [21.4% to 39.7%]) articles that included supplemental materials, including methods sections, videos, tables, survey materials, and/or figures, either as a detailed appendix at the end ofthe article or online. However, none ofthe supplementary materials allowed for a reconstruction ofa full protocol."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Lack of detail in methods,"Although 10 (10.3% [5.3% to 18.6%]) articles had statements ofboth study novelty and some form ofreplication, 26 (26.8% [18.6% to 36.9%]) had no statement or an unclear statement in the abstract and/or introduction about whether the article presented novel findings or replication efforts."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Lack of detail in methods,"Among the eight articles classified as partial or full replication studies based on information provided in the abstract and/or introduction, four had enough information in the abstract alone to establish whether they were replication studies."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Lack of detail in methods,"Approximately half (55 of123, 44.7% [35.8% to 53.9%]) ofthe articles claiming to present some novel findings based on the abstract and/or introduction could be classified as novel according to the abstract only. Ofthe 10 articles that had statements ofboth study novelty and some form ofreplication, only four could be classified based on the abstract only."
MAKING REPLICATION MAINSTREAM,Article in Press,Lack of detail in methods,"An inability to specify the conditions needed to produce an effect is a serious impediment to scienti?c progress. The ability to specify a clear set of procedures that reliably elicit a predicted effect allows for independent veri?cation and provides the foundation for practical applications and studies that extend the original result. For a discovery to be counted as scienti?c, it should be accompanied by a description of the procedure that led to the discovery so that others can replicate it. Several authors have lamented the lack of procedural speci?city in many psychology articles. They call for more detailed descriptions of experiments, such that the conditions under which an effect is expected to replicate are speci?ed (Fabrigar & Wegener 2016; Simons et al., 2017)."
MAKING REPLICATION MAINSTREAM,Article in Press,Lack of detail in methods,"Likewise, it should be possible to specify the skills needed to conduct a particular study to produce a particular effect. It might be impossible to prespecify all such conditions and required experimenter skills, but in cases where a replication attempt fails to obtain the original result, claims of context effects or limited skills of the experimenter should be proposed as testable hypotheses that can be followed up with future work."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Lack of detail in methods,methods that are not precisely described so that results cannot be reproduced
Reproducible Research A Retrospective,"Report, policy document or website",Lack of detail in methods,"Rapid advances in computing technology have led to large-scale and high-throughput data collection coupled with the creation and implementation of complex statistical algorithms for data analysis. In the past, it might have su?ced to describe the data collection and analysis using a few key words and high-level language. However, with today’s computing-intensive research, the lack of details about the data analysis in particular can make it impossible to re-create any of the results presented in a paper."
Original article experimental design in ocean acidification research: Problems and solutions,Article,Lack of detail in methods,"Furthermore, 21% of studies did not provide any details of experimental design."
Original article experimental design in ocean acidification research: Problems and solutions,Article,Lack of detail in methods,"Around half (57%) of the studies did not present suf?cient details on the design type to determine the precise design, and 34% of all studies total did not provide suf?cient details to determine whether it was an “A” type or “B” type design."
Original article experimental design in ocean acidification research: Problems and solutions,Article,Lack of detail in methods,"Many studies did not report important methods, such as how treatments were created and the number of replicates ofeach treatment."
How we can make ecotoxicology more valuable to environmental protection,Note,Lack of detail in methods,"Overall, the reporting of both the methodology and results in ecotoxicity studies appears to be incomplete and inadequate (Ågerstrand et al., 2011a, 2011b, 2014). This can decrease the likelihood that studies are cited by other authors, or used for regulatory purposes (ECA, 2012). Examples of missing or insuf?ciently reported aspects include the types andperformance ofcontrols, analytical methods and exposure con?rmation, test system design, information about statistical evaluations and statistical power, and presence ofpossible confounding factors. As a reader ofa peer-reviewed publication it can be challenging to decide whether missing information is due to insuf?cient reporting or inadequate design and performance of the experiment. Regardless of the cause, missing information decreases the value of ecotoxicology publications andmay lead to a paper being omitted fromsubsequent interpretative work."
What you see is what you get? Enhancing methodological transparency in management research,Review,Lack of detail in methods,"Yet, as Aguinis et al. (2013) found, many authors made generic statements, such as “outliers were eliminated from the sample,” without offering details on how and why they made such a decision."
"Data management plans, the missing perspective",Note,DMPs are not regularly updated by researchers,"Lack of maintenance of the data management plan documentation through the active phase of a study risks lack of documentation to support research results, thus, status of data management plan maintenance after award should be an important consideration of research funders."
A manifesto for reproducible science,Review,"Pre-registrations do not guarantee usability, quality, or complete reporting","Improving the quality of reporting. Pre-registration will improve discoverability of research, but discoverability does not guarantee usability."
A manifesto for reproducible science,Review,"Pre-registrations do not guarantee usability, quality, or complete reporting","Even with pre-registration of clinical trials, one study observed that just 13% of trials published outcomes completely consistent with the pre-registered commitments. Most publications of the trials did not report pre-registered outcomes and added new outcomes that were not part of the registered design (see www.COMPare-trials.org)."
A manifesto for reproducible science,Review,"Pre-registrations do not guarantee usability, quality, or complete reporting","Franco and colleagues observed similar findings in psychology; using protocol pre-registrations and public data from the Time-sharing Experiments for the Social Sciences project (http://www.tessexperiments.org/), they found that 40% of published reports failed to mention one or more of the experimental conditions of the experiments, and approximately 70% of published reports failed to mention one or more of the outcome measures included in the study. Moreover, outcome measures that were not included were much more likely to be negative results and associated with smaller effect sizes than outcome measures that were included."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registries do not work with all study type or disciplines,"In addition to offering researchers many bene?ts, particularly with respect to issues posing threats to research quality, Registered Reports could have some perceived weaknesses. One of the primary concerns is that by only following registered protocols, researchers would be limited to hypothesis testing rather than exploration and discovery (e.g., Goldin-Meadow, 2016a). However, Registered Reports certainly do permit the reporting of exploratory or serendipitous ?ndings (Lindsay, Simons, & Lilienfeld, 2016) in a section clearly labeled as exploratory analyses."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registries do not work with all study type or disciplines,"Finally, there is a concern that Registered Reports might be only relevant to particular epistemological or methodological approaches. Indeed, in the development of Registered Reports, Chambers noted that Registered Reports are not applicable to all research approaches and are not intended to replace various other forms of inquiry (Chambers, 2013). Nonetheless, given that Registered Reports may be perceived as most easily accommodating certain types of studies, such as short-term laboratory research, the high value placed upon Registered Reports might inadvertently and undeservedly have the effect of “marginalizing studies for which preregistration is less ?tting” (Goldin-Meadow, 2016b, p. 14). Registered Reports at Language Learning were developed to be as inclusive of different research approaches as possible. For example, there is no reason why a study with observational or interview data, a long-term design, or a naturalistic context could not be submitted as a registered manuscript. Critically, any study where at least some of the methods and analyses can be predetermined is open to registered submission. We certainly acknowledge, however, that Registered Reports are not applicable or desirable for all epistemologies."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Some authors fear scooping when publishing registered reports,"Another concern about Registered Reports is that if protocols, which include materials and procedures, are publicly registered prior to data collection, researchers who are not associated with the approved protocol may take the materials, run the study, and publish the results before the report is completed; that is, researchers could be scooped. This concern is easily addressed by having the journal itself hold the registered manuscript and protocols before the second-stage review, which is the approach implemented at Language Learning. However, in the spirit of open science, just as Language Learning encourages the sharing ofmaterials and data, we encourage public registration. Again, even with this route, the concern about being scooped is easily addressed, as embargo dates can be set to release protocols to coincide with, for example, ?nal publication of the article (e.g., see http://help.osf.io/m/registrations/l/524205register-your-project)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Pre-registration allows for embargo, which not everyone knows","I am worried that someone will steal my project idea. No reason to worry. You can embargo your plans, and they will only become publicly visible later."
The challenges of replication,Editorial,The rigidity of registered reports can also be a problem,"The first five Replication Studies have also highlighted a potentially serious shortcoming of the Registered Report/Replication Study approach. The practice of specifying in advance precisely which experiments will be done, down to numbers of cells and replicates, is a strength because it avoids the possibility of biasing outcomes by mid-course changes in experiment. However, it has also proved to be a weakness in some cases because it has prevented experiments from being redone in different ways when the results were uninterpretable. This happened in a number of cases where control tumors grew with different kinetics than in the original studies despite attempts to use the same cells, same cell doses and same recipient mice."
The challenges of replication,Editorial,The rigidity of registered reports can also be a problem,"However, restricting the scientists performing the replications to the experimental designs in the Registered Report meant that, in general, they were not able to redo the experiments with different cell doses to achieve more interpretable kinetics. This has been particularly problematic with tumor formation assays in vivo, in which variability is often high and results depend upon the experience of the investigator."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,The rigidity of registered reports can also be a problem,"Prespecified protocols may not be feasible in certain scientific fields or for all types ofinvestigation, in which hypothesis-generated experiments are the norm, and trying to enforce them in such applications would be spurious and could cause investigators to be dishonest about their real intentions. However, when hypotheses and ideas can be prespecified, detailed analytical plans can improve the credibility ofresearch."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,The rigidity of registered reports can also be a problem,"Academics likely have a range of scientific endeavours, and RRs may suit the timescale or more confirmatory nature of some studies, but not necessarily others in their portfolio. For example, the timing of final year undergraduate student projects may be difficult to fit into a RR format."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Need for better reporting standards for authors and peer reviewers,"Although some journals have developed checklists of required information that must accompany each article, most of these checklists fall short in regard to reporting requirements recommended for preclinical studies."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Need for better reporting standards for authors and peer reviewers,"While the TIDieR checklist provides useful guidance for how to report some aspects of an exercise intervention, further precise information about the type of exercise, as well as details such as its dosage, intensity and frequency, and whether or not it requires supervision or individualisation, are also required to fully understand the intervention and how to replicate it."
On the issue of transparency and reproducibility in nanomedicine,Letter,Need for better reporting standards for authors and peer reviewers,"Implementing MIRIBEL may standardize the way the manuscripts are written and the formats to present the data, which is an indisputable bonus. However, it will not necessarily improve data reproducibility or have other expected benefits of its implementation."
How we can make ecotoxicology more valuable to environmental protection,Note,Need for better reporting standards for authors and peer reviewers,This exercise suggests that guidance on publication standards provided by peer-reviewed journals requires improvement.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Reporting guidelines are not common to all research areas,"For larger, multidisciplinary journals publishing many types of research, identifying and enforcing the growing number of relevant reporting guidelines, which can vary from paper to paper that is submitted, is inherently more complex and time-consuming."
A manifesto for reproducible science,Review,Reporting guidelines can be bureaucratic or overly restrictive,Reporting guidelines are easily perceived by researchers as bureaucratic exercises rather than means of improving research and reporting.
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Reporting guidelines can be bureaucratic or overly restrictive,"However, 25% considered MISEV2014 to be too restrictive, whereas 16% said it was not stringent enough (Figure 2(b)).Fourper cent thought that MISEV2014 was an unnecessary imposition on the field. In conclusion, most survey participants are in favour of standards, with a majority of respondents supporting MISEV2014. It is clear, though, that sizeable portions ofthe ISEV constituency might favour updates and changes to MISEV2014."
On the issue of transparency and reproducibility in nanomedicine,Letter,Reporting guidelines can be bureaucratic or overly restrictive,There is an inevitable trade-off between having a fully comprehensive and potentially burdensome checklist for all areas of bio–nano research and one that is less ambitious and only covers specific areas.
A manifesto for reproducible science,Review,Uncertainty cannot be fully removed by using reporting guidelines,Improved reporting may be insufficient on its own to maximize research quality.
A manifesto for reproducible science,Review,Uncertainty cannot be fully removed by using reporting guidelines,"The negative outcome is the empirical evidence that reporting guidelines may be necessary, but will not alone be sufficient, to address reporting biases. The impact of guidelines and how best to optimize their use and impact will be best assessed by randomized trials."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Uncertainty cannot be fully removed by using reporting guidelines,"Standardization through MISEV updates is important, but of course uncertainty will remain that cannot be removed through minimal requirements alone. On many specific questions in EV research, including, for example, some quite basic matters of pre-analytical variables, the jury is still out."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Uncertainty cannot be fully removed by using reporting guidelines,"A guideline method (e.g., Organisation for Economic Co-operation and Development [OECD], International Organization for Standardization [ISO]) or modi?ed guideline used? Use of a guideline method (OECD, ISO, USEPA, or comparable) does not necessary re?ect the reliability of a study, and it should therefore never be a critical criterion. A guideline study may be unreliable if there are ?aws in the design, conduct, and/or (statistical) interpretation or if results give rise to doubt."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Reporting guidelines have been more prevalent in journals with high impact factors,"The positive impact of endorsement of reporting guidelines by journals has however been limited (Percie du Sert et al. 2018), in part due to reporting guidelines often being implemented by passive endorsement on journal websites (including them in information for authors). Some journals, such as the medical journals BMJ and PLOS Medicine, have mandated the provision of certain completed reporting guidelines, such as CONSORT, as a condition of submitting manuscripts. Endorsement and implementation of reporting guidelines has been more prevalent in journals with higher impact factors (Shamseer et al. 2016)."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Researchers do not always report conflicts of interest,"Of the 439 eligible articles, 184 listed funding sources at the full text level. Nearly two-thirds (115 of 184, 62.5% [55.0% to 69.4%]) included some funding information under the “Publication type, MeSH terms, Secondary source ID” tab on PubMed (e.g., “Research Support, NonUS Gov’t”). There were 39 (21.2% [15.7% to 28.0%]) additional articles in which PubMed provided at least one specific funding source (i.e., a specific grant number). None ofthe articles disclosed competing interests under a “Conflict ofinterest statement” tab on PubMed."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Researchers do not always report conflicts of interest,"While the proportion of articles with any information related to potential conflicts ofinterest disclosures has increased rather steadily over time, likely in response to strengthening ofbiomedical journal disclosure policies, the proportion of articles reporting no conflicts ofinterest has remained fairly constant and may underestimate the true prevalence ofconflicts in biomedical research."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Researchers do not always report conflicts of interest,"While disclosure ofconflicts ofinterest has become more common as a result ofthe uniform forms developed by the ICMJE, which is supported by hundreds ofbiomedical journals, it is possible that authors are not complying with the guidelines or are unaware ofpotential conflicts that can impact the design, conduct, and analyses ofstudies."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Code, data and methods are not always available or complete","but the specific code researchers write for a particular piece of work is often left unpublished, hindering reproducibility."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Code, data and methods are not always available or complete","Some authors may describe computational methods in prose, as part of a general description of research methods. But human language lacks the precision of code, and reproducing such methods is not as quick or as reliable as it should be."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Code, data and methods are not always available or complete","However, whereas the direct output in most shells can only be text, notebooks can include rich output such as plots, formatted mathematical equations, and even interactive controls and graphics. Prose text can be interleaved with the code and output in a notebook to explain computational narrative."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,"Code, data and methods are not always available or complete","Besides issues related to the baselines, an additional challenge is that researchers use various types of datasets, evaluation protocols, performance measures, and data preprocessing steps, which makes it difcult to conclude which method is the best across diferent application scenarios. This is in particular problematic when source code and data are not shared. While we observe an increasing trend that researchers publish the source code of their algorithms, this is not the common rule today even for top-level publication outlets."
On the Reproducibility of Psychological Science,Article,"Code, data and methods are not always available or complete","The ?rst concerns the issue of training—from simple parameter tuning (e.g., for BM25) to a complete learning-to-rank setup. In particular, the latter would provide useful baselines for researchers pushing the state of the art in retrieval models. We have not yet converged on a methodology for including “trained” models that is not overly burdensome for developers. For example, would the developers also need to include their training code? And would the scripts need to train the models from scratch? Intuitively, the answer seems to be “yes” to both, but asking developers to contribute code that accomplishes all of this seems overly demanding."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,"Code, data and methods are not always available or complete","Software details (application framework used for analysis, programming language, and version) were not reported in detail in 16 studies."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,"Code, data and methods are not always available or complete","Furthermore, none ofthe articles mentioned any sharing ofscripts/code."
"Most computational hydrology is not reproducible, so is it really science?",Note,"Code, data and methods are not always available or complete","Abstract Reproducibility is a foundational principle in scienti?c research. Yet in computational hydrology the code and data that actually produces published results are not regularly made available, inhibiting the ability of the community to reproduce and verify previous ?ndings."
"Most computational hydrology is not reproducible, so is it really science?",Note,"Code, data and methods are not always available or complete","The prominence of computational research across scienti?c disciplines—from big data analysis in genomic research to computational modeling in climate science—has brought increased focus on the reproducibility issue. This is because the full code and work?ow used to produce published scienti?c ?ndings is typically not made available, thus inhibiting attempts to verify the provenance of published results"
What you see is what you get? Enhancing methodological transparency in management research,Review,"Code, data and methods are not always available or complete","Clearly noting the software employed and making available the syntax used to carry out data analysis facilitates our understanding of how the assumptions of the analytical approach affected results and conclusions (Freese, 2007a; Waldman & Lilienfeld, 2016). For example, there are multiple scripts and packages available within the R software to impute missing data."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Code, data and methods are not always available or complete","In sum, issues such as the initial reporting ofmethods, results, and analysis; the availability of the initial materials and data; and the resources needed may all reduce the likelihood, quality, or usefulness of replication (even when a replication is clearly warranted)."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"Code, data and methods are not always available or complete","A second potential reason for the low amount and, arguably, the low validity ofreplication research is the very poor availability of materials. For example, Derrick (2016) reported that only 17% of research materials were available within published articles or online sources, and Marsden, Thompson, and Plonsky (in press) found that just 27% were available. This means that future researchers wishing to systematically extend prior studies must either recreate materials, thus introducing unplanned heterogeneity, or work directly with the initial study’s authors, thus introducing potential bias."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"Code, data and methods are not always available or complete","A third problem that impedes high-quality replication research is the very poor availability of raw data, as discussed by Larson-Hall and Plonsky (2015), which likely affects the quality of the research itself (Wicherts, Bakker, & Molenaar, 2011)."
A manifesto for reproducible science,Review,"Code, data and methods are not always available or complete","Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant. In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,"Code, data and methods are not always available or complete","A survey of a random sample of biomedical articles published from 2000–2014 suggested that the literature lacked transparency in important dimensions and that reproducibility was not valued appropriately. For instance, protocols and raw data were not directly available, and the majority ofstudies did not disclose funding or potential conflicts of interest."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,"Code, data and methods are not always available or complete","Among the 590 articles published between 2000 and 2017 in eligible research fields directly related to biomedicine, 520 were non-PMCOA articles. Among the 520 non-PMCOA articles, 81 (15.6% [12.6% to 19.1%]) had a PMCID, thus a PDF is available for each individually. However, full text XML (i.e., Extensible Markup Language) for these articles cannot be downloaded in bulk. Therefore, 439 articles did not have a full text available in PubMed."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,"Code, data and methods are not always available or complete","Among the 263 articles in which protocol or data sharing would be relevant (excluding articles without some form ofempirical data, model/modeling studies, and case studies or case series), there was one systematic review with a registered protocol on PROSPERO that included a University ofYork Centre for Reviews and Dissemination (CRD) number at the abstract level."
Reproducible Research A Retrospective,"Report, policy document or website","Code, data and methods are not always available or complete","While seemingly a straightforward concept, reproducibility of analyses is typically thwarted by the lack of availability of the data and computer code that were used in the analyses."
About ReproZip,"Report, policy document or website","Code, data and methods are not always available or complete","For reviewers, even with a compendium in their hands, it may be hard to reproduce the results. There may be no instructions about how to execute the code and explore it further; the experiment may not run on his operating system; there may be missing libraries; library versions may be different; and several issues may arise while trying to install all the required dependencies, a problem colloquially known as dependency hell."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","The interpretation of experiments often relies on probabilities (P-values) of < 0.05 as the gold standard test for statistical significance, which creates a sharp but somewhat arbitrary cut-off."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Because a small P-value could result from random variation alone, Fisher (1937) wrote that “no isolated experiment, however significant in itself, can suffice for the experimental demonstration of any natural phenomenon.” And Boring (1919) said a century ago, “scientific generalization is a broader question than mathematical description.” Yet today we still indoctrinate students with methods that claim to produce scientific generalizations from mathematical descriptions of isolated studies."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","A core problem is that both scientists and the public confound statistics with reality. But statistical inference is a thought experiment, describing the predictive performance of models about reality."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","But the P-value itself is not supposed to be “reliable” in the sense of staying put (Greenland 2019a). Its fickleness indicates variation in the data from sample to sample. If sample averages vary among samples, then P-values will vary as well, because they are calculated from sample averages. And we don’t usually take a single sample average and announce it to be the truth. But if instead of simply reporting the P-value, we engage in “dichotomania” (Greenland 2017) and use it to decide which hypothesis is wrong and which is right, such scientifically destructive behavior is our fault, even if socially encouraged; it is not the fault of the P-value."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","But what comes next? There are countless possibilities. The most common proposal is to replace hypothesis tests with interval estimates. While doing so is helpful for sophisticated researchers, it has not reduced what we see as the core psychological problems—which is unsurprising, because the classical confidence interval is nothing more than a summary of dichotomized hypothesis tests. Consider that a 95% confidence interval encompasses a range of hypotheses (effect sizes) that have a P-value exceeding 0.05. Instead of talking about hypothetical coverage of the true value by such intervals, which will fail under various assumption violations, we can think of the confidence interval as a “compatibility interval” (Greenland 2019a, 2019b), showing effect sizes most compatible with the data according to their P-values, under the model used to compute the interval."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Not use the words “significant” or “confidence” to describe scientific results, as they imply an inappropriate level of certainty based on an arbitrary criterion, and have produced far too much confusion between statistical, scientific, and policy meanings."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","We do not suggest to completely abandon inference from our data to a larger population (although the title ofa preprint ofthis paper was “Abandon statistical inference”; Amrhein, Trafimow, and Greenland 2018). But we say this inference must be scientific rather than statistical, even if we use inferential statistics. Because all statistical methods require subjective choices (Gelman and Hennig 2017), there is no objective decision machine for automated scientific inference; it must be we who make the inference, and claims about a larger population will always be uncertain."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Some promising avenues to resolve the current replication crisis could include the preregistration ofstudy objectives, compulsory prestudy power calculations, enforcing minimally required power levels, raising the statistical significance threshold to p < 0.001 ifNHST is used, publishing negative findings once study design and power levels justify this, and using Bayesian analysis to provide probabilities for both the null and alternative hypotheses."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","A major problem is that we tend to take small pvalues at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability ofresearch, because they are hardly replicable even ifan alternative hypothesis is true. Also significance (p?0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third ofthe cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","No outright fraud, no technical fault or bad experimental design are necessary to render a study irreproducible; it is sufficient that we report results preferentially if they cross a threshold of significance."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Calculations of power needed to reject the null hypothesis are conventional, but they can mislead because they assume that no problem will occur during the study, no other evidence will be available to inform decision makers, and that the arbitrary ? 0·05 strikes the proper balance between false acceptance or rejection of the null hypothesis. These conditions hardly ever exist. Moreover, a study with high power to reject the null hypothesis that fails to reject it at the conventional (5%) ?-error-level might still support the alternative hypothesis better than it does the null."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted",The quest for adequate statistical power might lead researchers to choose outcome measures that are clinically trivial or scientifically irrelevant.
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","The Klimisch method has also been criticized for being biased toward interests of industry and for promoting use of guideline studies performed according to good laboratory practices (GLP). Altogether, this could result in a situation in which risk assessors arrive at different conclusions regarding the reliability and relevance of a study and whether it could be included in a speci?c regulatory process."
How Bayes factors change scientific practice,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","A Bayes factor is a form of statistical inference in which one model, say H1, is pitted against another, say H0. Both models need to be specified, even if in a default way. Significance testing (using only the p-value for inference, as per Fisher, 1935) involves setting up a model for H0 alone—and yet is typically still used to pit H0 against H1. I will argue that significance testing is in this way flawed, with harmful consequences for the practice of science (Wagenmakers, 2007)."
How Bayes factors change scientific practice,Article,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted",The key problem created by the asymmetry of the p-value is that significance testing per se (i.e. inference by use of p-values) cannot provide evidence for the null hypothesis.
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Contrary to the fact that in Figure 1 all 10,000 true H0 and 10,000 true H1 samples were simulated from identical H0 and H1 distributions, the t scores and the associated p-values re?ect a dramatic spread. That is, p-values are best viewed as random variables which can take on a range of values depending on the actual data (Sterling, 1959; Murdoch et al., 2008). Consequently, it is impossible to tell from the outcome of a single (published) experiment delivering a statistically signi?cant result whether a true e?ect exist."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","Simply put, the p-value is pretty much the only thing that NHST computes but scientists usually would like to know the probability of their theory being true or false in light of their data (Pollard and Richardson, 1987; Goodman, 1993; Jaynes, 2003; Wagenmakers, 2007). That is, researchers are interested in the post-experimental probability of H0 and H1. Most probably, for the reason that researchers do not get what they really want to see."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted",Ioannidis (2005) has shown that most published research ?ndings relying on NHST are likely to be false.
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,"Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","NHST Is Unsuitable for Large Datasets In consequence of the recent ‘big data’ revolution access to large databases has increased dramatically potentially increasing power tremendously (though, large data sets with many variables are still relatively rare in neuroscience research). However, NHST leads to worse inference with large databases than with smaller ones (Meehl, 1967; Khoury and Ioannidis, 2014). This is due to how NHST tests statistics are computed, the properties of real data and to the lack of specifying data predicted by H1 (Bruns and Ioannidis, 2016)."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website","Current standards, particularly statistical ones (e.g., p<0.05), are problematic and misinterpreted","P-values alone provide a weak basis for inference, and no information regarding whether a result is biologically significant or clinically meaningful."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,"The ""decline effect"" and the ways findings are reported on affects public perceptions","Also the decline effect is well known and documented and explains how initially strong results of all kinds tend to diminish over time (de Bruin & Della Sala, 2015). Likewise, the examples of exaggerations in the presentation of scientific findings discussed in the previous sections are certainly not isolated cases. In medicine and particularly in genetics almost every discovery is hailed as a “breakthrough” (or, since this word has already been devalued through frequent use, “major breakthrough”). If only a part of this were true, we would be living by now in a world free of diseases."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,High standardisation is necessary for reproducibility may actually reduce external validity,"Another related issue is the high level of standardization to make results as generally valid and reproducible as possible. But, as Wurbel emphasized, this can actually have the opposite effect. “The standard approach to evidence generation in preclinical animal research are single-laboratory studies conducted under highly standardized conditions, often for both the genotype of the animals and the conditions under which they are reared, housed and tested”, he said. “Because of this, you can never know for sure whether a study outcome has or hasn’t got external validity."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,High standardisation is necessary for reproducibility may actually reduce external validity,He therefore advocated the development of pathogen-free stocks from wildtrapped progenitors for study of ageing and late-life pathophysiology. It has now inspired calls for greater genetic diversity among laboratory rodents for preclinical research in drug development and ageing.
"Data management plans, the missing perspective",Note,DMP requirements vary across funders and don't consistently cover all the 'operations to be performed on data for a study',"They classi?ed each DMP requirement document according to seventeen criteria that they adapted from those previously reported by the Digital Curation Centre (DCC) focused on data sharing, curation and preservation. In summary, they found that no single policy addressed all criteria and that data policies were missing a signi?cant number of elements. "
"Data management plans, the missing perspective",Note,DMP requirements vary across funders and don't consistently cover all the 'operations to be performed on data for a study',"Soon most grants funded by Federal agencies in the United States will require some form of data management plan. A tool has been collaboratively developed to facilitate authoring Data Management Plans for a broad range of funders, Data Management Plan creation tool (https://dmptool.org) and has been adopted by more than 100 institutions. The DMP tool, however appears largely focused on creation of DMPs to accompany funding applications rather than a DMP that comprehensively speci?es all operations to be performed on data for a study, serves as a reference and job aid throughout the study, and results in explanatory documentation to be archived with study data."
"Data management plans, the missing perspective",Note,DMP requirements vary across funders and don't consistently cover all the 'operations to be performed on data for a study',"We offer several possible reasons for the lack of requirement for DMPs following award. First, most doctoral programs do not include training in data management, thus, scientists in leadership positions in funding institutions may not be aware of the DMP as a tool in data related quality assurance and control. Second, the differences in language and scope of data management tasks between the library science community and the therapeutic development and information technology communities indicates that data management-related terminology is not consistent between disciplines and may also play a role. For example, while DMPs are seen as the mechanism for data related quality assurance and control by some, others may not conceptually group the tasks associated with data collection, processing and storage as data management and may refer to or provide for them in some other way."
"Data management plans, the missing perspective",Note,DMP requirements vary across funders and don't consistently cover all the 'operations to be performed on data for a study',"The low percentage of funders requiring DMPs versus data sharing plans indicates a greater emphasis on sharing and reuse of data than on data collection and processing. This disproportionate emphasis should be reexamined and equal weight given to upstream processes that determine data quality and support research results. The least-required items in a data management plan were aspects about data collection and processing. This is unfortunate, because data collection including the original observation and processing are not only large determinants of data quality, they are the determinants for which the opportunity to intervene is lost as time moves on."
On the issue of transparency and reproducibility in nanomedicine,Letter,Researchers don't see the immediate benefit of good lab practice reports or studies,"GLP studies take longer to design, schedule and complete, and they are unavoidably more expensive than their comparable non-GLP counterparts. Following the GLP standards ensures the results reproducibility, as long as there is no change in the source of reagent or qualification/training of staff conducting such studies. If any change needs to occur, GLP requires re-validation or crossvalidation. At this point, most academic labs have neither the infrastructure nor adequate budgets to support GLP studies."
Reproducibility Crisis: Are We Ignoring Reaction Norms?,Letter,Standardisation is not always possible - 'standardisation fallacy',"This is mirrored by the ‘standardisation fallacy’, the erroneous belief that reproducibility can be improved through ever more rigorous standardisation. Because many environmental factors resist standardisation between laboratories, animals within laboratories will be more homogeneous than animals between laboratories. Increasingly rigorous standardisation will therefore produce results that are increasingly distinct between laboratories and hence less reproducible. Thus, instead of trying to spirit biological variation away through standardisation, researchers should eventually start to embrace it in view of improving the external validity and hence the reproducibility of their results."
On the issue of transparency and reproducibility in nanomedicine,Letter,Standardisation is not always possible - 'standardisation fallacy',Achieving universal standardization practices for nanomaterials is not feasible as strict mandatory requirements may slow down basic research efficiency.
Reproducible Research A Retrospective,"Report, policy document or website",Encouraging data sharing won't prevent fraud,"One could hypothesize that if an investigator knew in advance that the data and the code would be publicly available for scrutiny, then they would take the extra e?ort to make sure that the analyses were properly done. Perhaps if Potti et al. had been forced to make their code publicly available, they would have checked it ?rst. In the case of Potti et al. we now know that requiring reproducibility or even just code sharing would not have made much di?erence. Reporting done by The Cancer Letter showed de?nitively that the investigators were aware of numerous statistical and coding errors with the analysis but did not think they were serious problems. Rather, they were considered “di?erences of opinion”. The notion that requiring reproducibility can lead to improved data analyses relies on the critical assumption that the investigators are able to recognize what is an error in the ?rst place. If they do recognize the error and hide it, then that is fraud. If they do not recognize the error and publish it anyway, then that is at best careless. However, in both cases, forcing the data and code to be published would not have made any di?erence."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Authors may be unwilling to share data,"Some authors are happy to share their data and code, but in many other cases the crucial information—the data itself—are not available. For example, Nieuwland et al. (2018) tried but failed to obtain the data for the published result (DeLong, Urbach, & Kutas, 2005) that they attempted to replicate. Many researchers have generously released their data to us in connection with the present and other replication attempts. But attempts to obtain data from published studies are often unsuccessful. Wicherts, Borsboom, Kats, and Molenaar (2006) report an attempt to obtain data from 141 articles from major psychology journals, which had a total of 249 experiments. Of these, 73% of the data were not released. Wicherts and colleagues report that this is approximately the same non-response rate as in 1962. A common objection we hear is that anyone could defeat the purpose of pre-registration by first collecting the data and then depositing a fake pre-registration. However, this would just be scientific fraud; pre-registration is not designed to solve that problem."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Authors may be unwilling to share data,"Approximately 20% ofthe articles in which data sharing would have been pertinent included a statement related to data sharing. Although one article disclosed that the data were available “on reasonable request,” which could be a result ofdata sharing statements required by the journal, this does not guarantee that the raw data would be made available."
Reproducibility literature analysis - a federal information professional perspective,Article,Authors may be unwilling to share data,"For example, lack of ‘data sharing’ is cited as a hindrance to reproducibility. Others cite ‘data sharing’ as a potential solution. Again, where applicable, we append other terms to these definitions that are related to those we highlight here. For a full list of the selected solutions and their definitions, review the related data file available at https://doi.org/10.18434/M32150."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Harking,"The above approach ofexploratory data analysis means that a fairly large number ofhypotheses get tested in a very short time (i.e. without careful thinking about speci?c hypotheses considered plausible) and this comes with a high risk of drawing a false-positive conclusion if we only report on the subset of signi?cant predictors. In fact, such exploratory analysis could be seen as an act of generating hypotheses rather than as an act of testing hypotheses, because you only start thinking about the respective hypothesis once you have discovered a signi?cant association. This approach is not wrong per se, as long as you are aware and honest about the fact that the hypothesis was derived from the data."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Harking,"The problem starts where researchers fail to acknowledge this. The psychologist Norbert Kerr called this ‘HARKing’ (hypothesising after the results are known; Kerr, 1998)."
MAKING REPLICATION MAINSTREAM,Article in Press,Harking,"Researchers also form hypotheses after having examined the data, a practice called HARKing (hypothesizing after the results are known; Kerr 1998). When HARKing is undisclosed to readers of a paper, it might strike some researchers as deceptive. However, this strategy was once presented as the hallmark of sophisticated psychological writing (Bem 2003)."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Lack of integrity and questionable research practices,"A broader group of methodological concerns, ?tting under the broad banner of questionable research practices (Chambers, 2017), has also been raised, again across many disciplines and particularly those that rely heavily on null hypothesis signi?cance testing. One such practice is p hacking, which refers to testing more participants until a p value is achieved that is deemed to be statistically signi?cant or to applying various data elimination criteria and presenting only the one that leads to a statistically signi?cant result. Another such practice is known as HARKing (hypothesizing after results are known), where exploratory analyses are presented as if they were con?rmatory, thereby implying an unwarranted theoretical kudos and so presenting ?ndings with a level of con?dence that may not be as reproducible as inferred by the article’s argumentation. Although these practices may be common and not intentionally deceptive, they pose systematic challenges to the validity, reliability, and reproducibility ofresearch ?ndings (see similar arguments by Nosek, Ebersole, DeHaven, & Mellor, 2017)."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Lack of integrity and questionable research practices,"Frequently, the question is: where do sloppiness, wishful thinking, and, perhaps, morally innocent selectivity in data presentation transition into falsi?cation? A key element in making the distinction is whether there is intention to deceive, as opposed to errors, self-deception, or honest differences of opinion. It should be obvious that such distinctions are very dif?cult to make in individual cases. Those individuals required by circumstance and institutional role to make these dif?cult distinctions face an onerous task, since intention is often hard to assess, and careers typically hang in the balance."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Lack of integrity and questionable research practices,"There are many well documented and reported cases of scientists engaging in the willful fabrication and/or falsi?cation of data that makes its way to publication, only to be retracted after the duplicity is discovered. Such cases are not new in the history of science. In some cases, the extent and duration of such fabrication/falsi?cation is astounding. An individual committed to and highly skilled in such deception can make it dif?cult or impossible for colleagues, and certainly reviewers and journals, to identify the fraud. On the other hand, in many such cases retrospective analysis revealed signs that should have provided vigilant colleagues reason to suspect, well before publication, that all was not well with the perpetrator and the data."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Lack of integrity and questionable research practices,"Errors also result from research misconduct, including fraudulent or unethical research, and plagiarism."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Lack of integrity and questionable research practices,"Plagiarism and self-plagiarism are common forms of misconduct and common reasons for papers being retracted (Fang et al. 2012). In the last decade, many publishers have adopted plagiarism detection software, and some apply this systematically to all submissions."
Reproducibility literature analysis - a federal information professional perspective,Article,Lack of integrity and questionable research practices,"‘The National Science Foundation (2001) defined scientific misconduct as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research or in reporting research results. Such misconduct is committed intentionally, knowingly, or in disregard of accepted practices. Fabrication of data involves totally inventing a data set, while falsification refers to manipulation of equipment or changing data such that the research is not accurately represented in the research report’ (Stroebe, Postmes and Spears, 2012). [Related terms: trust; data quality]"
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Lack of integrity and questionable research practices,"There are also multiple reports of shocking cases of fraud, alarming rates of questionable research practices and foot-dragging from practitioners, editors, authors and institutions when dealing with retractions and corrections."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website",Lack of integrity and questionable research practices,"Across the research world, there is growing suspicion about sloppiness and outright misconduct in the scientific literature. The number of retractions of research papers has risen to around 1,400 a year, compared with about 40 at the turn of the millennium, notes Ivan Oransky, a journalist in New York City who co-founded the website Retraction Watch, which monitors and reports on retractions. In 2016, Elisabeth Bik, a microbiologist then at Stanford University in California, reported that around 4% of more than 20,000 biomedical papers she had examined contained inappropriately duplicated images. (Bik is now a full-time research-integrity consultant.) And last year, Bucci reported that about 6% of a sample of 1,364 papers he had looked at contained at least one instance of image manipulation."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Lack of integrity and questionable research practices,"ABA researchers have published studies that do not report positive effects, but the investigations have usually centered on questionable practices that have little or no credible empirical evidence to begin with, such as sensory-based interventions (Barton, Reichow, Schnitz, Smith, & Sherlock, 2015; Cox, Gast, Luscre, & Ayres, 2009; Losinski, Cook, Hirsch, & Sanders, 2017)."
What you see is what you get? Enhancing methodological transparency in management research,Review,Lack of integrity and questionable research practices,"We focus on the relative lack of methodological transparency because it masks outright fraudulent acts (as committed by, for example,Hunton&Rose, 2011and Stapel & Semin, 2007), serious errors (as committed by, for example, Min & Mitsuhashi, 2012; Walumbwa, Luthans, Avey, & Oke, 2011), and questionable research practices (as described by Banks, et al., 2016a)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Lack of integrity and questionable research practices,"Accumulating evidence suggests that authors actively engage in pushing signi?cance levels just below the magic threshold of p = .05, a phenomenon referred to as ‘p-hacking’ or ‘search for asterisks’ (Bettis, 2012; Brodeur et al., 2016). Similarly, some authors appear to engage in HARKing, which stands for Hypothesizing After the Results are Known (Bosco et al., 2016; Kerr, 1998). The problem of both practices is that the reported signi?cance levels are misleading because readers are given no information how many nulls and negatives ended up in the research dustbin along the way."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Lack of integrity and questionable research practices,"A recent upsurge of scandals such as Stapel’s data-fabricating misconduct in social psychology (New York Times, 2011) triggered a powerful movement toward changing the ways in which the scienti?c community has institutionalized practices that stimulate rather than discourage such behavior."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Lack of integrity and questionable research practices,"Scholars under pressure of ‘publish or perish’ face a slippery slope, moving from the subjectivity of ‘sloppy science’, to incomplete reporting that inhibits replication, to deliberate exclusion of key variables and/or observations, to manipulation of data, and to outright fabrication of data."
Reproducibility and Research Integrity,Note,Lack of integrity and questionable research practices,"Irreproducibility, by contrast, may indicate a problem with any of the steps involved in the research such as, but not limited to, the experimental design, variability of biological materials (such as cells, tissues or animal or human subjects), data quality or integrity, statistical analysis, or study description (Landis et al 2012, Shamoo and Resnik 2015)."
Reproducibility and Research Integrity,Note,Lack of integrity and questionable research practices,"Some of the irreproducibility in scientific research may be due to data fabrication or falsification (Shamoo 2013, 2016; Collins and Tabak 2014, Kornfeld and Titus 2016). Misconduct or suspected misconduct accounts for more than two-thirds of retractions (Fang et al 2012)."
Reproducibility and Research Integrity,Note,Lack of integrity and questionable research practices,"Reproducibility is not just a scientific issue; it is also an ethical one. When scientists cannot reproduce a research result, they may suspect data fabrication or falsification. In several well-known cases, reproducibility issues led to allegations of data fabrication or falsification. For example, in 1986, postdoctoral researcher Margot O’Toole accused her supervisor, Tufts University pathology assistant professor Thereza Imanishi-Kari, of fabricating and falsifying data in a National Institutes of Health (NIH)-funded study on using foreign genes to stimulate antibody production in mice, published in the journal Cell. O’Toole became suspicious of the research after she was unable to reproduce a key experiment conducted by Imanishi-Kari and found discrepancies between the data recorded in Imanishi-Kari’s laboratory notebooks and the data reported in the paper. The case made national headlines, in part, because Nobel Prizewinning molecular biologist David Baltimore was one of the coauthors on the paper, even though he was never implicated in the scandal. A Congressional committee headed by Rep. John Dingell discussed the case during its hearings on fraud in federally-funded research. In 1994, the Office of Research Integrity, which oversees NIH-funded research, found that Imanishi-Kari committed misconduct, but a federal appeals panel overturned this ruling in 1996 (Shamoo and Resnik 2015)."
Reproducibility and Research Integrity,Note,Lack of integrity and questionable research practices,"Adherence to some commonly recognized principles of responsible conduct of research (RCR) plays an important role in promoting reproducibility in science. One of the key pillars of RCR is that scientific records, including laboratory notebooks, protocols, and other documents, should describe one’s research in sufficient detail to allow others to reproduce it (Schreier et al 2006, Shamoo and Resnik 2015). Records should be accurate, thorough, clear, backed-up, signed, and dated. Failure to record a vital piece of information, such as a change in an experimental design, the pH of a solution, or the type of food fed to an animal, the time of year, can lead to problems with reproducibility (Buck 2015, National Institutes of Health 2016, Firestein 2016)."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Lack of integrity and questionable research practices,"Opaquely conducted research may contribute to a loss of trust in research findings, as well as bias in meta-analyses and systematic reviews."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Lack of integrity and questionable research practices,"Irreproducible research violates basics scientific principles and makes it impossible to detect coding errors. Unethically conducted research increases the likelihood of harming research subjects (through insufficient protocols), can harm the reputation of the IDB, and threatens the reputation, employment, and funding of project teams. Additionally, failure to comply with certain elements of transparency (registration), reproducibility (code and data sharing), and ethics (IRB approval) may prevent researchers from publishing in academic journals."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Lack of integrity and questionable research practices,"High rates of questionable research practices, increasing numbers of retractions, and several high-profile cases of scientific misconduct have increased the need scrutiny of scientific practices and the resulting body of work."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Lack of integrity and questionable research practices,"Unfortunately this flexibility introduces the potential for a range of questionable research practices. A survey of over 2000 scientist working in psychology indicated high rates of questionable research practices. These included rounding down p values favouring significance (22-23%), selectively reporting studies that ‘worked’ (46-50%), reporting an unexpected finding as being predicted at the start (27-35%), removing data after looking at the impact on the results (38-43%), deciding to collect more data after finding the results were non-significant (56-58%), and falsifying data (0.6-1.7%). These findings are supported by statistics in published papers, where a study showed that in 95% of papers with inaccurate statistics, p-values were rounded down to meet significance thresholds."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Method-shopping,"Crucial data-analysis decisions are often made after examining the data. For example, the same researcher will often include or exclude data on different criteria, so that it eventually passes the statistical significance filter. Sometimes, excluding or including a few data points can make the difference between significance and non-significance (Vasishth et al., 2013). Another example is region-of-interest selection in reading studies: researchers often change the region of interest from study to study or even within a study, driven exclusively by the search for significance (an example is discussed in Vasishth & Nicenboim, 2016). Another common approach is to run the study, check for significance, then either run more participants if significance is desired but not reached, or stop collecting data if a null result is desired. These decisions are often not reported in the published paper. Pre-registration would remove these degrees of freedom and thereby ensure a clear separation between confirmatory and exploratory analyses (De Groot, 1956/2014)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Method-shopping,"There is evidence that researchers may engage in ‘method shopping’ for techniques that provide greater sensitivity, at a potential cost of increased error rates."
Reproducibility of Published Research,"Report, policy document or website",Method-shopping,"The third case refers to a classical component of the scienti?c method, namely corroborating research results by other researchers through application of various approaches and methods. If it is not possible to reproduce a research result with other methods, there is not necessarily any problem with that. However, if it is possible, a given research result is usually considered more robust."
What you see is what you get? Enhancing methodological transparency in management research,Review,Method-shopping,"We simply do not know whether what we see is what we get. Most things seem just right: measures are valid and have good psychometric qualities, hypotheses described in the Introduction section are mostly supported by results, statistical assumptions are not violated (or not mentioned), the “storyline” is usually neat and straightforward, and everything seems to be in place. But, unbeknownst to readers, many researchers have engaged in various trial-and-error practices (e.g., revising, dropping, and adding scale items), opaque choices (e.g., including or excluding different sets of control variables), and other decisions (e.g., removing outliers, retroactively creatinghypothesesafter thedatawereanalyzed) that are not disclosed fully. Researchers inmanagement and other fields have considerable latitude in terms of the choices, judgment calls, and trial-and-error decisions they make in every step of the research process—from theory, to design, measurement, analysis, and reporting of results (Bakker et al., 2012; Simmons et al., 2011). Consequently, other researchers are unable to reach similar conclusions due to insufficient information (i.e., low transparency) of what happened in what we label the “research kitchen” (e.g., Bakker et al., 2012; Bergh et al., 2017a, 2017b; Cortina et al. 2017b)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Multiple testing and p-hacking,"There is the issue of multiple hypothesis testing that comes in various forms and can sometimes be deceivingly cryptic (Parker et al., 2016). Here it is important to keep track of the extent of multiple testing."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Multiple testing and p-hacking,"A threshold of P<0.05 seems fairly reasonable when only a few P-values are shown and these P-values mostly lie below the 0.05 threshold. By contrast, referees may request a more stringent threshold when many non-signi?cant results are presented alongside, because the long list clearly reveals the extent of multiple testing. Problematically, when authors are free to choose which results to present in their publication, it becomes impossible to judge the appropriate statistical signi?cance ofthe ?ndings."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Multiple testing and p-hacking,"Just as with an iterative procedure, it is unreasonable to assess the statistical signi?cance ofindividual variables in the ‘best’ model without correction for multiple comparisons."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Multiple testing and p-hacking,"The above approach of exploratory data analysis means that a fairly large number of hypotheses get tested in a very short time (i.e. without careful thinking about speci?c hypotheses considered plausible) and this comes with a high risk of drawing a false-positive conclusion if we only report on the subset of signi?cant predictors. In fact, such exploratory analysis could be seen as an act of generating hypotheses rather than as an act of testing hypotheses, because you only start thinking about the respective hypothesis once you have discovered a signi?cant association. This approach is not wrong per se, as long as you are aware and honest about the fact that the hypothesis was derived from the data"
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Multiple testing and p-hacking,"It is not uncommon for scientists testing the ef?cacy of a new therapeutic agent to try numerous doses and concentrations and time points, employing different modes of administration in multiple animal models, using mice of different ages and health status, to ?nally land on the experiment that ‘‘works the best,’’ and this one experiment makes it into the paper as a key ?gure. None of the dozens of experiments that did not work are ever divulged."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Multiple testing and p-hacking,"Statistical issues affecting replicability are largely the same in many fields of research. Such issues are low statistical power, and ‘data dredging’ or ‘p-hacking’ by trying alternative analyses until a significant result is found, which then is selectively reported without mentioning the nonsignificant outcomes."
Reboot undergraduate courses for reproducibility,"Report, policy document or website",Multiple testing and p-hacking,"Vast numbers of projects, limited time and resources, small sample sizes, the potential for undisclosed analytic flexibility (P-hacking) and a premium on novelty: together, a recipe for irreproducible results."
Reproducibility literature analysis - a federal information professional perspective,Article,Multiple testing and p-hacking,"Cherry picking data includes suppressing evidence, or the fallacy of incomplete evidence by pointing to individual cases or data that seem to confirm a particular position or statistical significance, while ignoring a significant portion of related cases or data that may contradict that position (Baker, 2016a). [Related terms: bias; P-hacking]"
Original article experimental design in ocean acidification research: Problems and solutions,Article,Multiple testing and p-hacking,An experimental culture system can be designed correctly but still result in an inappropriate manipulation experiment when: (i) measurements are interdependent through time (i.e. multiple measurements on one experimental unit over time are treated as independent measurements) and (ii) multiple measurements are made on the same experimental unit and are treated as independent measurements at the same time point.
How Bayes factors change scientific practice,Article,Multiple testing and p-hacking,One way people can cheat with inferential statistics is to make many comparisons and then focus on the one that was significant taken on its own.
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Statistics can be gamed or unreliable,"The usual reporting of these two types of results—either as significant and therefore “reliable” and publishable, or not significant and therefore either not publishable, or seen as showing that the null hypothesis is true—is misleading because it implies an inappropriate level of certainty in rejecting or accepting the null."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Statistics can be gamed or unreliable,Experiment 1–6 showed that the statistically significant (or nearly-significant) effects found in Levy and Keller (2013) are noisy enough that a broad range of possible outcomes—including no effect—can be seen as consistent with the original studies’ estimates.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Statistics can be gamed or unreliable,"Using simulations, Simmons et al. (2011) show that the combination of always choosing the better option in four consecutive arbitrary steps (each of which seems of minor importance, e.g. analysing yearlings and adults together versus separately) adds up to a dramatic effect of raising the ?-level from ? =0.050 to 0.607. That means, if we systematically chose the option that reduces the P-value in each of the four steps, we will be able to present an effect of interest as being statistically signi?cant (P<0.05) in 607 out of 1000 cases in which no real effect exists (hence the formulation ‘allows presenting anything as signi?cant’). Ifthis scenario ofraising ? to 60.7% is applied to Ioannidis (2005) calculations, we would see 535 false positives (red in Fig. 1J) compared to approximately 95 true positives (blue in Fig 1J; note that this latter number is a rough guess and not based on simulations), which would mean that about 85% of all positive ?ndings would be false."
Australia NHMRC's Research Quality Strategy,"Report, policy document or website",Australia - National Health and Medical Research Council,"The Australian and international community expects research to be conducted responsibly, ethically and with integrity. High quality research that is rigorous, transparent and reproducible maximises the opportunity for benefits to be gained. High quality research: contributes to scientific progress is essential for the translation of research outcomes to practical and clinical applications and evidence-based policy that benefit the community delivers the highest possible value for research investment respects research participants, the wider community, animals and the environment, and promotes community trust in scientific findings."
Australia NHMRC's Research Quality Strategy,"Report, policy document or website",Australia - National Health and Medical Research Council,"Support high quality in the development, design, methodology, conduct and analysis of NHMRC-funded research NHMRC will develop guidance for researchers, and peer reviewers of NHMRC funding applications, to ensure the rigour and reproducibility of NHMRC-funded research. Early initiatives will include developing guidance about available tools for systematic review and meta-analysis, and for improving research design for specific types of research. By providing guidance for researchers about the communication and reporting of research methodology, data and findings, the aim is to improve the transparency of NHMRC-funded research in accordance with international standards. Early initiatives will include developing guidance about registration of studies."
Human samples policy - Canadian Cancer Society,"Report, policy document or website",Canada - Canadian Cancer Society,"CCS is committed to ensuring that high quality bio?specimens are used in research that it funds, as these yield high, reproducible quality data. It is the responsibility of the Principal Investigator to ensure that appropriate evidence that the PI has registered/enrolled for bio-specimen collection with a quality assurance program is submitted to the CCS at the time of funding. This applies equally to all prospective (new) bio-specimens used in the CCS-funded research that will be collected and/or all retrospective (old) bio-specimens used in the CCS-funded research that have previously been collected and will come from a biobank(s). There are a number of internationally recognized programs that provide assurance of a known standard and level of quality for biospecimens. These programs include those available from the Canadian Tissue Repository Network (CTRNet) and programs such as CAP, ISO or CLIA (learn more (https://biobanking.org/webs/quality_programs)). Participation in external quality assurance programs will be considered eligible grant expenses."
Europe CHIST-ERA Open Science Policy Statement,"Report, policy document or website",EU - CHIST-ERA Open Science Policy Statement,"Why Open Science? Science is one of those few concepts that surpasses any subjective considerations. It is a universal institution that must be constantly challenged, tested or even reproduced in different conditions. In order to ensure its optimal transmission that lead to new results and discoveries, open availability of research results, free of charge for the user, must be ensured, improving by the same means transparency, reproducibility, visibility and democratisation of research."
Europe Open Science - European Commission,"Report, policy document or website",EU - European Commission Open Science Policy,All publicly funded research in the EU should adhere to commonly agreed standards of research integrity. The results of Research & Innovation activities should be reproducible. A Scoping Report on the was published in December 2020.
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",European Medicines Agency,"The European Medicines Agency (EMA) mandates registration of all paediatric trials and all adult trials at phase 2 through to phase 4 of medical interventions. New guidelines will be implemented in 2019, with an aim to streamlining approvals and providing greater transparency."
Germany Replicability of Research Results A Statement by DFG,"Report, policy document or website",Germany - DFG Replicability of Research Results,"Given a structural framework that can all too easily be misunderstood as an invitation to quick-and-dirty research practices, the DFG also acknowledges its own responsibility. In exercising this responsibility, the DFG must be mindful of various aspects of its activity as an organisation promoting research and scientific self-governance. The DFG will focus on the specific insights to be expected from a research project when it comes to evaluating project proposals; will ensure that, in the ongoing development of its funding portfolio and in the review, evaluation and decision-making processes for which it is responsible, the main criterion for scientific judgement will be the quality of publications rather than their quantity or location; will also take into account that replication as a method for testing experimental and empirical quantitative research results must be systematically strengthened; will therefore facilitate and support processes of subject-specific investigation of questions concerning the replicability of research results; this also includes the development of subject-specific criteria for funding replication studies5 as well as the funding itself; will continue to pay particular attention to questions of research data management and current challenges that emerge from digitalisation; will promote the development of infrastructure and methodological tools as well as their use for this purpose; will remain fully committed to its wide-ranging efforts to promote good scientific practice, which set standards in the German science system; calls on academic publishers, scientific institutions and ethics commissions, as well as lawmakers and scientific policymakers, to do everything in their power to combat the structural reasons for replication difficulties."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Germany - DFG Replicability of Research Results,"Growth in open data practices are also evidenced by funders’ polices in other countries, such as the German Research Foundation (dfg.de/download/pdf/foerderung/programme/ lis/ua_inf_empfehlungen_200901.pdf)"
International DIME Research Reproducibility Standards,"Report, policy document or website",International - World Bank Development Impact Evaluation (DIME) Research Reproducibility Standards,"All DIME projects will use GitHub to document data work. DIME Research assistants will regularly participate in peer code review sessions. All DIME projects will have a master script that runs all the other scripts that are needed for the project, in order. Computational Reproducibility must be verified by DIME Analytics prior to publication for all DIME Working Papers and academic publications. For implementation resources, see the Research Reproducibility Guidelines."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,UK - Academy of Medical Sciences,"In the United Kingdom, the Academy of Medical Sciences recently convened a joint meeting with several other funders to explore these issues, and the US National Institutes of Health has an ongoing initiative to improve research reproducibility."
UK BBSRC Statement on Safeguarding Good Scientific Practice,"Report, policy document or website",UK - Biotechnology and Biological Sciences Research Council,"There is increasing concern within the biomedical research community about the lack of reproducibility of key research findings. If too many results are irreproducible, it could hinder scientific progress, delay translation and waste valuable resource. It also threatens the reputation of the life sciences and the public’s trust in research findings. BBSRC is working with the Academy of Medical Sciences, the MRC and the Wellcome Trust to explore the challenges and opportunities for improving the reproducibility and reliability of biomedical research in the UK. The funders have produced a report and a summary of possible strategies to improve reproducibility. We encourage our researchers to consider the above strategies, and avail themselves, where required, of training opportunities in research reproducibility as part of continuous professional development."
UK BBSRC Statement on Safeguarding Good Scientific Practice,"Report, policy document or website",UK - Biotechnology and Biological Sciences Research Council,"In the interests of making publicly-funded research increasingly accessible, the Research Councils, under the auspices of Research Councils UK, have a policy on access to published research outputs. The aim is for all users to be able to read published research papers in an electronic format and to search for and re?use (including download) the content of published research papers, both manually and using automated tools (such as those for text and data mining), provided that any such re?use is subject to full and proper attribution. Block grants to institutions have been made available to help cover the cost of open access publication."
UK BBSRC Statement on Safeguarding Good Scientific Practice,"Report, policy document or website",UK - Biotechnology and Biological Sciences Research Council,"Researchers also need to be aware of their obligations with respect to sharing other outputs generated from BBSRC-funded research. Researchers should refer to the BBSRC Data Sharing Policy and, where such exists, the Data Management Plan associated with the specific research grant, as well as the UK Concordat on Open Research Data. Our expectations for the sharing of biological resources are set out in Section 4 of the BBSRC Research Grants Guide. All recipients of BBSRC funding are required to report emerging outputs, outcomes and impacts for the duration of their awards and for up to five years beyond. BBSRC uses the researchfish® online system to collect information on the outputs, outcomes and impacts that have arisen from BBSRC-funded research and training."
UK Position Statement - Reproducibility and reliability of biomedical research_MRC,"Report, policy document or website",UK - Medical Research Council,"The scientific community is working to ensure that research is carried out as efficiently and productively as possible, in part by looking at ways to improve the reproducibility of scientific findings. Results are regarded as reproducible when an independent researcher conducts an experiment under similar conditions to a previous study, and achieves commensurate results. Recent discussion of this issue both in the scientific community and in the media provides an opportunity for all those involved, including funders, researchers, publishers and research institutions, to reflect on the way we work and consider what we can do, individually and collectively, to improve the situation. The MRC, along with the Academy of Medical Sciences, the Wellcome Trust and the Biotechnology and Biological Sciences Research Council, held a symposium in April 2015 to explore how to improve the reproducibility and reliability of pre-clinical biomedical research in the UK. A report produced from this symposium was published in October 2015, summarising potential causes of irreproducibility and strategies for countering poor practice."
UK NERC Data Policy - Guidance Notes,"Report, policy document or website",UK - Natural Environment Research Council,"Data that underpin research publications are the data that have been analysed and reported on within a publication. The NERC Data Policy requires all research publications that arise from NERC funding to include a statement on how the supporting data and any other relevant research materials can be accessed. By having a statement on data and research materials, NERC is looking to ensure that the research it funds is transparent and reproducible, to allow others to confirm or challenge the research. The simplest way to include a statement on how the supporting data can be accessed is through formal data citation of the dataset in question. The data centres’ catalogue pages for the dataset will provide the text for the data citation that should be used. Where provided, these citations should include a DOI (Digital Object Identifier) or other clickable link, in order to facilitate quick and easy access to the dataset landing page."
UK+Japan Nomination for the 2021 JSPS Postdoctoral Fellowship for Research in Japan (Standard) through the Royal Society,"Report, policy document or website",UK - The Royal Society,"Experimental Design Assistant The Royal Society recommends that applicants use the Experimental Design Assistant (EDA), which is a free resource from the NC3Rs to support researchers in the planning of animal experiments. This will help to facilitate robust study design and reliable and reproducible findings. The EDA helps applicants build a machine-readable diagram representing their experimental plan, following capture of their methodology, and allows the applicant to then generate a PDF report which provides a transparent description of the experimental design in a standardised format, which can be uploaded to the application form."
UK Open research – UKRI,"Report, policy document or website",UK - UKRI Open Research,"Transparency, openness, verification and reproducibility are important features of research and innovation. Open research helps to support and uphold these features across the whole lifecycle of research – improving public value, research integrity, re-use and innovation. Open research also helps to support collaboration within and across disciplines. It is integral to a healthy research culture and environment. All of this is underpinned by open research policies, practices and procedures that support those undertaking research. They raise awareness of expected standards and behaviours and promote wider systems changes for the benefit of the whole community."
UK Open research – UKRI,"Report, policy document or website",UK - UKRI Open Research,"Open research is integral to UKRI’s mission to deliver economic and social benefit. Accessible, transparent, reproducible and cooperative research: underpins quality and efficiency in the research process ensures research outputs are readily shared. Our ambition is to lead improvements through policy, practice and technological innovations to achieve an open research system that operates internationally. To do this we will: support the adoption of open research through collaboration and alignment with national and international partners support the transition towards an open and transparent research system and address key challenges – by developing policy, rewards and incentives to achieve open research and the necessary digital research infrastructure ensure open access and open research data – two key aspects of open research – are core priorities."
"UK Data, software and materials management and sharing policy - Grant Funding","Report, policy document or website",UK - Wellcome Trust,"Making data available in a timely and responsible way ensures other research can verify it, build on it and use it to advance knowledge and make health improvements. Similarly, making software or materials – such as antibodies or cell lines – available to the research community supports reproducibility and can underpin further research."
"Data management plans, the missing perspective",Note,USA - National Institutes of Health,"The National Institutes of Health (NIH) and National Science Foundation (NSF) share a similar focus on post-publication data management. Since October 1, 2003, the NIH has required that any investigator submitting a grant application seeking direct costs of $500,000 or more in any single year include a plan to address data sharing in the application or state why data sharing is not possible. More recently, starting January 18, 2011, the NSF proposals submitted to the NSF required inclusion of a Data Management Plan. Continuing the focus on management of data post-publication, in February of 2013, the White House Of?ce of Science and Technology Policy (OSTP) in a memorandum, directed Federal agencies providing signi?cant research funding to develop a plan to expand public access to research. Among other requirements, the plans must, ‘‘Ensure that all extramural researchers receiving Federal grants and contracts for scienti?c research and intramural researchers develop data management plans, as appropriate, describing how they will provide for longterm preservation of, and access to, scienti?c data in digital formats resulting from federally funded research, or explaining why long-term preservation and access cannot be justi?ed”."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,USA - National Institutes of Health,"In response, the National Institutes of Health (NIH) has called for action to raise standards for carrying out and reporting experiments (Collins and Tabak, 2014). This initiative encourages the scientific community, including funding bodies, academic centers and publishers, to take measures to help enhance reproducibility in science (Kilkenny et al., 2010)."
USA Implementing Rigor and Transparency in NIH & AHRQ Research Grant Applications,"Report, policy document or website",USA - NIH and AHRQ,"New Authentication of Key Biological and/or Chemical Resources Attachment
Grant applications for the activity codes covered by the policy must include a new PDF attachment related to the authentication of key biological and/or chemical resources.
Authentication of Key Biological and/or Chemical Resources Briefly describe methods to ensure the identity and validity of key biological and/or chemical resources used in the proposed studies.
Key biological and/or chemical resources may or may not be generated with NIH funds and: 1) may differ from laboratory to laboratory or over time; 2) may have qualities and/or qualifications that could influence the research data; and 3) are integral to the proposed research.
These include, but are not limited to, cell lines, specialty chemicals, antibodies, and other biologics.
Standard laboratory reagents that are not expected to vary do not need to be included in the plan. Examples are buffers and other common biologicals or chemicals.
https://grants.nih.gov/grants/guide/notice-files/not-od-16-011.html 2/4
21/01/2021 NOT-OD-16-011: Implementing Rigor and Transparency in NIH & AHRQ Research Grant Applications
Reviewers will assess the information provided in this Section. Any reviewer questions associated with key biological and/or chemical resource authentication will need to be addressed prior to award.
Information in this section must focus only on authentication and/or validation of key resources to be used in the study; all other methods and preliminary data must be included within the page limits of the research strategy. Applications identified as non-compliant with this limitation will be withdrawn from the review process (see NOT-OD-15-095).
Applications submitted for due dates between January 25, 2016 and May 24, 2016 will use the FORMS-C forms and application guide. The general application guide will be updated by November 25, 2015 with instructions for this new attachment and guidance to upload your PDF document (titled ""Authentication of Key Resources Plan"") in the ""Other Attachments"" section of the ""Other Project Information"" form.
Applications submitted for due dates on or after May 25, 2016, will use updated FORMS-D forms. The PHS 398 Research Plan form will include a new ""Authentication of Key Biological and/or Chemical Resources"" attachment field. FORMS-D application forms and instructions will be available for all active Funding Opportunity Announcements at least 60 days prior to due dates that fall on or after May 25, 2016.
Application Review Information
Unless stated otherwise in the Funding Opportunity Announcement, reviewers will be asked to consider additional review questions in order to assess rigor and transparency in research grant applications. By November 25, 2015, all active Funding Opportunity Announcements will be updated to reference these additional review questions.
Scored Review Criteria Significance
Is there a strong scientific premise for the project Approach
Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?
Have the investigators presented adequate plans to address relevant biological variables, such as sex, for studies in vertebrate animals or human subjects?
Additional Review Considerations
Authentication of Key Biological and/or Chemical Resources For projects involving key biological and/or chemical resources, reviewers will comment on the brief plans proposed for identifying and ensuring the validity of those resources.
Research Performance Progress Reports
Research Performance Progress Reports (RPPR) submitted January 25, 2016 or later will be expected to emphasize rigorous approaches taken to ensure robust and unbiased results. Rigor should be addressed in the RPPR for any grant that funds research or training in research; grants that support other activities do not need to address rigor. This includes non-competing continuation reports (Type 5) for grants reviewed and awarded before implementation of the policy. The RPPR instructions will be updated by January 25, 2016. Reporting on rigor in RPPR will help NIH implement and evaluate the policy for both current and new awards, as well as prepare non-competing renewals for the next competitive renewal."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",USA - NIH and AHRQ,"But other funders have linked reproducibility directly to evaluation. The US’s NiH and AHRQ put in place a policy, resources and training to support reproducibility, as part of their wider ‘rigour’ agenda. This includes revised guidance concerning directly the evaluation of prior research in the instructions and review criteria for career development award applications and for Research Grant Applications."
"Data management plans, the missing perspective",Note,USA - National Science Foundation,"The National Institutes of Health (NIH) and National Science Foundation (NSF) share a similar focus on post-publication data management. Since October 1, 2003, the NIH has required that any investigator submitting a grant application seeking direct costs of $500,000 or more in any single year include a plan to address data sharing in the application or state why data sharing is not possible. More recently, starting January 18, 2011, the NSF proposals submitted to the NSF required inclusion of a Data Management Plan. Continuing the focus on management of data post-publication, in February of 2013, the White House Of?ce of Science and Technology Policy (OSTP) in a memorandum, directed Federal agencies providing signi?cant research funding to develop a plan to expand public access to research. Among other requirements, the plans must, ‘‘Ensure that all extramural researchers receiving Federal grants and contracts for scienti?c research and intramural researchers develop data management plans, as appropriate, describing how they will provide for longterm preservation of, and access to, scienti?c data in digital formats resulting from federally funded research, or explaining why long-term preservation and access cannot be justi?ed”."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website",USA - National Science Foundation,"1. Proposals should clarify how the given reproducibility or replication study would build on prior studies and contribute to the development of fundamental knowledge of ways to improve learning and other education outcomes. For example: a. For early-stage or exploratory research, proposals should explain how the reproducibility or replication study would contribute to the accumulation of knowledge regarding relationships among important constructs in education and learning and/or establish logical connections that might form the basis for future interventions or strategies to improve those outcomes. b. If conducting a replication of an impact study (e.g., efficacy, effectiveness, scale-up), proposals should establish the replication’s potential to enhance understanding of the impact of a strategy or intervention under the same (direct replication) or under somewhat changed (conceptual replication) circumstances.
2. Proposals to conduct a conceptual replication should clearly specify the proposed variations from the prior study, along with a rationale for the proposed systematic variations.
3. Proposals for reproducibility or replication studies should ensure objectivity. If the original investigator is involved in the proposed reproducibility or replication study, safeguards need to be included to ensure the objectivity of the findings. At other times (e.g., in re-analysis studies), objectivity may be best accomplished by conducting a separate, independent investigation."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website",USA - National Science Foundation,"4. Transparency is a necessary precondition when designing scientifically valid research. For all evaluations (initial and all replications) that test the impact of an intervention (i.e., efficacy, effectiveness, and scale-up), a pre-registration of the proposed research design and methods can help ensure the integrity and transparency of the proposed research.
5. Education research should continue to strive toward open data access policies, the development of commonly agreed upon data sharing guidelines, and the use of publicly available repositories to store data and other materials. In education research, the term data should continue to be defined in the broadest possible terms to include measures, data dictionaries and codebooks, social network analyses, user generated data, outcome data, and analytic models.
6. Analyses should be described in sufficient detail as to allow other researchers to reproduce the results using the same dataset."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website",USA - National Science Foundation,"7. Researchers should document the features (e.g., population, context, fidelity of implementation) of their study that would be salient to future replications.
8. Researchers should budget resources necessary to engage in the documentation, curation, and sharing activities necessary to facilitate efforts to reproduce and replicate their work.
9. To the extent possible, consent forms and Institutional Review Board (IRB) approvals should reference future public sharing of data and stipulate the conditions that will be put in place to protect the privacy of participants.
10. Researchers should be aware of data management policies across agencies including the Data Management for NSF EHR Directorate Proposals and Awards and the Policy Statement on Public Access to Data Resulting from IES Funded Grants along with the Frequently Asked Questions about Providing Public Access to Data document."
USA Companion Guidelines on Replication & Reproducibility in Education Research,"Report, policy document or website",USA - National Science Foundation,"11. Data used to support claims in publications should be made available in public repositories along with data processing and cleaning methods, relevant statistical analyses, codebooks as well as analytic code.
12. Researchers should analyze and report how the results from their reproducibility or replication study compare to previous studies.
13. Researchers should clearly describe criteria used for exclusion of data or subjects, include results that were omitted for any reason (especially if the results do not support the main findings and/or hypotheses), and describe outcomes or conditions that were measured or used and are for some reason not included in the report.
14. Final reports to funding agencies should include details about how all data and relevant supporting documentation are being made available and can be accessed."
USA Eligibility & Policies - RSF,"Report, policy document or website",USA - RSF,"Transparency & Reproducibility:
Data Release Plan: Publishing data and all associated materials from research projects is valuable because it allows others to examine the robustness of reported results and facilitates fuller re-use of collected data in general. As a condition for providing substantial support for new data collection, RSF requires that the investigators make their data sets publicly available to the social science research community. Investigators must include as part of their proposal a plan for public release of the data and documentation. RSF will consider exceptions for proprietary data and qualitative data.
Pre-registration: Pre-registration is important for various reasons: to ameliorate “publication bias,” as a source of results for meta-analysis, to ~nd out about available survey instruments, and to access and download data. As a condition for providing substantial support for randomized controlled trials (RCTs) , RSF requires RCTs to be preregistered. Investigators must include as part of their application where and when they plan to pre-register the trials. In other cases, RSF may strongly recommend pre-registration as a funding condition. Some existing registries include: The AEA Social Science Registry"
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",USA - USAID,"In the US, USAID fashioned a strategy on ‘transparency’ that pivots on data, and created a Development Data Library for the large number of projects it funds."
Our path to better science in less time using open data science tools,Article,"Borrowing the philosophies, tools and workflows created for software development can help to develop data management processes that ensure reproduciblity","But when we began to reproduce our workflow a second time and repeat our methods with updated data, we found our approaches to reproducibility were insufficient. However, by borrowing philosophies, tools, and workflows primarily created for software development, we have been able to dramatically improve the ability for ourselves and others to reproduce our science, while also reducing the time involved to do so: the result is better science in less time."
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,Code sharing helps readers re-run the analyses behind published work,"Finally, Galaxy provides collaborative and transparent analyses by enabling users to share and publish their analyses via the Web. Once shared, Galaxy analyses can be inspected at every level of detail as well as copied and extended"
Recommendations for open data science,Note,Code sharing helps readers re-run the analyses behind published work,Software packages are critical in data-driven studies. Other researchers can only fully utilize results if the details of the computational methods used are completely transparent. Provide source code for all original software in public repositories such as Github with a license permitting as free use as possible. Cite precise versions and command line prompts for all previously published tools. Do not use proprietary software that cannot be accessed by other researchers.
Recommendations for open data science,Note,Code sharing helps readers re-run the analyses behind published work,"As ‘big data’ analyses become more widespread, best practices for publishing pipelines are still evolving. Full transparency requires that all code, data, and steps be shared in formats that allow another researcher to easily regenerate the same results."
Our path to better science in less time using open data science tools,Article,Code sharing helps readers re-run the analyses behind published work,"We built confidence using these tools by sharing our imperfect code, discussing our challenges and learning as a team. These tools quickly became the keystone of how we work, and have overhauled our approach to science, perhaps as much as e­mail did in decades prior. They have changed the way we think about science and about what is possible. The following describes how we have been using open data science practices and tools to overcome the biggest challenges we encountered to reproducibility, collaboration and communication."
Our path to better science in less time using open data science tools,Article,Code sharing helps readers re-run the analyses behind published work,"It is paramount that our methods are transparent, reproducible, and also repeatable with additional data for tracking changes through time. We now collaboratively code and use version control for all"
How we can make ecotoxicology more valuable to environmental protection,Note,Code sharing helps readers re-run the analyses behind published work,"The bene?ts to data users includes the context required to interpret and integrate the results with all other information evaluated during a risk assessment, less uncertainty in the decision-making process, studies and data that can be used across regulatory jurisdictions, saving time and money, and positions based on data that will be more broadly accepted."
Practical Computational Reproducibility in the Life Sciences,Note,"Fully reproducible analysis workflow tools (e.g., Conda, Docker, singularity) can help ensure that computing softwares and environments are portable for reproducibility","Here, we focus on one important aspect of the reproducibility challenge: ensuring computational analysis can be run reproducibly, even in different environments. We describe a three-layer technology stack composed of open, well-tested, and community-supported components. This three-layer design re?ects steps necessary to support fully reproducible analysis: (1) managing software dependencies, (2) isolating analyses from the idiosyncrasies of local computational environments, and (3) virtualizing entire analyses for complete portability and preservation against time. Fully reproducible analysis work?ows require both ensuring reproducible computations and precise tracking of parameters and source data provenance."
Practical Computational Reproducibility in the Life Sciences,Note,"Fully reproducible analysis workflow tools (e.g., Conda, Docker, singularity) can help ensure that computing softwares and environments are portable for reproducibility","Because most software tools rely on external libraries and analysis work?ows use multiple tools, it is necessary to record versions of numerous components. Given a multitude of operating systems and local con?gurations, ensuring the consistency of analysis software is a considerable challenge. Conda (https://conda.io), a powerful and robust open source package and environment manager, has been developed to address this issue. It is operating system independent, does not require administrative privileges, and provides isolated virtual execution environments"
Practical Computational Reproducibility in the Life Sciences,Note,"Fully reproducible analysis workflow tools (e.g., Conda, Docker, singularity) can help ensure that computing softwares and environments are portable for reproducibility","Conda allows multiple versions of any software tool at the same time, provides isolated environments, and runs on all major Linux distributions, macOS, and Windows. Thus, Bioconda is particularly enabling because it overcomes issues that, to some degree, plague other package managers and is championed by the community—an extremely important metric for ensuring future sustainability."
Practical Computational Reproducibility in the Life Sciences,Note,"Fully reproducible analysis workflow tools (e.g., Conda, Docker, singularity) can help ensure that computing softwares and environments are portable for reproducibility","An additional level of isolation to solve this problem is provided by containerization platforms (or, simply, containers), such as Docker (https://www.docker.com), Singularity (Kurtzer et al., 2017), or rkt (http:// coreos.com/rkt). Containers are run directly on the host operating system’s kernel but encapsulate every other aspect of the runtime environment, providing a level of isolation that is far beyond of what Conda environments can provide."
Practical Computational Reproducibility in the Life Sciences,Note,"Fully reproducible analysis workflow tools (e.g., Conda, Docker, singularity) can help ensure that computing softwares and environments are portable for reproducibility","When the combinations of dependencies required are known in advance, these containers can be created automatically as well; for example, we can create containers for all tool dependencies used in the Galaxy ToolShed (https:// galaxyproject.org/toolshed). Containers provide isolated and reproducible compute environments, but still depend on the operating system kernel version and underlying hardware. An even greater isolation can be achieved through virtualization, which runs analysis within an emulated virtual machine (VM) with precisely de?ned hardware speci?cations. Virtualization, which provides the third layer of our reproducibility stack, can be achieved via commercial clouds, on public clouds, such as Jetstream (https://jetstream-cloud.org), or by using hypervisors or virtual machine applications on a local computer (such as VMware, KVM, Xen, and VirtualBox)."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Git, knitr, Jupyter, and other computational and softwares allow better, direct, data sharing with version control","We present Jupyter notebooks, a document format for publishing code, results and explanations in a form that is both readable and executable."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Git, knitr, Jupyter, and other computational and softwares allow better, direct, data sharing with version control","Jupyter aims to bring notebooks to a broader audience. Jupyter is an open source project, which can work with code in many different programming languages. Different language backends, called kernels, communicate with Jupyter documented protocol; over 50 such backends have already been written, for languages ranging from C++ to Bash. Jupyter grew out of the IPython project (Pérez & Granger, 2007), which initially provided this interface only for the Python language. IPython continues to provide the canonical Python kernel for Jupyter."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Git, knitr, Jupyter, and other computational and softwares allow better, direct, data sharing with version control","Authors can publish notebooks on GitHub along with an environment specification in one of a few common formats. By pointing the Binder web service at the repository, a temporary environment is automatically created with the notebooks and any libraries and data required to run them. This allows authors to publish their code in an interactive and immediately verifiable form."
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,"Git, knitr, Jupyter, and other computational and softwares allow better, direct, data sharing with version control","Several papers have been published with supporting notebooks to reproduce the analysis, or the creation of key plots."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journal and publisher policies on code sharing are becoming more common, but are still less common than research data policies","To assess data quality and enable reproducibility, transparency and sharing of computer code and software (and supporting documentation) are also important – as is, where applicable, the sharing of research materials. Materials include samples, cell lines and antibodies. Journal and publisher policies on sharing code, software and materials are becoming more common but are generally less well evolved and less widely established compared to research data policies."
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Tools and frameworks (e.g., Galaxy project) can help researchers achieve complex computation without needing training","Since 2005, the Galaxy project has worked to address this problem by providing a framework that makes advanced computational tools usable by non experts. Galaxy seeks to make data-intensive research more accessible, transparent and reproducible by providing a Web-based environment in which users can perform computational analyses and have all of the details automatically tracked for later inspection, publication, or reuse."
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Tools and frameworks (e.g., Galaxy project) can help researchers achieve complex computation without needing training",Galaxy’s Web-based graphical user interface (GUI) makes it simple to do everything needed for relatively large data analyses.
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Tools and frameworks (e.g., Galaxy project) can help researchers achieve complex computation without needing training","Using the Galaxy GUI, users can upload their own data or retrieve data from public databases, choose analysis tools, set tool inputs and parameters and run tools. The Galaxy GUI also includes a workflow editor where users can create automated, multistep analyses using drag and drop. Galaxy analyses are completely reproducible."
Galaxy-M: a Galaxy workflow for processing and analyzing direct infusion and liquid chromatography mass spectrometry-based metabolomics data,"Editorials, notes, and letters","Tools and frameworks (e.g., Galaxy project) can help researchers achieve complex computation without needing training","To bring a set of non-targeted DIMS and LC-MS based metabolomics processing and analysis tools into the Galaxy workflow platform. With this we aim to strengthen the move towards standardized, reproducible, transparent and shareable workflows in metabolomics while providing a much more intuitive interface for researchers without programming experience and ultimately providing a platform that can integrate this omics approach with the many others that already exist in the Galaxy environment (e.g. genomics and proteomics)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Validated and Standard code and software should be used to allow discovering errors more easily,"Researchers should avoid the trap of the ‘not invented here’ philosophy: when the problem at hand can be solved using software tools from a well-established project, these should be chosen instead of re-implementing the same method in custom code. Errors are more likely to be discovered when code has a larger user base, and larger projects are more likely to follow better software-development practices. Researchers should learn and implement good programming practices, including the judicious use of software testing and validation."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Validated and Standard code and software should be used to allow discovering errors more easily,"We do not advocate any particular method or software package, and will consider manuscripts that use any of these approaches, provided that the method in question has been validated."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Validated and Standard code and software should be used to allow discovering errors more easily,"Future research should be explicit about what specific kind of qualitative method has been implemented (e.g., narrative research, grounded theory, ethnography, case study, phenomenological research)."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Validated and Standard code and software should be used to allow discovering errors more easily,Code Readability and Dynamic Documents to standardize coding style and legibility within a team. These practices are meant to facilitate the reproduction of the analysis across multiple researchers (including the original researcher on a later occasion).
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,High journal impact factors correlate with lower power,"Concern is the negative correlation between power and journal impact factors. This suggests that high impact factor journals should implement higher standards for prestudy power (optimally coupled with preregistration ofstudies) to assure the credibility of reported results. Speculatively, it is worth noting that the high FRP allowed by low power also allows for the easier production of somehow extraordinary results, which may have higher chances to be published in high impact factor journals."
Our path to better science in less time using open data science tools,Article,Data preparation and curation are critical aspects of making science reproducible,"Using the OHI framework, we lead annual global assessments . Despite our best efforts, we struggled to efficiently repeat our own work during the second assessment in 2013 because of our approaches to data preparation. Data preparation is a critical aspect of making science reproducible but is seldom explicitly reported in research publications; we thought we had documented our methods sufficiently in 130 pages of published supplemental materials of 220 coastal nations and territories, completing our first assessment in 2012, but we had not."
How to Make More Published Research True,Article,Adoption of more appropriate statistical methods,"Adoption of more appropriate statistical methods, standardized definitions and analyses and more stringent thresholds for claiming discoveries or ‘‘successes’’ may decrease false-positive rates in fields that have to-date been too lenient (like epidemiology, psychology, or economics). It may lead them to higher credibility, more akin to that of fields that have traditionally been more rigorous in this regard, like the physical sciences."
Reproducible Research A Retrospective,"Report, policy document or website",Data sharing makes replication easier,"The general conclusion was that delivering a research end product such as a ?gure or table was no longer su?cient. Rather, the software environment and the means to create the end product must also be delivered, as those additional elements represent the actual scholarship. In order to satisfy this requirement, one would have to make available the data and the computer code used to generate the results."
Reproducible Research A Retrospective,"Report, policy document or website",Data sharing makes replication easier,"One could summarize the goal of reproducible research as providing a means to answer the question, “Do I understand and trust this data analysis?” With the computational nature of today’s research, we cannot hope to answer that question without being able to look at the data and the code."
Open is not enough,"Report, policy document or website","Data sharing alone may be problematic in some disciplines, e.g. where very large amounts of complex data are needed","In the particular case of particle physics, it may even be true that openness itself, in the sense of unfettered access to data by the general public, is not necessarily a prerequisite for the reproducibility of the research. Take the LHC collaborations as an example: while they generally strive to be open and transparent in both their research and their software development, analysis procedures and the previously described challenges of scale and data complexity mean that there are certain necessary reproducibility use cases that are better served by a tailored tool rather than an open data repository. Such tools need to preserve the expertise of a large collaboration that flows into each analysis. Providing a central place where the disparate components of an analysis can be aggregated at the start, and then evolve as the analysis gets validated and verified, will fill this valuable role in the community. Confidentiality might aid this process so that the experts can share and discuss in a protected space before successively opening up the content of scrutiny to ever larger audiences, first within the collaboration and then later via peer review to the whole HEP community."
Promises and pitfalls of data sharing in qualitative research,Note,"Data sharing alone may be problematic in some disciplines, e.g. where very large amounts of complex data are needed","It should be suf?cient to enable another team of investigators to conduct a reproduction test. But if reproduction, rather than veri?cation, is the goal, then of what relevance is a data sharing policy?"
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,FAIR principles can help change standards and expectations about open scholarship,"Develop increased expectations for open data as the standard approach in all publications This should apply not only to clinical investigations and “big data” assemblies where such expectations are already prevalent, but as much as possible to standard laboratory work. This would likely require use of quality lab data management tools, now lacking in many labs. Data being publicly available after publication would allow additional analyses to be performed and would facilitate identi?cation of errors in published work. The FAIR guiding principles for scienti?c data management and stewardship (?ndable, accessible, interoperable and reusable) are a major global effort in this regard."
Open is not enough,"Report, policy document or website",FAIR principles can help change standards and expectations about open scholarship,"All of these services, developed through free and open source software, strive to enable FAIR compliant data and can be set up for other communities as they are implemented using flexible data models. For all these services, capturing and preserving data provenance has been a key design feature. Data provenance facilitates reproducibility and data sharing as it provides a formal model for describing published results"
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Using journal and funder policies on data sharing to lead behavioural change,"Obtaining and reanalyzing the original data (using the originally used code) is the most reliable way to obtain accurate estimates for evidence synthesis. Leading journals could trigger a positive change by requiring data and code release for all articles, and introducing a special article type (e.g., a pre-registered Replication Report) for direct replication attempts."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Using journal and funder policies on data sharing to lead behavioural change,"New procedures by journals to enhance quality of manuscripts and reviews. Such changes might include rigorous checklists to promote appropriate design features, enhanced statistical assessment by journals, and encouragement or requirement that raw data be provided during submission to be available online."
"Data management plans, the missing perspective",Note,Using journal and funder policies on data sharing to lead behavioural change,"DMP requirements were found to emphasize post-publication data sharing rather than upstream activities that impact data quality, provide traceability or support reproducibility."
Recommendations for open data science,Note,Using journal and funder policies on data sharing to lead behavioural change,"Journal editors and reviewers must enforce computational reproducibility. These recommendations will likely only take effect if required for publication. Many journals encourage authors to publish source code and data, but often do not have specific requirements about what should be provided, and how."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Using journal and funder policies on data sharing to lead behavioural change,"While some journals have adopted badges to acknowledge open science practices, funding agencies can also play a key role in promoting data sharing. For instance, in 2015, the NIH Public Access Plan outlined that the “NIH intends to make public access to digital scientific data the standard for all NIH-funded research”."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Using journal and funder policies on data sharing to lead behavioural change,"Raising awareness of issues and encouraging behavioural and cultural change, by introducing consistent journal policies on sharing research data, code and materials."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Using journal and funder policies on data sharing to lead behavioural change,"Scholarly publishers and journals can help to raise awareness of issues through their wide or community-focused readership – with editorials, opinion pieces and conference and news coverage. Behavioural change can be created by changing journal and publisher policies, as researchers are motivated to comply with them when submitting papers (Schmidt et al. 2016)."
"Most computational hydrology is not reproducible, so is it really science?",Note,Using journal and funder policies on data sharing to lead behavioural change,"Journals and funding bodies clearly have a role to play in facilitating the change to more open science. Some publishers and hydrological journals are revising their policies to encourage authors to make data and computer codes available to readers [Bl€oschl et al., 2014], notably Vadose Zone Journal with the launch of a reproducible research program, which will verify that code is technically sound and can be used to reproduce the key results of the paper [Skaggs et al., 2015]. AGU Publications also encourages references to data and software to ?nd source material, facilitating transparency and recognition [Hanson and Van Der Hilst, 2014]."
"Most computational hydrology is not reproducible, so is it really science?",Note,Using journal and funder policies on data sharing to lead behavioural change,"Other journals go further. Science, for example, states that all codes used in creation and analysis of data must be available to readers [Sciencemag.org, 2016]. Nosek et al. [2015] have developed guidelines to facilitate gradual adoption of open practices by journals. Funding guidelines for science funding bodies in the U.S. (NSF) and UK (NERC) have moved toward more open science practices, and both require that data and other research materials are made open [NERC, 2016; NSF, 2016]. NERCs open data policy, for example, is designed to ‘‘support the integrity, transparency, and openness of the research it supports.’’ However, despite the intent, these guidelines currently fall short of software sharing, which is only encouraged by the NSF. Finally, changes such as the replacement of the ‘‘Publications’’ section in the NSF biosketch format for grant applications with a ‘‘Products’’ section to recongize other research outputs like software provides important additional incentives for open science practice."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Using journal and funder policies on data sharing to lead behavioural change,"The third concern, that a journal will reject the study if the data show poor experimental control, is perhaps the most legitimate given that experimental control has been widely viewed as a necessary feature of rigorous SCRD (Baer et al., 1968; Cooper, Heron, & Heward, 2007;Horner et al., 2005). Cooper et al. (2007) suggested, “An experiment is interesting and convincing, and yields the most useful information for application, when it provides an unambiguous demonstration that the independent variable was solely responsible for the observed behavior change”"
Promises and pitfalls of data sharing in qualitative research,Note,Using journal and funder policies on data sharing to lead behavioural change,The experiences in these ?elds suggest that leading journals can implement unilateral changes that eventually contribute to building a culture in which data sharing becomes the norm.
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Data sharing can help to reduce questionable research practices or highlight honest errors,"The social contingencies associated with openness, transparency, and prioritizing the advancement of knowledge may effectively compete with publication metrics and related consequences in ways that undermine publication bias and replication research."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Data sharing can help to reduce questionable research practices or highlight honest errors,Open sharing of research protocols and data may discourage ABA researchers from engaging in questionable research practices that enhance publishability.
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Data sharing makes replication easier,"Another key characteristic that affects whether, and how well, a replication study can be carried out is the transparency of the initial research because availability of materials and data, as well as thorough reporting, are needed for replication and are particularly important for independent or direct and partial replications. For example, the availability of data helps replicability and the evaluation of reproducibility because researchers can (a) increase the sample size of previous research; (b) combine their data with previous data in new analyses; (c) reanalyze data to assess the reliability of the initial analyses (which is speci?cally termed reproducibility by National Science Foundation, 2015); and (d) evaluate the parity of samples, which is particularly critical in L2 research as participant demographics, such as pro?ciency, age, and ?rst language (L1), are known to affect language development."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Data sharing makes replication easier,Recommendation: Make more research fully transparent and open for replication by making data available.
Making sense of replications,Article,Data sharing makes replication easier,"Finally, all methods, proposed analyses and data are made publicly accessible via the Open Science Framework to maximize transparency and accountability (Nosek et al., 2015). This approach has two main benefits: first, it improves the experimental designs and protocols with expert input prior to performing the experiments; second, by removing the possibility that the results of the experiments will influence the peer review process, it avoids certain biases (such as the bias against negative results, and the possibility that referees will accept results that are favorable to their point of view and reject results that are not; Chambers, 2013; Nosek and Lakens, 2014)."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Data sharing makes replication easier,"We therefore tried to obtain the code and the data for all relevant papers from the authors. In case these artifacts were not already publicly provided, we contacted all authors of the papers and waited 30 days for a response. In the end, we considered a paper to be reproducible, if the following conditions were met: • A working version of the source code is available or the code only has to be modifed in minimal ways to work correctly. • At least one dataset used in the original paper is available. A further requirement here is that either the originally-used train-test splits are publicly available or that they can be reconstructed based on the information in the paper."
Are we really making much progress? A worrying analysis of recent neural recommendation approaches,Conference Paper,Data sharing makes replication easier,"Mult-VAE, is a collaborative fltering method for implicit feedback based on variational autoencoders. The work was presented at WWW ’18. With Mult-VAE, the authors introduce a generative model with multinomial likelihood, propose a diferent regularization parameter for the learning objective, and use Bayesian inference for parameter estimation. They evaluate their method on three binarized datasets that originally contain movie ratings or song play counts. The baselines in the experiments include both a matrix factorization method from 2008, a linear model from 2011, and a more recent neural method. Accoring to the reported experiments, the proposed method leads to accuracy results that are typically around 3% better than the best baseline in terms of Recall and the NDCG. Using their code and datasets, we found that the proposed method indeed consistently outperforms our quite simple baseline techniques. The obtained accuracy results were between 10% and 20% better than our best baseline. Thus, with Mult-VAE, we found one example in the examined literature where a more complex method was better, by a large margin, than any of our baseline techniques in all confgurations. To validate that Mult-VAE is advantageous over the complex non-neural models, as reported in, we optimized the parameters for the weighted matrix factorization technique and the linear model (SLIM using Elastic Net) for the MovieLens and Netfix NDCG, the optimization goal, are quite small. In terms of the Recall, however, Mult-VAE improvements over SLIM seem solid. Since the choice of the used cutofs (20 and 50 for Recall, and 100 for NDCG) is not very consistent in, we made additional measurements at diferent cut of lengths. The results are provided in Table 9. They show that when using the NDCG as an optimization goal and as a performance measure, the diferences between SLIM and MultVAE disappear on this dataset, and SLIM is actually sometimes slightly better. A similar phenomenon can be observed for the MovieLens dataset. In this particular case, therefore, the progress that is achieved through the neural approach is only partial and depends on the chosen evaluation measure.
Reproducibility and Scalability In some ways, establishing reproducibility in applied machine learning should be much easier than in other scientifc disciplines and also other subfelds of computer science. While many recommendation algorithms are not fully deterministic, e.g., because they use some form of random initialization of parameters, the variability of the obtained results when repeating the exact same experiment confguration several times is probably very low in most cases. Therefore, when researchers provide their code and the used data, everyone should be able to reproduce more or less the exact same results. Given that researchers today often rely on software that is publicly available or provided by academic institutions, the barriers regarding technological requirements are mostly low as well. In particular, virtualization technology should make it easier for other researchers to repeat an experiment under very similar conditions."
"Data management plans, the missing perspective",Note,Data sharing makes replication easier,"The 29% of funders requiring a DMP prior to award suggests acknowledgement that data management planning is important to the credibility of a research proposal or to the intended results. However, the low percentage (8%) of funders requiring a DMP after award is particularly disheartening."
How to Make More Published Research True,Article,Data sharing makes replication easier,"Sharing of data, protocols, materials, and software has been promoted in several -omics fields, creating a substrate for reproducible data practices. Promotion of data sharing in clinical trials may similarly improve the credibility of clinical research."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Data sharing makes replication easier,Sharing of research data is essential for reproducible research.
"Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation",Review,Data sharing makes replication easier,We posit that the adherence to the best practices for data curation is important for both experimental scientists who generate primary data and deposit them in chemical genomics databases and computational researchers who rely on these data for model development.
Our path to better science in less time using open data science tools,Article,Data sharing makes replication easier,"Communication outside the project Open data science tools have made us re­imagine what communication can mean for science and management. They enable us to not only share our code online, but to create reports, e­books, interactive web applications, and entire websites, which we can share for free to communicate our work."
On the issue of transparency and reproducibility in nanomedicine,Letter,Data sharing makes replication easier,"Ideally, this information should be made available via curated online and open-access repositories. Such practice will allow researchers to apply in silico modelling and data mining on large experimental datasets to better understand and predict complex nanotechnology– biology interactions."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Data sharing makes replication easier,"Data disclosure. Future qualitative research should make raw materials available (e.g., transcripts, video recordings). While this criterion is necessary only for exact replication, the disclosure of the raw material is useful for error checking. Authors could make the data available to others researchers directly, in data repositories, or by request. An example is Gao et al. (2017), whose data are available for downloading from the Business History Initiative website."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Data sharing makes replication easier,"To increase reproducibility, the following elements are of specific importance: the integrity of datasets; the availability of data and the transparency of data collection methods (what was not reported, what was not used, why)."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Open data infrastructures can be used to set standards in emerging fields where no standards exist,"To homogenize radiomic reproducibility and repeatability studies, we suggest that the community perform benchmarking studies on common, shared, and publicly available data sets. In particular, this concept has already been proposed within the Image Biomarker Standardization Initiative, where different institutions computed features on a common data set. However, to expand this effort, we have been working on (1) providing users with a common repository with shared data sets for feature benchmarking; (2) providing a computational infrastructure, which directly connects to the repository; and (3) suggesting a standardized way of reporting and collecting computational results."
Open is not enough,"Report, policy document or website",Open data infrastructures can be used to set standards in emerging fields where no standards exist,"Services and tools should be developed with the idea of meshing seamlessly with existing research procedures, encouraging the pursuit of reusability as a natural part of researchers’ daily work. In this way, the generated research products are more likely to be useful when shared openly."
Open is not enough,"Report, policy document or website",Open data infrastructures can be used to set standards in emerging fields where no standards exist,"These JSON components define everything from experimental configurations to data samples and from analysis code to links to presentations and publications. By assembling such schemas, we are creating a standard way to describe and document an analysis in order to facilitate its discoverability and reproducibility."
Open is not enough,"Report, policy document or website",Open data infrastructures can be used to set standards in emerging fields where no standards exist,"Build on what is there. There are many dependable tools available, such as data and code repositories, and methods to facilitate computational reproducibility. Do not reinvent the wheel by creating new solutions from scratch unless really necessary. Use existing tools that are already popular and available, tailor them to your needs and extend them if necessary. Opt for open source solutions with large user communities."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Shared code or data may be highly cited,"Sharing research data has been associated with an increase in the number of citations that researchers’ papers receive (Piwowar et al. 2007; Piwowar and Vision 2013; Colavizza et al. 2019) and an increase in the number of papers that research projects produce (Pienta and Alter 2010). Some researchers report that increased academic credit (Science et al. 2017), and increased visibility of their research (Wiley Open Science Researcher Survey 2016; Schmidt et al. 2016), motivates them to share research data. Publishers and other service providers to researchers can help to both solve problems and increase motivations, in particular those relating to academic credit, impact and visibility."
"Most computational hydrology is not reproducible, so is it really science?",Note,Shared code or data may be highly cited,"Making one’s code reuseable in the ?rst instance, then reproducible, undoubtedly requires extra effort. This is notwithstanding the effort to reproduce someone else’s work, with little reward in the current system of publication to reproduce, and therefore validate, either positively or negatively, a prior result. Thus, it is a perfectly valid question to ask: why go to the effort? Within the current system of academic reward through citation [Koutsoyiannis et al., 2016], making code available and reuseable reduces the barriers to the adoption of developed methods, which as considered above, is more likely to lead to further citation and greater impact in the community."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Shared code or data may be highly cited,Data sharing can increase impact of research by making it more credible.
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Shared code or data may be highly cited,"Data sharing can increase impact for researchers by increasing citations on data, as well as research."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Data sharing specific examples (and data management),"Since 2014, there have been new or intensified efforts to promote open science practices across the biomedical literature. Although it is unlikely that individual interventions have single-handedly resulted in drastic changes, these efforts may cumulatively reflect a gradual shift toward the adoption ofa culture that embraces transparency and replication. For instance, in January 2015, the Institute ofMedicine issued a report that recommended that all stakeholders in clinical trials “foster a culture in which data sharing is the expected norm,” and that funders, sponsors, and journals promote and support data sharing. The International Committee of Medical Journal Editors (ICMJE) also proposed a policy requiring data sharing as a condition ofpublication, even though no formal policy changes have been enacted. Other stakeholders have also supported raw data sharing and some journals have started requesting full protocol sharing, since access to detailed protocols is necessary to allow study procedures to be repeated."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Data sharing specific examples (and data management),"Data sharing is a critical component ofresearch transparency and reproducibility, as it allows independent investigators to explore new hypotheses, synthesize evidence across studies, and implement the same experimental methods using the same data (i.e., a replication study). Our findings indicate that substantial progress been made since our previous evaluation ofbiomedical articles published between 2000 and 2014, with the proportion of articles with information related to data sharing appearing to have increased since 2014. Although many scientific fields still do not promote open science practices, recent incentives have focused on changing the data sharing culture. For instance, since 2015, author guidelines to promote transparency and reproducibility, proposed by Nosek and colleagues, have accumulated hundreds ofjournal signatories. Moreover, certain journals, such as the PLOS journals, are now requiring “authors to make all data underlying the finding described in their manuscript fully available without restriction, with rare exceptions”."
Open is not enough,"Report, policy document or website",Data sharing specific examples (and data management),"Cases in point are the CERN Analysis Preservation (CAP) and Reusable Analyses (REANA), which will be described in more detail below. Their key feature is that they leave the decision as to when a dataset or a complete analysis is shared publicly in the hands of the researchers. Open access can be supported, but the architecture does not depend on either data or code being publicly available. This gives the experimental collaborations full control over the release procedure and thus fully supports internal processing, review protocols and possible embargo periods. Hence, the service is accessible to the thousands of researchers who need the information it contains in order to replicate or reuse results, but the publicfacing functions in HEP are better served by other services, such as CERN Open Data11, HEPData12 and INSPIRE13."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Data sharing specific examples (and data management),"In 2016 Springer Nature, which publishes more than 2,500 journals, begun introducing standard, harmonised research data policies to its journals (Hrynaszkiewicz et al. 2017a). Similar initiatives were introduced by some of the other largest journal publishers Elsevier, Wiley and Taylor and Francis in 2017, greatly increasing the prevalence of journal data sharing policies. These large publishers have offered journals a controlled number (usually four or ?ve) of data policy types, including a basic policy with fewer requirements compared to the more stringent policies."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Data sharing specific examples (and data management),"There have also been research data policy initiatives from communities of journals and journal editors. In 2010 journals in ecology and evolutionary biology joined in supporting a Joint Data Archiving Policy (JDAP) (Whitlock et al. 2010), Public Library of Science (PLOS) introduced a strong data sharing policy to all its journals in 2014, and in 2017 the International Committee of Medical Journal Editors (ICMJE) introduced a standardised data sharing policy (Taichman et al. 2017) for its member journals, which include BMJ, Lancet, JAMA and the New England Journal of Medicine. The main requirement of the ICMJE policy was not to mandate data sharing but for reports of clinical trials to include a data sharing statement."
"Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation",Review,Data sharing specific examples (and data management),We propose an integrated chemical and biological data curation work?ow incorporating speci?c protocols for curating both chemical structures and bioactivities in chemical genomics databases that should precede any model development.
The Materials Data Facility: Data Services to Advance Materials Science Research,Article,Data sharing specific examples (and data management),"With increasingly strict data management requirements from funding agencies and institutions, expanding focus on the challenges ofresearch replicability, and growing data sizes and heterogeneity, new data needs are emerging in the materials community. The materials data facility (MDF) operates two cloudhosted services, data publication and data discovery, with features to promote opendata sharing, self-service datapublicationand curation, andencouragedata reuse, layered with powerful data discovery tools."
The Materials Data Facility: Data Services to Advance Materials Science Research,Article,Data sharing specific examples (and data management),"To this end, a variety of materials-related databases and data repositories have been established, for example, the Materials Project, the Open Quantum Materials Database (OQMD), the NIST Materials Data Repository, NREL MatDB, NIMS MatNavi, Automatic-FLOW for Materials Discovery, Novel Materials Discovery (NoMaD) repository, Computational Materials Data Network, Citrine Informatics’ Citrination platform, and AiiDA. So too have more general scienti?c repositories such as Zenodo, Dryad, Figshare, and Dataverse. There are also emerging activities in e-science and improved collaboration spaces for scientists such as the PRISMS Materials Commons. For further reading, see the review by Kalidindi and De Graef."
The Materials Data Facility: Data Services to Advance Materials Science Research,Article,Data sharing specific examples (and data management),"In the process, we are working to develop new, reusable metadata schemas with collaborators at NIST and to, explore deploying additional data storage locations. We are also investigating integration with complementary materials data and tool repositories with the goal of developing an integrated ecosystem of materials capabilities."
Our path to better science in less time using open data science tools,Article,Data sharing specific examples (and data management),"However, by adopting the data science principles and freely available tools that we describe below, we began building an ‘OHI Toolbox’ and fundamentally changed our approach to science."
Our path to better science in less time using open data science tools,Article,Data sharing specific examples (and data management),"Actually doing reproducible science. As we began the second global OHI assessment in 2013 we faced challenges across three main fronts: (1) reproducibility, including transparency and repeatability, particularly in data preparation; (2) collaboration, including team record keeping and internal collaboration; and (3) communication, with scientific and broader communities. We knew that environmental scientists are increasingly using R because it is free, cross­platform, and open source11 and support provided by developers, and also because of the training and independent groups alike. We decided to base our work in R and RStudio for coding and visualization, Git for version control, GitHub for collaboration, and a combination of GitHub and RStudio for organization, documentation, project management, online publishing."
"Most computational hydrology is not reproducible, so is it really science?",Note,Data sharing specific examples (and data management),"For example, initiatives are relatively well developed in hydrology for opening up and sharing data from individual catchments and cross-catchment data sets [McKee and Druliner, 1998; Renard et al., 2008; Kirby et al., 1991; Newman et al., 2015; Duan et al., 2006], including (quite recently) the development of infrastructures and standards for sharing open water data [Emmett et al., 2014; Leonard and Duffy, 2013; Tarboton et al., 2009; Tarboton et al., 2014]. In addition, different code packages have been made available by developers. Prominent examples include the hydrologic models such as Topmodel [Beven and Kirkby, 1979], VIC [Wood et al., 1992], FUSE [Clark et al., 2008], HYPE [Lindstr€om et al., 2010], open-source groundwater models including MODFLOW [Harbaugh, 2005] and PFLOTRAN, and codes linked to modeling, including optimization/uncertainty algorithms such as SCE [Duan et al., 1993], SCEM [Vrugt et al., 2003] or GLUE [Beven and Binley, 1992]. By being made open, such code has helped spread new ideas and concepts to advance hydrology, and made reproducing each-others’ work easier. However, while sharing data and code are important ?rst steps, sharing alone does not provide the critical detail on implementation contained within a work?ow that is required to reproduce published results."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Data sharing specific examples (and data management),"The open science movement (OSM) aims to increase the transparency of science and reproducibility of scientific findings by encouraging free, open access to research and data (Open Science Collaboration, 2012;UNESCO, 2018). The Open Science Foundation, for instance, maintains an online registry (https://osf.io/) that researchers can use to freely share unpublished manuscripts (e.g., methodologically sound intervention studies that produce limited or no beneficial effects). The availability of such papers on preprint servers (e.g., PsyArXiv) increases access to studies that might otherwise be difficult to publish due to publication bias. Researchers may share their protocols and data on open science platforms that permit access to all. For example, Perspectives on Behavior Science allows authors to archive online material (Hantula, 2016a, 2016b), including data protocols and software."
Promises and pitfalls of data sharing in qualitative research,Note,Data sharing specific examples (and data management),"In 2014, the Public Library of Science (PLOS) journals unveiled a policy stipulating that authors must make available all data underlying the ?ndings described in their published manuscript (Bloom et al., 2014). The implementation of this new policy was something of a watershed moment; although PLOS Medicine was not the ?rst high-impact medical journal to require data sharing as a matter of policy, it is the only one that routinely publishes ?ndings from qualitative studies and qualitative meta-syntheses."
Promises and pitfalls of data sharing in qualitative research,Note,Data sharing specific examples (and data management),"At the American Economic Review, for example, authors make publicly available the raw data and statistical programming code needed to reproduce all of the ?ndings in the published manuscript, and these materials are uploaded to the journal web site prior to publication (Bernanke, 2004)."
Practical Computational Reproducibility in the Life Sciences,Note,Specific coding and software examples,"Galaxy (Afgan et al., 2016), GenePattern (Reich et al., 2006), Jupyter (Kluyver et al., 2016), R Markdown (Baumer et al., 2014), and VisTrails (Scheidegger et al., 2008). These environments automatically record details of analyses as they progress and therefore implicitly make them reproducible."
Practical Computational Reproducibility in the Life Sciences,Note,Specific coding and software examples,"Conda (https://conda.io), a powerful and robust open source package and environment manager, has been developed to address this issue. It is operating system independent, does not require administrative privileges, and provides isolated virtual execution environments"
Towards reproducibility in recommender-systems research,Article,Specific coding and software examples,"To achieve the goal, we experimented with the news recommender system Plista and our research-paper recommender system of Docear. More precisely, we a) varied the recommendation scenarios and kept using the same implementations and evaluation methods, and b) varied the recommendation approaches and kept the same scenarios and evaluation methods."
Towards reproducibility in recommender-systems research,Article,Specific coding and software examples,"Our results demonstrate the challenge of achieving reproducibility in recommender-systems research. When minor variations in recommendation approaches, scenarios and evaluations lead to major changes in the recommendation effectiveness, it becomes dif?cult to reproduce research results. Some researchers accept these dif?culties as “just the way things are”. Others, including ourselves, believe that the current state of reproducibility in recommender-system research leaves room for signi?cant improvement. Improving the status quo will be a gradual journey given the large task at hand. Nevertheless, we want to encourage the community to tackle this challenge by partaking in the following actions: 1. Surveying and learning from other research ?elds. 2. Finding a common understanding of reproducibility. 3. Identifying and understanding the factors that affect reproducibility. 4. Conducting more comprehensive experiments. 5. Adjusting publication practices. 6. Fostering the development and use of recommender-system framework. 7. Creating and establishing best-practice guidelines."
Galaxy-M: a Galaxy workflow for processing and analyzing direct infusion and liquid chromatography mass spectrometry-based metabolomics data,"Editorials, notes, and letters",Specific coding and software examples,"We demonstrate the ease of using these Galaxy workflows via the analysis of DIMS and LC-MS datasets, and provide PCA scores and associated statistics to help other users to ensure that they can accurately repeat the processing and analysis of these two datasets."
Galaxy-M: a Galaxy workflow for processing and analyzing direct infusion and liquid chromatography mass spectrometry-based metabolomics data,"Editorials, notes, and letters",Specific coding and software examples,"The Galaxy platform has enabled us to produce an easily accessible and reproducible computational metabolomics workflow. More tools could be added by the community to expand its functionality. We recommend that Galaxy-M workflow files are included within the supplementary information of publications, enabling metabolomics studies to achieve greater reproducibility."
Recommendations for open data science,Note,Specific coding and software examples,"For analysis of large data sets using standard tools, use citable pipelines (e.g. Galaxy), or a standardized pipeline language such as Yet Another Workflow Language (YAWL)."
Recommendations for open data science,Note,Specific coding and software examples,"For smaller pipelines, post documented source code at a stable website, such as a Git repository, containing scripts and links to data that is specific to a manuscript. This code could take the form of reproducible notebook tools such as Jupyter notebooks or Sweave/knitr scripts, which integrate code with results and figures. Another option is to provide a script or makefile that can regenerate all analyses ofa paper with a single command. Finally, containerizing analyses (e.g. using Docker) can allow them to be easily rerun by others without having to install additional dependencies."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Specific coding and software examples,"It may be unrealistic to expect reviewers to evaluate code in addition to the manuscript itself, although this is standard in some journals such as the Journal of Statistical Software."
Open is not enough,"Report, policy document or website",Specific coding and software examples,"We argue that physics analyses ideally should be automated from inception in such a way that they can be executed with a single command. Automating the whole analysis while it is still in its active phase permits to both easily run the ‘live’ analysis process on demand as well as to preserve it completely and seamlessly once it is over and the results are ready for publication. Thinking of restructuring a finished analysis for eventual reuse after its publication is often too late. Facilitating future reuse starts with the first commit of the analysis code. This is the purpose served by the Reusable Analyses service, REANA: a standalone component of the framework dedicated to instantiating preserved research data analyses on the cloud. While REANA was born from the need to rerun analyses preserved in the CERN Analysis Preservation framework, it can be used to run ‘active’ analyses before they are published and preserved."
Open is not enough,"Report, policy document or website",Specific coding and software examples,"RECAST is a notable example of an application built around reusable workflows, which targets a specific particle physics use case. In particular, RECAST provides a gateway to test alternative physical theories by simulating what those theories predict and then running the simulated data through the analysis workflow used for a previous publication. The application programming interface exposes a restricted class of trustworthy, high-impact queries on the data. The experiment’s data and the data processing workflow need not be exposed directly. Furthermore, the experimental collaborations can optionally maintain an approval process for the new result."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific coding and software examples,"In 2015 the Nature journals introduced a policy across all its research titles that encourages all authors to share their code and provide a “code availability” statement in their papers (Nature 2015). Nature Neuroscience has taken this policy further, by piloting peer review of code associated with research articles in the journal (Nature 2017). Software-focused journals such as the Journal of Open Research Software and Source Code for Biology and Medicine tend to have the most stringent requirements for availability and usability of code."
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences,Article,Specific coding and software examples,"Alternative statistical approaches In our reliability analyses, we adopted the ANOVA-based approach to estimating components of variance (McGraw & Wong, 1996;Shrout& Fleiss, 1979). This is perhaps the most commonly used method in psychology, produced by popular packages such as SPSS. Variance components can alternatively be estimated via the use of linear mixed-effects (LMMs) and generalized linear mixed-effects models (GLLMs; Nakagawa & Schielzeth, 2010). These models allow greater flexibility in dealing with distributional assumptions and confounding variables."
A manifesto for reproducible science,Review,Specific incentive change examples,Policies to promote open science can include reporting guidelines or specific disclosure statements.
A manifesto for reproducible science,Review,Specific incentive change examples,"For example, journals are adopting badges to acknowledge open practices, Registered Reports as a results-blind publishing model and TOP guidelines to promote openness and transparency."
A manifesto for reproducible science,Review,Specific incentive change examples,"Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation’s Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies."
A manifesto for reproducible science,Review,Specific incentive change examples,"Institutions are wrestling with policy and infrastructure adjustments to promote data sharing, and there are hints of open-science practices becoming part of hiring and performance evaluation (for example, http://www.nicebread.de/open-sciencehiring-practices/). Collectively, and at scale, such efforts can shift incentives such that what is good for the scientist is also good for science — rigorous, transparent and reproducible research practices producing credible results."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Specific incentive change examples,"One effort to acknowledge this is the OHBM Replication Award, which is to be awarded for the first time in 2017 for the best neuroimaging replication study in the previous year."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Specific incentive change examples,"In response, the National Institutes of Health (NIH) has called for action to raise standards for carrying out and reporting experiments (Collins and Tabak, 2014). This initiative encourages the scientific community, including funding bodies, academic centers and publishers, to take measures to help enhance reproducibility in science (Kilkenny et al., 2010)."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific incentive change examples,"The Center for Open Science offers digital badges that are displayed on published articles to highlight, or reward, papers where the data and materials are openly available and for studies that are pre-registered. Badges signal to the reader that the content has been made available and certify its accessibility in a persistent location. More than 40 journals, in 2018, offered or were experimenting with the award of badges to promote transparency (Blohowiak 2013)."
Raising research quality will require collective action,"Report, policy document or website",Specific incentive change examples,"Earlier this year, the University of Bristol, where I work, made the use of data sharing and other open-research practices an explicit criterion for promotion. But one institution will make little difference on its own. For better practices to become the norm, many universities need to act collectively. Changes to incentives at a single institution will not make new behaviours stick, not least because practices required in only one place can act as a career tax on its scientists. Only if changes occur across many institutions will the impacts permeate scientific culture."
Reboot undergraduate courses for reproducibility,"Report, policy document or website",Specific incentive change examples,"In an effort to disrupt this culture, I set up the GW4 Undergraduate Psychology Consortium with colleagues at the universities of Bath, Bristol, Cardiff and Exeter. We wanted to embed rigorous research practices into undergraduate education, incorporating procedures such as pre-registration of study protocols, designing studies with sufficient statistical power and transparent reporting of methods and results."
Reproducibility of Published Research,"Report, policy document or website",Specific incentive change examples,"Horizon 2020 requires participants to meet the highest standards of research integrity, as set out in the European Code of Conduct for Research Integrity.9 Various elements safeguard adherence to these principles and enable the detection of research misconduct, including different tools to detect cases of misconduct during the evaluation process and the technical review of project proposals."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Specific incentive change examples,"Delft University of Technology in the Netherlands began building a community of data champions across all faculties, from aerospace engineering to technology, policy and management.
These champions’ role? To nudge staff and students to manage their research data better. Among other incentives, they can apply for dedicated grants to do so."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Specific incentive change examples," 
London now shuns journal-based metrics in staff assessment; it relies more on peer judgement of research quality."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Specific incentive change examples,"At Mahidol University in Bangkok, Thailand, all staff sign the university’s code of good governance, agreeing to uphold integrity, impartiality and social responsibility, for example."
Replication in strategic management,Editorial,Specific incentive change examples,SMJ is interested in replications that accord with prior findings as well as those that do not.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific Peer-review examples,"The journal Biostatistics employs an Associate Editor for reproducibility, who awards articles “kite marks” for reproducibility, which are determined by the availability of code and data and if the Associate Editor for reproducibility is able to reproduce the results in the paper (Peng 2009). Another journal, npj Breast Cancer, has involved an additional editor, a Research Data Editor (a professional data curator), to assess every accepted article and give authors editorial support to describe and share link to the datasets that support their articles (Kirk and Norton 2019)."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Specific Peer-review examples,"This month, BMC Psychology launches a pilot to trial a new ‘results-free’ peer-review process, to address the bias in the editorial process. Editors and reviewers will be blinded to the study’s results, and decide whether to accept or reject manuscripts based on the scientific merits of their rationale and methods alone. Authors submit otherwise complete manuscripts, but omit any discussion of results, and provisional acceptance is based on peer review of the background and methods alone."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Specific Peer-review examples,"For one, some journals are introducing the option to submit the theory ?rst, and the empirical tests and results later (see, e.g., Comprehensive Results in Social Psychology, and Management and Organization Review; cf. Lewin et al., 2017)"
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Specific pre-registration examples,"Language Learning is enhancing its participation in the open science movement by launching Registered Reports as an article category as ofJanuary 1, 2018."
A manifesto for reproducible science,Review,Specific pre-registration examples,"Support for study pre-registration is increasing; websites such as the Open Science Framework (http://osf.io/) and AsPredicted (http://AsPredicted. org/) offer services to pre-register studies, the Preregistration Challenge offers education and incentives to conduct pre-registered research (http://cos.io/prereg), and journals are adopting the Registered Reports publishing format
to encourage pre-registration and add results-blind peer review."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Specific pre-registration examples,"Over the last few years, support for preregistration and protocol development has increased and various preregistration and protocol sharing platforms, including the Open Science Framework (http://osf.io) and AsPredicted (http://AsPredicted.org/), have been introduced. On April 4th, 2017, the PLOS journals announced the addition ofthe protocols.io platform to the guidelines in all oftheir journals. Protocols.io is an online platform that allows researchers to create and publish protocols. Since 2017, over 200 other journals, including eLife, have partnered with protocols.io. These efforts have reduced barriers to sharing and will likely result in improved preregistration and protocol sharing practices among journals in the future."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific pre-registration examples,"Registration is well established – and mandatory – for clinical trials, using databases such as ClinicalTrials.gov and the ISRCTN register."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific pre-registration examples,"Preregistration has been adopted by other areas of research, and databases are now available for preregistration of systematic reviews (in the PROSPERO database) and for all other types of research, with the Open Science Framework (OSF) and the Registry for International Development Impact Evaluations (RIDIE)."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Specific pre-registration examples,"Following medicine’s example, several platforms supporting protocol publication have been launched to promote transparency in psychology and the social sciences (e.g., the Centre for Open Science’s Open Science Framework, and the Berkeley Initiative for Transparency in the Social Science (BITSS), to name just a few). Some journals, including the medical journals of the BMC series, also publish study protocol articles in an effort help to improve the standard of medical research, reduce publication bias and improve reproducibility."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Specific replication study examples,"In particular, some journals have been lowering obstacles for researchers to publish replication studies. For instance, Research Notes, a BioMed Central journal, publishes null results and provides “an open access forum for sharing data and useful information”. Similarly, Elsevier has developed a new article type especially for replication studies. Moving forward, the publication ofreplication studies will be facilitated by emergence ofmore journals soliciting “non-novel” manuscripts."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Specific replication study examples,"Systematic replication projects have been reported in psychology (Open Science Collaboration, 2015), economics (Evanschitzky & Armstrong, 2010), medicine (Ioannidis, 2005), and business (Hubbard & Armstrong, 1994)."
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,Specific reporting guidelines examples,ISEV endorses the EV-TRACK knowledge base as a facilitating and updatable tool for comprehensive reporting of EV experimental studies.
A manifesto for reproducible science,Review,Specific reporting guidelines examples,"The Transparency and Openness Promotion (TOP) guidelines offer standards as a basis for journals and funders to incentivize or require greater transparency in planning and reporting of research. TOP provides principles for how transparency and usability can be increased, while other guidelines provide concrete steps for how to maximize the quality of reporting in particular areas. For example, the Consolidated Standards of Reporting Trials (CONSORT) statement provides guidance for clear, complete and accurate reporting of randomized controlled trials. Over 300 reporting guidelines now exist for observational studies, prognostic studies, predictive models, diagnostic tests, systematic reviews and meta-analyses in humans, a large variety of studies using different laboratory methods, and animal studies. The Equator Network (http://www.equator-network. org/) aggregates these guidelines to improve discoverability."
A manifesto for reproducible science,Review,Specific reporting guidelines examples,"Guidelines for improving the reporting of research planning; for example, the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) statement for reporting of systematic reviews and meta-analyses, and PRISMA-P for protocols of systematic reviews. The Preregistration Challenge workflow and the pre-registration recipe for social-behavioural research also illustrate guidelines for reporting research plans."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Specific reporting guidelines examples,We therefore developed the Consensus on Exercise Reporting Template (CERT) to provide additional direction for reporting exercise interventions.
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Specific reporting guidelines examples,"While the proportion of articles with any information related to potential conflicts ofinterest disclosures has increased rather steadily over time, likely in response to strengthening of biomedical journal disclosure policies, the proportion of articles reporting no conflicts ofinterest has remained fairly constant and may underestimate the true prevalence ofconflicts in biomedical research."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Specific reporting guidelines examples,"Previous estimates suggest that up to 69% ofpublished clinical research articles have some form offinancial conflict. While disclosure ofconflicts ofinterest has become more common as a result ofthe uniform forms developed by the ICMJE, which is supported by hundreds ofbiomedical journals, it is possible that authors are not complying with the guidelines or are unaware ofpotential conflicts that can impact the design, conduct, and analyses ofstudies."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Specific reporting guidelines examples,"Similar to guidelines in other scientific fields, “MISEV2014”, as we will call it here, provided recommendations on experimental methods and minimal information in reporting."
How to Make More Published Research True,Article,Specific reporting guidelines examples,"Reporting, review, publication, dissemination, and post-publication review of research shape its reliability. There are currently multiple efforts to improve and standardize reporting (e.g., as catalogued by the EQUATOR initiative) and multiple ideas about how to change peer review (by whom, how, and when) and dissemination of information."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Specific reporting guidelines examples,"As of May, 2013, investigators had registered more than 1000 systematic review protocols from 27 countries. The National Institutes for Health Research, UK, have mandated the registration of systematic reviews that they fund. The Canadian Institutes of Health Research are working on a similar policy initiative. Systematic Reviews, an open-access Medline-indexed journal, publishes systematic review protocols. Since launch in February, 2012, the journal has published 89 protocols (as of November, 2013). These initiatives will also enable researchers to periodically assess the association between review protocols and final publications."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific reporting guidelines examples,"The prevalence and endorsement of reporting guidelines, catalogued by the EQUATOR Network (http://www.equator-network.org), in journals has increased substantially in the last decade. Reporting guidelines usually comprise a checklist of key information that should be included in manuscripts, to enable the research to be understood and the quality of the research to be assessed. Reporting guidelines are available for a wide array of study designs, such as randomised trials (the CONSORT guideline), systematic reviews (the PRISMA guidelines) and animal preclinical studies (the ARRIVE guidelines; discussed in detail in another chapter in this volume)."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific reporting guidelines examples,The processes of checking manuscripts for adherence to guidelines can however be supported with arti?cial intelligence tools such as https://www.penelope.ai/.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Specific reporting guidelines examples,"In 2017 the Nature checklist evolved into two documents, a “reporting summary” that focuses on experimental design, reagents and analysis and an “editorial policy checklist” that covers issues such as data and code availability and research ethics. The reporting summary document is published alongside the associated paper and, to enable reuse by other journals and institutions, is made available under an openaccess licence (Announcement 2017)."
Original article experimental design in ocean acidification research: Problems and solutions,Article,Specific reporting guidelines examples,"This study highlights the need for increased attention to the design, performance, analysis, and reporting of experimental procedures used during ocean acidi?cation manipulation experiments. It is encouraging that most studies follow the “Guide to Best Practices for Ocean Acidi?cation Research and Data Reporting (Gattuso et al., 2010)” for manipulating seawater carbonate chemistry,"
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Specific reporting guidelines examples,"In several other research areas, systematic reporting recommendations have been developed to guide researchers, reviewers, and editors during the publication process, for instance, the STROBE statement in the ?eld of epidemiology, the ARRIVE guideline for in vivo toxicity studies, Nature’s reporting checklist for life sciences articles, and the MIAME reporting standard for microarray experiments."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Specific reporting guidelines examples,"The CRED project aims to improve the reproducibility, consistency, and transparency of reliability and relevance evaluations of ecotoxicity studies, both within and between regulatory frameworks, countries, institutes, and individual assessors. Additional aims are to improve the usability of peerreviewed literature for regulatory purposes and to facilitate the exchange of assessments between frameworks. Furthermore, the CRED project aims to improve the reporting of ecotoxicity studies by providing recommendations for reporting methodological details and results. The present study describes the CRED evaluation method, including extensive guidance on how to use the reliability and relevance criteria and it describes the CRED recommendations for reporting of ecotoxicity studies. The CRED project addresses aquatic ecotoxicity studies but can be adapted to other types of ecotoxicity studies."
How we can make ecotoxicology more valuable to environmental protection,Note,Specific reporting guidelines examples,"We propose a series of nine reporting requirements, followed by a set of recommendations for adoption by the ecotoxicology community. These reporting requirements will provide clarity on the the test chemical, experimental design and conditions, chemical identi?cation, test organisms, exposure con?rmation, measurable endpoints, how data are presented, data availability and statistical analysis."
How we can make ecotoxicology more valuable to environmental protection,Note,Specific reporting guidelines examples,"The model checklist provided below will assist authors and peer reviewers of ecotoxicology studies to improve their reporting and assessment. Reporting requirement
1. Test compound source and properties Source and purity provided? Technical name?
2. Experimental design Hypotheses, if any, stated? Number of treatments and their exposure levels? Number and type of controls? Duration of exposures? Number of replicates?
3. Test organism characteristics Name, source, and strain of species reported? Control performance criteria met? Husbandry protocols listed?
4. Experimental conditions General test conditions reported? Source and condition of media? Acclimation and feeding?
5. Exposure con?rmation Clear statement of which samples were analyzed? Method LOD and LOQ provided? Nominal or measured used in subsequent analyses?
6. Endpoints All endpoints monitored, regardless of response, provided? Clear de?nitions and measurement units provided?
7. Presentation of results and data All data, regardless of statistical signi?cance is discussed? Untransformed data provided?
8. Statistical analysis Statistical ?owchart? Transformations justi?ed? All outliers are reported Justi?cation for model selection and variables? NOE-LOEC: power of test and percent change reported? ECx: Model estimates and con?dence intervals provided?
9. Raw data Nominal and measured concentrations provided? Untransformed response by replicate available in some form?"
The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences,Article,Specific reporting guidelines examples,"In noting that measures are constructed to achieve different aims in experimental and correlational research, we can also consider whether it is problematic to attempt to
experimentally manipulate behavior on measures constructed to reliably measure individual differences. For example, self-report measures such as the UPPS-P are developed with the explicit purpose of assessing stable traits (Whiteside & Lynam, 2001), such that they should be purposefully robust to natural or induced situational variation."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Specific reporting guidelines examples,"Journals’ publishing ethos and guidance Many journals provide guidance to encourage authors and reviewers to focus on assessing study quality. For example, in 2015, BioMed Central introduced a Minimum Standards of Reporting checklist for authors and reviewers."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Using badges to motivate reproducible and transparent research,"Badges make good scienti?c practice visible. The Open Science Framework (https://osf.io/) has also started an initiative to make good scienti?c practice visible by awarding badges to studies that meet certain criteria, currently including preregistration, the availability of archived data, and the availability of detailed materials."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Using badges to motivate reproducible and transparent research,"While some journals have adopted badges to acknowledge open science practices [25], funding agencies can also play a key role in promoting data sharing. For instance, in 2015, the NIH Public Access Plan outlined that the “NIH intends to make public access to digital scientific data the standard for all NIH-funded research”."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Using badges to motivate reproducible and transparent research,increasing incentives for practising open research with data journals and software journals and implementing data citation and badges for transparency;
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Using badges to motivate reproducible and transparent research,"The journal Biostatistics employs an Associate Editor for reproducibility, who awards articles “kite marks” for reproducibility, which are determined by the availability of code and data and if the Associate Editor for reproducibility is able to reproduce the results in the paper (Peng 2009)."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Using badges to motivate reproducible and transparent research,incentives to promote transparency by providing opportunities for additional articles and citations and new forms of incentive such as digital badges.
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Using badges to motivate reproducible and transparent research,"Publish Raw Data In an ideal world researchers should publish all raw data. This is easy with small volumes of behavioral data but it has serious monetary and time investment costs with large neural data volumes (see also Nichols et al., 2017). Some repositories have already been set up and it is important that funders cover these costs and optimally provide infrastructure (see Pernet and Poline, 2015; Nichols et al., 2017). Incentives such as a badge system may help promote availability of more raw data (Nosek et al., 2015). In our opinion it is important to publish unprocessed raw data because processed data may already have been distorted/biased in undocumented ways. In general, it is more and more usual to reanalyse data from large repositories, so much further development can be expected in this area (e.g., Eklund et al., 2012)."
What you see is what you get? Enhancing methodological transparency in management research,Review,Using badges to motivate reproducible and transparent research,"others can be used to give “badges” to accepted manuscripts that are particularly transparent (Kidwell et al., 2016)."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Crowdsourcing for replication studies has shown to be useful,"Another approach is to crowdsource proposals for replication (see PsychFileDrawer http://www.psych?ledrawer.org/top-20), whereby a social media platform allows people to propose and vote on the studies that they would like to see replicated. Since it began in 2012, this archive of replications currently holds 71 reports, but the extent to which such an initiative, which is outside the standard publication venues, will have a lasting impact on the number or quality of replications is unclear."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,High profile journals are starting to recognise the need to publish negative findings,"Increased willingness to publish negative and con?rmatory studies Efforts should be made to increase the ease of publishing well conducted studies with negative results, and studies that con?rm prior research. Though such studies offer fewer incentives to both scientists and journals, for science as a whole they are of critical importance, and we should ?nd ways to celebrate the best among them. Having said this, negative or con?rmatory studies can be poorly done, sloppy, fraudulent, or defective in other ways, so high standards are important for publishing these studies as well. Although all journals should see the importance of doing this, their failure to date has led to new venues arising that speci?cally seek to accommodate negative studies. High pro?le journals are beginning to take up the call to recognize and publish replication studies, sometimes creating new venues in which to do this."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,High profile journals are starting to recognise the need to publish negative findings,"But there are many journals that encourage publication of all methodologically sound research, regardless of the outcome. The BioMed Central (BMC) journals launched in 2000 with this mission to assess scienti?c accuracy rather than impact or importance and to promote publication of negative results and single experiments (Butler 2000). Many more “sounds science” journals – often multidisciplinary “mega journals” including PLOS One, Scienti?c Reports and PeerJ – have since emerged, almost entirely based on an online-only open-access publishing model (Björk 2015)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,In the long run reproducible science may be beneficial to individual researchers,"Then, if one of your ?ndings is contradicted by later work you need not worry about your reputation, and maybe more important, you can feel good about the fact that you interpreted the data in the most objective way. In this context, a recent commentary by Markowetz (2015) is an interesting read. He lists ?ve sel?sh reasons why you should work reproducibly. We have already touched on a couple ofpromising ways to make science more reliable, and we add more details below."
Detecting and avoiding likely false-positive findings – a practical guide,Article,In the long run reproducible science may be beneficial to individual researchers,"Preregistering your study may take you a couple ofdays, but in the long run it will bene?t you tremendously by forcing you to think through your study plans very carefully."
Detecting and avoiding likely false-positive findings – a practical guide,Article,In the long run reproducible science may be beneficial to individual researchers,"Likely you will discover some weaknesses in your questions, study design or analysis plan and you still have time to ?x or amend these issues before starting data collection. Also, the preparatory work will make the data analysis easier since you will already have a detailed plan, and it will make writing your paper much easier, since you will already have written your Methods section."
Detecting and avoiding likely false-positive findings – a practical guide,Article,In the long run reproducible science may be beneficial to individual researchers,"If you are thinking ofpreregistering your study now, it is not unlikely that you will be among the ?rst in your ?eld to do so. Would you like to be able to say that you were among the ?rst to embrace this new tool that ensures objectivity? Give it a try and you will likely discover that preregistering is emotionally rewarding like the submission of a manuscript. It also lends importance to your project."
Detecting and avoiding likely false-positive findings – a practical guide,Article,In the long run reproducible science may be beneficial to individual researchers,Such a change in incentives would also be good news for the above-mentioned junior scientists who would worry about the rigour and merit of their experiments rather than the outcome. They could move forward knowing that well-designed tests of interesting ideas would make all their parameter estimates valuable and publishable.
Open is not enough,"Report, policy document or website",In the long run reproducible science may be beneficial to individual researchers,"Open science and reproducible research have become pervasive goals across research communities, political circles and funding bodies. The understanding is that open and reproducible research practices enable scientific reuse, accelerating future projects and discoveries in any discipline."
"Most computational hydrology is not reproducible, so is it really science?",Note,In the long run reproducible science may be beneficial to individual researchers,"Furthermore, making code reuseable is bene?cial for our own work ef?ciency [Donoho et al., 2009]. Across hydrology, much duplicated code is likely to be written for common tasks that are not deemed worthy of publication. However, if open, reuseable practices are adopted by the broader community to make all code open and citable, this would reduce the amount of individual code to be written, and lead to improved ef?ciency at a community level."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Incentives are more and more promoting reproducible research and open scholarship,"The American Association for Applied Linguistics (2017) recently amended its guidelines to recommend that “high quality replication studies, which are critical in many domains ofscienti?c inquiry within applied linguistics, be valued on par with non-replication-oriented studies.” Engaging students with conducting replication studies has also been discussed (see Frank & Saxe, 2012; Porte, 2012), and we are aware of several graduate programs where replication is an integral part of training and assessment. Recommendation: Encourage efforts (e.g., via teaching and training infrastructures, institutional recognition, and professional association conferences and promotion guidance) to reward those who include replication research in their work."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Incentives are more and more promoting reproducible research and open scholarship,"Awarding Open Science badges to encourage authors to make materials and data available on a sustainable open repository and to preregister their studies (Tro?movich & Ellis, 2015); joining the Centre for Open Science preregistration award scheme in 2016 (https://cos.io/prereg); and promoting the IRIS Replication Award in 2017 (https://www.iris-database.org/iris/app/home/replication_award)."
A manifesto for reproducible science,Review,Incentives are more and more promoting reproducible research and open scholarship,"Change is occurring. The TOP guidelines promote open practices, while an increasing number of journals and funders require open practices (for example, open data), with some offering their researchers free, immediate open-access publication with transparent post-publication peer review (for example, the Wellcome Trust, with the launch of Wellcome Open Research)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Incentives are more and more promoting reproducible research and open scholarship,"The neuroimaging community should acknowledge replication reports as scientifically important research outcomes that are essential in advancing knowledge. One effort to acknowledge this is the OHBM Replication Award, which is to be awarded for the first time in 2017 for the best neuroimaging replication study in the previous year."
A manifesto for reproducible science,Review,New and user friendly tools and infrastructures exist to support reproducible research practices,Commercial and non-profit organizations are building new infrastructure such as the Open Science Framework to make transparency easy and desirable for researchers.
A manifesto for reproducible science,Review,New and user friendly tools and infrastructures exist to support reproducible research practices,"As public forms of pre- and post-publication review, these new services introduce the potential for new forms of credit and reputation enhancement. In the conventional model, peer review is done privately, anonymously and purely as a service. With public commenting systems, a reviewer that chooses to be identifiable may gain (or lose) reputation based on the quality of review. There are a number of possible and perceived risks of non-anonymous reviewing that reviewers must consider and research must evaluate, but there is evidence that open peer review improves the quality of reviews received."
Our path to better science in less time using open data science tools,Article,New and user friendly tools and infrastructures exist to support reproducible research practices,Powerful tools exist and are freely available to use; the barriers to entry seem to be exposure to relevant tools and building confidence using them.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Replications may be highly cited,Recent attempts to replicate classic studies in psychology have received citations at amuch higher rate than the average study in the journal in question.
Detecting and avoiding likely false-positive findings – a practical guide,Article,Replications may be highly cited,"Of course this may be in part due to the current novelty of replication research (ironically). However, robust, well-conducted replications of important work will presumably attract considerable attention in the future, especially as awareness grows about the importance of replication in assessing validity of prior work. We expect that as more journals explicitly invite replications (as some are beginning to do), more researchers will come to recognize their utility, and thus researchers will more often seek to cite replications because of the strong inferences they facilitate."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Replications may be highly cited,"At the other end of the spectrum are papers of high impact with results of potential importance and great interest to the community. Sometimes, and thankfully quite rarely, the conclusions of such papers are rapidly shown to be false. In such cases, other scientists have been actively involved in the same research question, and at the time of publication already have data bearing on the new papers’ claims. They view the success of their own research as requiring the new claims to be rapidly veri?ed or rejected. If their work fails to replicate it, they would be motivated to quickly publish the negative results. Two prominent examples of high pro?le studies whose major ?ndings were quite rapidly shown to be false, were the claims about the putative beta cell stimulatory hormone betatrophin [13e15] and STAP cells as a facile approach to creating totipotential stem cells [16e18]. In such cases, though the corrections and/or retractions may be quickly published, it is important to understand why the erroneous ?ndings came to be believed by the scienti?c teams and then published in high impact journals to public acclaim. Such outcomes can result from honest errors and/or incompetence, or more nefarious causes involving research misconduct or even outright fraud."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Reproducible research promotes collaboration,Reproducible research practices facilitate collaboration with other researchers and provide a strong foundation for future researchers to build on (increasing the likelihood of citations).
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Researchers are likely to follow the good example of mentors or other researchers they respect,"Researchers are generally eager to do high-quality research, and institutions should avoid reforms that are perceived as bureaucratic, or they will undermine intrinsic motivation."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Some funders are starting to invest in reproducibility,"StudySwap is currently run by volunteers and is not yet geared up for large-scale replication work. This will require active support from major funding agencies, and there are welcome signs of this happening, according to Brian Nosek, executive director of the Center for Open Science. “For example, the NWO (Netherlands Organisation for Scientific Research) has a 3 million Euro funding line for replications of important discoveries”, he said."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Some funders are starting to invest in reproducibility,"There is no shortage of projects focusing on replication, and there are signs of funding bodies devoting resources, as in the Netherlands. “This is changing, and research assessment structures are I think open to measures beyond the grants in/papers out approach”."
A manifesto for reproducible science,Review,Some funders are starting to invest in reproducibility,"Funders are also adopting transparency requirements, and piloting funding mechanisms to promote reproducibility such as the Netherlands Organisation for Scientific Research (NWO) and the US National Science Foundation’s Directorate of Social, Behavioral and Economic Sciences, both of which have announced funding opportunities for replication studies."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Some funders are starting to invest in reproducibility,"One effort to acknowledge this is the OHBM Replication Award, which is to be awarded for the first time in 2017 for the best neuroimaging replication study in the previous year."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Some funders are starting to invest in reproducibility,"In response, the National Institutes of Health (NIH) has called for action to raise standards for carrying out and reporting experiments (Collins and Tabak, 2014). This initiative encourages the scientific community, including funding bodies, academic centers and publishers, to take measures to help enhance reproducibility in science (Kilkenny et al., 2010)."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Some funders are starting to invest in reproducibility,"Research institutions, as employers, and funders, as sponsors, have an important role in changing such practices. Some funders, like the Wellcome Trust, have led the way on research integrity and start requiring a change in both attitudes and practices."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Journals can promote and incentivise open scholarship,"Publishing more research open access, so that papers are freely and immediately available online, is an obvious means to increase transparency."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Journals can promote and incentivise open scholarship,"Reuse of the research literature is essential for text and data mining research, and this kind of research can progress more ef?ciently with unrestricted access to and reuse of the published literature. Publishers can enable the reuse of research content published in subscription and open-access journals with text and data mining policies and agreements. Publishers typically permit academic researchers to programmatically access their publications, such as through secure content application programming interfaces (APIs), for text and data mining research (Text and Data Mining – Springer; Text and Data Mining Policy – Elsevier)."
How to Make More Published Research True,Article,Large scale collaborative approaches to research can promote reproducible practices,"One option is to transplant into as many scientific disciplines as possible research practices that have worked successfully when applied elsewhere. Adoption of large-scale collaborative research with a strong replication culture has been successful in several biomedical fields: in particular, in genetic and molecular epidemiology. These techniques have helped transform genetic epidemiology from a spurious field to a highly credible one. Such practices could be applied to other fields of observational research and beyond."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Large scale collaborative approaches to research can promote reproducible practices,"Publishers can promote transparency through collaboration. The biggest policy and infrastructural challenges that enable the publication of more reproducible research can only be tackled by multiple publishers collaborating as an industry and collaboration with other organisations that support the conduct and communication of research – repositories, institutions and persistent identi?er providers. Progress resulting from such collaborations has been seen in data citation (Cousijn et al. 2017), data policy standardisation (Hrynaszkiewicz et al. 2017b), reporting standards to enhance reproducibility (McNutt 2014) and provenance tracking of research outputs and researchers, through persistent identi?cation initiatives such as ORCID (https://orcid.org/organizations/publishers/best-practices). All of which, combined, help publishers and the wider research community to make practical improvements to the communication of research that support improved data quality and reproducibility."
Reboot undergraduate courses for reproducibility,"Report, policy document or website",Large scale collaborative approaches to research can promote reproducible practices,"Both the open-science movement and the growth in online platforms for behavioural tasks and questionnaires have made it easier for psychologists to work across institutions. Using these, we can be confident we are running the same experimental procedures across sites."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Open scholarship enhances credibility and transparency in science reporting,"Recommendation: Increase open availability of materials, including pro?ciency measures, for L2 research."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Open scholarship enhances credibility and transparency in science reporting,"Open science, with its various initiatives aimed at enhancing transparency in research methods, observation, data collection, data access, and communication of ?ndings, provides important mechanisms for enhancing the validity, credibility, and reliability of scienti?c endeavors."
A manifesto for reproducible science,Review,Open scholarship enhances credibility and transparency in science reporting,"Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust. Open science refers to the process of making the content and process of producing evidence and claims transparent and accessible to others. Transparency is a scientific ideal, and adding ‘open’ should therefore be redundant."
Reproducibility of Published Research,"Report, policy document or website",Open scholarship enhances credibility and transparency in science reporting,"Open Science can transform science into ‘better’ science. Better science means making science: good: by making science more credible and replicable; for example, by addressing governance and scienti?c integrity; ef?cient: by avoiding duplication of resources and optimising the re-usability of data; and, open: by improving the accessibility ofdata and knowledge at all stages of the research cycle, and enabling text and data mining by ensuring the appropriate conditions within copyright law."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Open scholarship is an example of culture change,"Overall, there is evidence that the application of these tools can improve replicability and verifiability of scientific endeavour. Open science practices have helped to shed the light on areas where research remains unreported, and to distinguish between exploratory and hypothesis-driven analytic practices."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Open scholarship is an example of culture change,"The drive towards open, transparent and reproducible science practices appears to be a cultural shift that is gathering momentum. This has been supported by practical changes made throughout the scientific community: by scientists themselves, by publishers, and by funding and regulatory bodies via mandated policies."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,"Committing specific sections of the paper to methodological details (analysis, data, etc.) can ensure more complete reporting","In order to facilitate implementation of the guidelines, journal instructions now require that every paper should contain a data and statistical analysis sub-section within the Methods and full detail of design within each protocol described."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,"Committing specific sections of the paper to methodological details (analysis, data, etc.) can ensure more complete reporting","When comparing groups, a level of probability (P) deemed to constitute the threshold for statistical signi?cance (typically in pharmacology this is P < 0.05) should be de?ned in Methods and not varied later in Results (by presentation of multiple levels of signi?cance)."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Committing specific sections of the paper to methodological details (analysis, data, etc.) can ensure more complete reporting","In the last 10 years, more journals, and types of journal article, have emerged that publish articles that describe speci?c parts of a research project. The print-biased format of traditional research articles does not always provide suf?cient space to communicate all aspects of a research project. These new publications include journals that specialise in publishing articles that describe datasets or software (code), methods or protocols. Established journals have also introduced new article types that describe data, software, methods or protocols."
"Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation",Review,"Committing specific sections of the paper to methodological details (analysis, data, etc.) can ensure more complete reporting","To start dealing with this issue, Nature recently reinforced the acceptance criteria for manuscripts by removing the space restrictions for method sections and requesting to have external statisticians to verify the correctness of statistical tests reported in the manuscripts considered for publication. This policy change has caused other journals in the Nature family to follow the suit; and now, the NIH maintains the current list of journals and associations or societies publishing preclinical research that endorse NIH-supported principles and guidelines facilitating the reproduction of published experiments. NIH also started a new “rigor and reproducibility” web portal in order “to communicate NIH endorsed principles and guidelines [...] concerning rigor and reproducibility”."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"Journals dedicated to negative results may exist but are now closing, as they are seen as awareness raising more than venues","There has been some recognition of the need to promote null results, notably through the Journal of Negative Results in Biomedicine (JNRBM). Surprisingly, it is scheduled to cease publication by BioMed Central in September, on the grounds that its mission has been accomplished. The publisher argues that results which would previously have remained unpublished were now appearing in other journals. Many though would contend that null results are still greatly underrepresented in the literature and that there is a shortage of both resources and motivation for replication studies in general."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals dedicated to negative results may exist but are now closing, as they are seen as awareness raising more than venues","Providing a suf?cient range of journals is a means to tackle publication bias. Some journals have dedicated themselves exclusively to the publication of “negative” results, although have remained niche publications and many have been discontinued (Teixeira da Silva 2015)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Supplementary materials can solve the problem of page or word count constraints,"If this makes the manuscript too long, you can always put large tables of results in an electronic supplementary ?le."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Supplementary materials can solve the problem of page or word count constraints,"Although some journals cite space limitations, precluding provision of more extensive information, the widespread use of online supplemental information should facilitate, not hinder, precise research communication."
What you see is what you get? Enhancing methodological transparency in management research,Review,Supplementary materials can solve the problem of page or word count constraints,"Again, the availability of online supplements will hopefully facilitate the implementation of many of our recommendations, while being mindful of page limitation constraints."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Supplementary materials can solve the problem of page or word count constraints,"Implementing recommendations about transparency provided by reviewers is now greatly facilitated by the availability of Supporting Information, as is done customarily in journals such as Nature and Science. In these journals, articles are usually very short compared to those in strategic management studies. But the Supporting Information are much longer and include details about research design, measurement, data collection, data analysis, and data availability."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Supplementary materials can solve the problem of page or word count constraints,"However, the growing  availability of digital information technologies has changed this; in many cases it is now possible to give a nearly complete public record of the research process, from research materials and experimental instructions, to data and analysis scripts. These digital tools hold promise for  improving replicability and verifiability of scientific research."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Papers can include a section where authors explain which replications would be useful,"The journal Language Teaching (LT) includes an article type in which authors justify and describe speci?c replications that should be done, and indeed, 12 such articles had been published at the time of writing (see Appendix S1 in the Supporting Information online). However, the extent to which this unique initiative leads to replication is unknown. Somewhat surprisingly, in our study sample (described below), we found no published replications that followed the suggestions made, nor did we observe a general increase in the number of replications published after these article types were introduced in 2014 (with Basturkmen, 2014)."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website",External review of manuscripts can help to spot mistakes,"But amid rising concern about the quality and reproducibility of research, particularly in the biomedical sciences, a handful of European institutions have told Nature that they have now hired external companies or dedicated in-house experts to check research manuscripts."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Open peer review could help increase high quality review,"If the major reasons for irreproducibility relate to poor experimental design, statistics, reagent reliability, and data selection, then reviewers (and editorial staff) could become more capable, collectively, of competently dealing with these topics in the course of reviews, realizing that the authors are ultimately responsible for the quality and integrity of the work. Apart from use of programs to identify image manipulation or plagiarism, reviewers and editors cannot be expected to identify most instances of willful and skillfully executed fraud. And the fact that reviewers are almost never paid for their efforts limits the incentive to be maximally attentive to the task. What changes to peer review would have the greatest opportunity to positively impact the reproducibility of research over the long term? I believe the most important changes would be to increase the trend of making the reviews themselves and various versions of the manuscripts available online, and more controversially, to strongly encourage or require reviewers to sign their reviews. There are many reasons to consider supporting these practices, now employed by a limited number of both venerable and more recently launched publications. In addition to the inherent value of transparency in scienti?c understanding, this approach would disincentivize super?cial and illinformed reviews, and those based on personal or professional con?icts, while incentivizing reviews of the greatest insight and quality. Indeed, the ability to cite high quality reviews and reviewers would for the ?rst time permit the academic community to properly recognize them, impossible today since the relevant data are hidden."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Open peer review could help increase high quality review,"Another major bene?t of transparent reviews would be to provide new data enabling research in the ?eld of peer review. This could allow us to determine which review and editorial policies actually promote reproducibility and quality of published papers. Though journals may be in possession of such data, there is a very limited tradition for their contributing research in this area. I strongly believe that valuable insights into optimization of the review process to enhance reproducibility and quality would emerge."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Open peer review could help increase high quality review,"Another option is to encourage or require full external peer review and publication of protocols in journals. Funding agencies or institutional review boards peer review some research protocols, but many are not reviewed. Public review might enhance the relevance and quality of these investigations, although empirical evidence is needed. Periodic comparisons of study protocols with published results30,55 could provide useful feedback to investigators, journals, and funding agencies."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Open peer review could help increase high quality review,"The development of electronic publishing could allow for post-publication ratings and comments on scientific work. One author (RT) has helped to create such a system at PubMed, which is called PubMed Commons. It is a new feature built into PubMed, researchers can add a comment to any publication, and read the comments of others."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Peer review process can be facilitated by clear pre-defined reporting standards,"Another more general but very real challenge in many researchers’ experience concerns protracted review timelines that can often end in rejection on the basis of methodological ?aws that cannot be addressed after the data are collected. This challenge leads to a costly investment for researchers and reviewers alike, and it impacts the overall rate of scienti?c progress. In fact, one of the more frequent requests that reviewers make is for greater methodological clarity (e.g., DeKeyser & Schoonen, 2007), a problem that would be almost entirely addressed by making full materials and protocols available to the review process."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Peer review process can be facilitated by clear pre-defined reporting standards,"In addition to our desire to publish high quality science and reliable results, there is also a practical motivation for this decision. In our experience, manuscripts that do not meet these standards are invariably reviewed unfavourably, resulting in extra workload for authors and reviewers. We hope that by adopting a few basic standards we will not only increase the reliability of the results published in Neuroimage: Clinical, but also improve the ef?ciency of the review process for everyone."
What you see is what you get? Enhancing methodological transparency in management research,Review,Peer review process can be facilitated by clear pre-defined reporting standards,"Moreover, implementing as many of these recommendations as possible will reduce the chance of future retractions and, perhaps, decrease the number of“risky” submissions, thereby lowering the workload andcurrent burdenon editors and reviewers."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Post-publication peer-review allows to extend the benefits of peer-review,"The increasing recognition that questioning existing results or reporting negative results has great value has fostered innovation in discussions of reproducibility through websites such as PubPeer (https://pubpeer.com/) or the Preclinical Reproducibility and Robustness publication channel developed by F1000Research (http://f1000research. com/channels/PRR). While these online forums are in their infancy, have growing pains, and may foster problematic discussions and accusations, the quality of the comments and motives of the participants should become more re?ned and improve over time. In the future, it seems likely that journals will develop their own portals, linked to individual papers, to facilitate the updating of published results and the reporting and tracking of efforts directed at reproducing key published ?ndings."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Post-publication peer-review allows to extend the benefits of peer-review,"Second new element relates to new venues for “post publication peer review”, such as PubMed Commons, PubPeer and other sites, where participants discuss published data in variably moderated online communities. The potential ability of such venues to enhance scienti?c communication seems obvious. Although many discussions on PubPeer have questioned the validity of published data and some have even led to retractions, the fact that most discussants are anonymous has been challenged and is a topic of ongoing debate. Overall, it seems likely that a robust capacity for extended online discussion of published research will eventually advance scienti?c progress, and may hasten discovery of problems with some papers, while creating unfortunate opportunities for anonymous and misdirected harassment in some cases."
A manifesto for reproducible science,Review,Post-publication peer-review allows to extend the benefits of peer-review,"PubMed Commons and PubPeer, offer public platforms to comment on published works facilitating post-publication peer review."
A manifesto for reproducible science,Review,Post-publication peer-review allows to extend the benefits of peer-review,"Using post-publication services, reviewers can make positive and critical commentary on articles instantly, rather than relying on the laborious, uncertain and lengthy process of authoring a commentary and submitting it to the publishing journal for possible publication, eventually."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Post-publication peer-review allows to extend the benefits of peer-review,"Post-publication peer review to encourage continued appraisal of previous research, which may in turn help improve future research."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,Providing peer-reviewers with checklists and guidance can help ensure that transparency and reproducibility elements are checked.,We have prepared a summary ?ow chart of how peer reviewers may quickly triage the key areas and check for compliance with BJP’score requirements. At the same time this ?ow chart explains to authors what BJP expects from them.
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Providing peer-reviewers with checklists and guidance can help ensure that transparency and reproducibility elements are checked.,"Firstly, this implies that we have to prevent reviewers from pushing authors towards practices that we critiqued above. This includes practices that we as editors occasionally see, such as demanding a different theoretical post hoc framing for the results already present in the paper, elimination of single hypotheses on the grounds of weak empirical support, and/or because they have been tested in prior research. Sharpening hypotheses or adding hypotheses is ?ne, but not around results already present in the original version. Offering post hoc alternative hypotheses to better align with ?ndings is a natural step in the scienti?c research cycle, if done in the open."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Providing peer-reviewers with checklists and guidance can help ensure that transparency and reproducibility elements are checked.,"Secondly, beyond avoiding negative practices, reviewers should look for positive contributions to enhance the rigor of a given study. For example, reviewers may suggest additional ways to illustrate empirical ?ndings, or robustness tests that enhance the credibility of the results. At the same time, reviewers should avoid being perfectionists and, e.g., ask for tests that require nonexistent data, but use best practice in the given line of research as their benchmark when assessing how to evaluate the rigor of a paper under review."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Reviewers of replications should also go back to the initial study,"In view of this, reviewers need to be familiar with the initial study and read it alongside the replication to be able to corroborate the claimed relationships. This will have implications for authorship blinding practices in cases where there is author overlap between the initial and replication studies."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Reviewers of replications should also go back to the initial study,Recommendation: Reviewers of replications should also read the initial study that is being replicated.
What you see is what you get? Enhancing methodological transparency in management research,Review,Two-way review processes can help ensure that methodological standards are set before reviewing full manuscripts,"In fact, several journals already offer alternative review processes (e.g., Management and Organization Review, Organizational Research Methods, and Journal of Business and Psychology). The review involves a two-stage process. First, there is a “pre-registration” of hypotheses, sample size, and data-analysis plan. If this pre-registered report is accepted, then authors are invited to submit the full-length manuscript that includes results and discussion sections, and the paper is published regardless of statistical significance and size of effects."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Two-way review processes can help ensure that methodological standards are set before reviewing full manuscripts,"For one, some journals are introducing the option to submit the theory ?rst, and the empirical tests and results later (see, e.g., Comprehensive Results in Social Psychology, and Management and Organization Review; cf. Lewin et al., 2017). If the theory is accepted after a thorough review, the ?nal manuscript will be published (of course, conditional on appropriate data and the state-of-the-art empirical analyses). This approach is nascent and it is still an open question how successful this twostep approach will be. For now, we therefore suggest that JIBS take the alternative route."
A manifesto for reproducible science,Review,Preprints accelerate dissemination and add possibility for broad review,"Dissemination is now easy and can be controlled by researchers themselves. For example, preprint services (arXiv for some physical sciences, bioRxiv and PeerJ for the life sciences, engrXiv for engineering, PsyArXiv for psychology, and SocArXiv and the Social Science Research Network (SSRN) for the social sciences) facilitate easy sharing, sorting and discovery of research prior to publication. This dramatically accelerates the dissemination of information to the research community. With increasing ease of dissemination, the role of publishers as a gatekeeper is declining. Nevertheless, the other role of publishing — evaluation — remains a vital part of the research enterprise."
A manifesto for reproducible science,Review,Preprints accelerate dissemination and add possibility for broad review,"More diverse evaluation processes are now emerging, allowing the collective wisdom of the scientific community to be harnessed. For example, some preprint services support public comments on manuscripts, a form of pre-publication review that can be used to improve the manuscript."
A manifesto for reproducible science,Review,Preprints accelerate dissemination and add possibility for broad review,"Both pre- and post-publication peer review mechanisms dramatically accelerate and expand the evaluation process. By sharing preprints, researchers can obtain rapid feedback on their work from a diverse community, rather than waiting several months for a few reviews in the conventional, closed peer review process."
A manifesto for reproducible science,Review,Preprints accelerate dissemination and add possibility for broad review,"In the conventional model, peer review is done privately, anonymously and purely as a service. With public commenting systems, a reviewer that chooses to be identifiable may gain (or lose) reputation based on the quality of review. There are a number of possible and perceived risks of non-anonymous reviewing that reviewers must consider and research must evaluate, but there is evidence that open peer review improves the quality of reviews received."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Preprints accelerate dissemination and add possibility for broad review,"Sharing of preprints has been common in physical sciences for a quarter of century or more, but since the 2010s preprint servers for biosciences (biorxiv.org), and other disciplines, have emerged and are growing rapidly (Lin 2018). Journals and publishers are, increasingly, encouraging their use (Luther 2017)."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,"Exploratory analyses can still happen in pre-registrations, and they should be distinguished from planned analyses","With preregistration, the researcher is still free to explore their data, but pre-registration is a valuable tool that clearly separates the prior analysis plan from the exploratory part (De Groot, 1956/2014)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Exploratory analyses can still happen in pre-registrations, and they should be distinguished from planned analyses","I need more ?exibility in study design. Preregistration does not limit the freedom of the researcher; it only documents the ideas and plans at any given time, making the process maximally transparent. You always have the possibility of modifying your study plans, and the exact time of modi?cation and reasons for modi?cation will be documented, probably still long before ?nal data analysis when researcher degrees of freedom would come into play. The original study plan will always remain visible with its date of registration, but making failed aspects of a plan visible to others might also save them from repeating the same mistake."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Exploratory analyses can still happen in pre-registrations, and they should be distinguished from planned analyses","What if I make an unexpected discovery? Having preregistered your study does not prevent you from publishing any analysis you believe is interesting or informative. All that preregistration does is clarify what tests were formulated a priori andwhat tests were not. For instance, you may subdivide your publication into a part that covers the original analysis plan (the rigorous a priori testing part) and a second part that explores the data post hoc and yields unexpected discoveries."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,"Exploratory analyses can still happen in pre-registrations, and they should be distinguished from planned analyses","Exploratory analyses (including any deviations from planned analyses) should be clearly distinguished from planned analyses in the publication. Ideally, results from exploratory analyses should be confirmed in an independent validation data set."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Funders can require pre-registration,"Influential global organisations, including the World Health Organisation (who.int/ictrp/ network/trds/en), the World Medical Association, and the International Committee of Medical Journal Editors, have issued calls for the prospective, public registration of all clinical trials."
How to Make More Published Research True,Article,Journals can require preregistration,"There are also opportunities in grasping the importance of the key currencies. For example, registration of clinical trials worked because all major journals adopted it as prerequisite for publication, a major reference currency in the reward chain. Conversely, interesting post-publication review efforts such as PubMed Commons have so far not fulfilled their potential as progressive vehicles for evaluating research, probably because there is currently no reward for such post-publication peer review."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Embargoes on pre-registration may reduce concerns about being scooped,"Pre-registration can take a number of formats, including reviewed registrations and unreviewed registrations. Often scientists are offered the opportunity for an embargo on the duration before registrations are made public, which may reduce concerns about being “scooped”."
A manifesto for reproducible science,Review,"Not all fields are familiar with pre-registration, but fields where journals mandate it became more familiar","While pre-registration is now common in some areas of clinical medicine (due to requirements by journals and regulatory bodies, such as the Food and Drug Administration in the United States and the European Medicines Agency in the European Union), it is rare in the social and behavioural sciences."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Pre-registration of study design can help reduce analysis and publication biases,"Registered Reports also carry other bene?ts: (a) they allow peer review to inform the study at the design stage (rather than when it can be too late to improve the study); (b) they reduce questionable research practices; and (c) they accommodate any methods where data collection, coding, and analyses can be predetermined (e.g., including observations and interviews)."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Pre-registration of study design can help reduce analysis and publication biases,"Every major claim should be either accompanied by a pre-registered direct replication, or even better, other researchers from competing labs should be encouraged to replicate the original result. Direct replications are necessary even for higher-precision studies, because population differences, lab practices, etc., can easily bias an individual result. As Chambers (2017) explains, pre-registration involves defining in advance the analysis that is planned and depositing this in an embargoed repository like OSF (osf.io) or aspredicted.org. OSF time-stamps the preregistration, which serves as a transparent way to demonstrate that the analysis plan was defined before the data were collected. Pre-registering will also minimize problems like p-hacking, HARKing (hypothesizing after the results are known), and the garden-of-forking paths problem (Forstmeier, Wagenmakers, & Parker, 2017; Gelman & Loken, 2016; Simmons, Nelson, & Simonsohn, 2011) that have plagued psychology and other areas."
The possibility and desirability of replication in the humanities,Note,Pre-registration of study design can help reduce analysis and publication biases,"The most important measures to introduce in the humanities may be preregistration of studies and uploading detailed methods, data analysis plans and data sets to suitable portals (Nosek et al., 2018)."
The possibility and desirability of replication in the humanities,Note,Pre-registration of study design can help reduce analysis and publication biases,"The development and use of reporting guidelines for study protocols, publications and data sets will most likely also be important for the humanities. The idea behind these measures is that they increase transparency, limit undesirable degrees of freedom researchers have (Wicherts et al., 2016), minimize selective reporting, and ensure replicability. Evidence from biomedical, natural, and social sciences suggests that these measures can improve replication rates substantially (KNAW, 2018)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Pre-registration of study design can help reduce analysis and publication biases,"Apart from publishing addenda, how should we go about conducting replication studies? First of all, the study should be preregistered (see Section III.3) in order to solve two issues: (i) preregistration of analysis plans takes out any researcher degrees of freedom that would risk biasing the observed effect size in the direction desired by the researcher (either con?rmation of the previous ?nding or clear refutation ofit), and (ii) preregistered studies that do not make it to the stage of publication will still be accessible at a public repository, documenting the attempt and hopefully also the reason for failure."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Pre-registration of study design can help reduce analysis and publication biases,"Incentives to preregister should be particularly strong for replication studies since a study and analysis plan effectively already exist, and preregistration will clearly signal to reviewers and editors that the presentation of results has not been altered in an attempt to achieve a particular outcome."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Pre-registration of study design can help reduce analysis and publication biases,"Preregistering your study may take you a couple of days, but in the long run it will bene?t you tremendously by forcing you to think through your study plans very carefully."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Pre-registration of study design can help reduce analysis and publication biases,"While it is certainly true that clinical trials reporting negative results are more slowly reported, and sometimes not at all, efforts to improve reporting, such as the initiative embodied within alltrials.net, are likely toproduce further improvements in comprehensive reporting. It would be viewed as completely unacceptable for an academic clinical trials unit to carry out dozens of clinical studies in human subjects and only report the most promising results from studies that produced a favorable outcome."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Pre-registration of study design can help reduce analysis and publication biases,"This points to a gap that can be filled by making this information readily available to complement the publication at the time of publication. We demonstrate a way to do this with each Replication Study, where the underlying methods/data/analysis scripts are made available using https://osf.io. And unique materials that are not available can be made available for the research community to reuse, for replication or new investigations”."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Pre-registration of study design can help reduce analysis and publication biases,"Target results for publication and interpretation before data are collected, that is, state our hypotheses and predictions in a defined protocol or a binding research proposal."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Pre-registration of study design can help reduce analysis and publication biases,"Before analyzing data (and preferably before collecting them),make an analysis plan (i.e., a pre-analysis protocol), setting out how data will be analyzed; and, in the publication, show what results the protocol produced before displaying the results of any analyses deviating from the predefined protocol."
A manifesto for reproducible science,Review,Pre-registration of study design can help reduce analysis and publication biases,"Pre-registration of study protocols for randomized controlled trials in clinical medicine has become standard practice. In its simplest form it may simply comprise the registration of the basic study design, but it can also include a detailed pre-specification of the study procedures, outcomes and statistical analysis plan. It was introduced to address two problems: publication bias and analytical flexibility (in particular outcome switching in the case of clinical medicine). Publication bias, also known as the file drawer problem , refers to the fact that many more studies are conducted than published. Studies that obtain positive and novel results are more likely to be published than studies that obtain negative results or report replications of prior results. The consequence is that the published literature indicates stronger evidence for findings than exists in reality. Outcome switching refers to the possibility of changing the outcomes of interest in the study depending on the observed results. A researcher may include ten variables that could be considered outcomes of the research, and — once the results are known — intentionally or unintentionally select the subset of outcomes that show statistically significant results as the outcomes of interest. The consequence is an increase in the likelihood that reported results are spurious by leveraging chance, while negative evidence gets ignored. This is one of several related research practices that can inflate spurious findings when analysis decisions are made with knowledge of the observed data, such as selection of models, exclusion rules and covariates. Such data-contingent analysis decisions constitute what has become known as P-hacking, and pre-registration can protect against all of these."
A manifesto for reproducible science,Review,Pre-registration of study design can help reduce analysis and publication biases,"In principle, this addresses publication bias by making all research discoverable, whether or not it is ultimately published, allowing all of the evidence about a finding to be obtained and evaluated. It also addresses outcome switching, and P-hacking more generally, by requiring the researcher to articulate analytical decisions prior to observing the data, so that these decisions remain data-independent. Critically, it also makes clear the distinction"
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Pre-registration of study design can help reduce analysis and publication biases,"We recommend pre-registration of methods and analysis plans. The details to be pre-registered should include planned sample size, specific analysis tools to be used, specification of predicted outcomes, and definition of any specific ROIs or localizer strategies that will be used for analysis."
Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature,Article,Pre-registration of study design can help reduce analysis and publication biases,"Some promising avenues to resolve the current replication crisis could include the preregistration ofstudy objectives, compulsory prestudy power calculations, enforcing minimally required power levels, raising the statistical significance threshold to p < 0.001 ifNHST is used, publishing negative findings once study design and power levels justify this, and using Bayesian analysis to provide probabilities for both the null and alternative hypotheses."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Pre-registration of study design can help reduce analysis and publication biases,Pre-registration of ROIs may be particularly useful as it guards against SHARKing.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Pre-registration of study design can help reduce analysis and publication biases,"Where this information is already in the public domain, it reduces the potential for outcome switching or other sources of bias to occur in the reported results of the study (Chan et al. 2017)."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Pre-registration of study design can help reduce analysis and publication biases,"Pre-registration of study protocols is a powerful tool against some forms of publications bias. The protocol repository provides an audit trail for studies, recording what should be present in a complete publication record and thus opening the file drawer."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Pre-registration of study design can help reduce analysis and publication biases,Study registration can improve detection of publication bias by allowing the comparison of published studies with those registered.
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Pre-registration of study design can help reduce analysis and publication biases,"Sufficiently detailed registries therefore have the potential not only improve identification and discovery of research activity, but also to reduce the potential for analytic flexibility, such as outcome switching after the data has been collected. Preregistrations also allows for a clearer distinction between confirmatory, hypothesis testing research and exploratory research (and therefore more provisional findings). Exploratory results require further testing in confirmatory studies, but at present much exploratory research is presented as confirmatory."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Pre-registration of study design can help reduce analysis and publication biases,"Pre-registration of protocols and plans for analysis to counteract some of the practices that undermine reproducibility in certain fields, such as the post-hoc cherry-picking of data and analyses for publication."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Pre-registration of study design can help reduce analysis and publication biases,"Pre-registration of protocols prevents one source of irreproducibility, i.e. post-hoc cherry-picking of data, which could be important in some fields."
A manifesto for reproducible science,Review,"There are many levels of pre-registration, with some more detailed than others","The strongest form of pre-registration involves both registering the study (with a commitment to make the results public) and closely pre-specifying the study design, primary outcome and analysis plan in advance of conducting the study or knowing the outcomes of the research."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Recommendation: Encourage journal editorial boards to consider accepting Registered Report article types and, where this is not possible, to consider undertaking results-free reviews."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Language Learning is enhancing its participation in the open science movement by launching Registered Reports as an article category as of January 1, 2018. Registered Reports allow authors to submit the conceptual justi?cations and the full method and analysis protocol oftheir study to peer review prior to data collection. Highquality submissions then receive provisional, in-principle acceptance. Provided that data collection, analyses, and reporting follow the proposed and accepted methodology and analysis protocols, the article is subsequently publishable whatever the ?ndings."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"First, a manuscript with a justi?cation for the study and a full methods protocol receives peer review and, possibly, in-principle acceptance (IPA) before data collection commences. Second, IPA cannot be revoked based on the outcomes of the study, after the data have been collected. In order to implement these two core elements of Registered Reports, their submission and review have two distinct stages. In the ?rst stage, the submitted manuscript includes an introduction to a question of interest, a review of literature to justify the study, the research questions and/or hypotheses that will be addressed, and the methods."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"As of the date of writing, 66 journals across multiple disciplines have established Registered Reports (for a list of participating journals, see https://cos.io/rr). Although all Registered Reports include the two core elements mentioned above, journals may vary their speci?c guidelines as appropriate for the ?eld and aims of speci?c journals. Registered Reports at Language Learning were developed to be feasible for the broad area of language sciences and as amenable to different methodological approaches as possible. Registered Reports will follow the general ?ow described above, with speci?c author guidelines available at http://onlinelibrary.wiley.com/lang. In addition, Language Learning aims to incentivize submissions under the Registered Report category by giving preference to a Registered Report proposal for one of the annual Early Career Research Grants (available under Grant Programs at http://onlinelibrary.wiley.com/lang)."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Registered Reports allow authors to gain valuable input from experts at the point that advice is needed and can be acted upon—before data collection. From the reviewers’ point of view, their role is arguably more satisfying because they have the opportunity to identify methodological ?aws before data are gathered, and we understand that reviewers indeed ?nd this rewarding (C. Chambers, personal communication, January 16, 2017)."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"A related bene?t is that although some may believe Registered Reports extend the length of time required for publication, in actuality they typically shorten the overall research process. Stage-one review clearly adds time in the initial phase of the publication process."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Perhaps the most obvious way in which Registered Reports improve general research practice is that they vastly reduce the opportunity for publication bias given that, after IPA, reviewers must be satis?ed with the methods: Negative reviews motivated (even unconsciously) by null ?ndings or by ?ndings that are contradictory to a reviewer’s expectations cannot affect the outcome of a review (see also Button, Bal, Clark, & Shipley, 2016)."
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"They can also be made openly available before data are collected so that researchers can conduct multisite replications, thus helping to address concerns about the small sample sizes of many individual studies. Regardless of whether this transparency is at the level of published transparency (i.e., behind a journal’s paywall) or open transparency (i.e., on a sustainable open repository), this would represent a huge step toward enriching our collaborative effort, as well as improving our capacity for independent replication and validation."
The possibility and desirability of replication in the humanities,Note,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"The adoption of registered reports, like journals in the social and biomedical sciences increasingly do, would be a big leap forward (Chambers, 2015). This implies that journals decide on the basis of the introduction and methods sections before any data are collected and analyzed. Thus, the relevance of the study question and the soundness of research methods are all that matter, and reviewers and editors are not distracted by the results and conclusions. We believe that registered reports can also for empirical research in the humanities be a powerful antidote against selective reporting, which is arguably the most prominent cause of poor replication success. Taken together ensuring replicability and replication in the humanities is a shared responsibility of multiple stakeholders (Bouter, 2018)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Registered reports involve preregistration, but with a registered report, the researcher ?rst submits the study and analysis plan to a journal for review, potential revision of methods plans, and preliminary acceptance prior to conducting the study. Thus a proposed replication could be reviewed, and ifjudged meritorious, provisionally accepted independent of results. This would give the scientist who published the original study the opportunity to recommend changes to methods of the replication before it was initiated, and thus increase the quality of the replication while also reducing the opportunity for a critique, spurious or genuine, of the quality of replication."
Making sense of replications,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Moreover, the project is using the Registered Report/Replication Study approach to publish its work and results. The Registered Report details the experimental designs and protocols that will be used for the replications, and experiments cannot begin until this report has been peer reviewed and accepted for publication. The results of the experiments are then published as a Replication Study, irrespective of outcome but subject to peer review to check that the experimental designs and protocols were followed."
The challenges of replication,Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"The project is employing a Registered Report/Replication Study approach to publish its work and results. The basic idea behind this approach is that a Registered Report detailing the proposed experimental designs and protocols for each replication is peer reviewed and published after suitable revisions. Crucially, data collection cannot begin until the Registered Report has been accepted for publication. The results of the experiments are then published as a Replication Study, irrespective of the outcome, but subject to peer review to check that the designs and protocols contained in the Registered Report were followed."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Consider “results blind evaluation” of manuscripts (Locascio 2017), that is, basing our decisions about the suitability of a study for publication on the quality of its materials and methods rather than on results and conclusions; the quality of the presentation of the latter is only judged after it is determined that the study is valuable based on its materials and methods."
A manifesto for reproducible science,Review,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Some journals are trialling ‘results-free’ review, where editorial decisions to accept are based solely on review of the rationale and study methods alone (that is, results-blind)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Some journals now provide the ability to submit a ‘Registered Report’ in which hypotheses and methods are reviewed before data collection, and the study is guaranteed publication regardless of the outcome (for a list of journals offering the Registered Report format, see https://osf. io/8mpji/wiki/home/). Mass univariate testing - An approach to the analysis of multivariate data in which the same model is fit to each element of the observed data (for example, each voxel)."
How to Make More Published Research True,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Registration of randomized trials (and, more recently, registration of their results) has enhanced transparency in clinical trials research and has allowed probing of selective reporting biases, even if not fully remedying them."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Extraordinary results can make referees less critical of experiments, and with registered reports, studies can be given in principle acceptance decisions by journals before the results are known, avoiding unconscious biases that may occur in the traditional peerreview process."
Reproducibility literature analysis - a federal information professional perspective,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"For instance, journals could require authors to register reports in advance so that the study protocol and analysis plan is locked in place before data collection even begins, and scientists should be encouraged to store methods, data, and code in repositories to help other groups reproduce experiments."
How Bayes factors change scientific practice,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"The Bayesian answer helps show why pre-registered reports, such as used in Cortex and now at least 16 other journals (Chambers, Feredoes, Muthukumaraswamy, & Etchells, 2014; Wagenmakers et al., 2012; see the website Registered Reports, 2015, for regular updates) are valuable. It is not due to the magical power of guessing Nature in advance. Rather, pre-registration ensures the public availability of all results that are pre-registered, regardless of the pattern, which is important for all approaches to statistical inference, Bayesian or otherwise (Goldacre, 2013). This alone is sufficient to justify an extensive use of pre-registered reports."
How Bayes factors change scientific practice,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,Pre-registration may help us judge such things as simplicity and elegance of theory more objectively.
How Bayes factors change scientific practice,Article,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Finally, and very importantly, pre-registration helps deal with the problem of analytic flexibility (Chambers, 2015). There are generally various ways of analysing a given data set, each roughly equally justified. What should the cut off for outliers be—two or three SD, or something else? What transformation might be used, if the data look roughly equally normal with several? Should a covariate be added? Should the dependent variables be combined, or one of them dropped?"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"For one, some journals are introducing the option to submit the theory ?rst, and the empirical tests and results later (see, e.g., Comprehensive Results in Social Psychology, and Management and Organization Review; cf. Lewin et al., 2017). If the theory is accepted after a thorough review, the ?nal manuscript will be published (of course, conditional on appropriate data and the state-of-the-art empirical analyses). This approach is nascent and it is still an open question how successful this twostep approach will be. For now, we therefore suggest that JIBS take the alternative route."
On Replication in Communication Science,Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"A special issue also employed a process similar to “registered reports.” The study authors submitted a proposal to replicate the original article and a description of their planned research methods and proposed statistical analyses. The data, in most cases, had not been collected at the time of the submission. Thus, the editors and reviewers evaluated the studies without knowing the results."
On Replication in Communication Science,Editorial,Registered reports act as pre-results peer-review to avoid biasing the publication outcomes by the results,"Nosek and Lakens argued that registered reports avoid the problem ofeditors and reviewers preferring findings that are statistically significant and congenial to the existing research record. Ifa study is on an important topic and the methodology is sound, then it should not matter if the findings are statistically significant or if they confirm the existing research record, and actually may be more interesting if they are not/do not."
MAKING REPLICATION MAINSTREAM,Article in Press,Replication studies can bring light to new testable hypotheses rather than dismissing past research,"If a failed replication brings to light some factor that could potentially affect the result and that differed between the original study and the replication, conducting further investigations into the impact that this factor has on the result is a reasonable scienti?c endeavor. In short, the post hoc consideration of differences in features should lead to new testable hypotheses rather than blanket dismissals of the replication result."
MAKING REPLICATION MAINSTREAM,Article in Press,Replication studies can bring light to new testable hypotheses rather than dismissing past research,"Direct replications are not only important with regard to earlier work. They are also necessary if researchers want to further explore a ?nding that emerged in exploratory research, for example, in a pilot study. In this case, the approach would normally be to make explicit the procedure that is likely to (re)produce the ?nding observed during the exploratory phase, preregister that procedure, and then run the experiment. In such cases one would not necessarily assume that the initial procedure was an appropriate test."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Replication studies can bring light to new testable hypotheses rather than dismissing past research,Replication studies are necessary for revealing truths about relationships between the environment and behavior.
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Replication studies can bring light to new testable hypotheses rather than dismissing past research,"Thus, as a line of research progresses, we would expect to see an increasing number of different characteristics represented in the literature, which translates into broader generality of the intervention, as long as positive effects continue to be observed. The progression of research also might lead to clarifications under which the intervention produces diminished or no effects (i.e., the boundaries of an otherwise effective intervention)."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Replication studies can bring light to new testable hypotheses rather than dismissing past research,"Well-designed studies that fail to replicate a previous and well-documented effect therefore constitute a valuable discovery and provide a potential stimulus for scientific innovation. Researchers can and should examine how varied intervention procedures, dosages, contexts, and participant responses modulate intervention effects and therefore inform practical decisions of social significance. This is achieved by considering not only those studies with positive effects, but also well-designed studies without positive effects (c.f. Perone, 2018)."
Reproducibility literature analysis - a federal information professional perspective,Article,Replication studies can help to identify bias in original reports,"Replication studies also help identify potential biases in the original study and serve as a basis for confirming or disconfirming prior findings (Spector, Johnson and Young, 2014) (Camerer et al., 2016)."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,Replication studies play an important role in providing more accurate estimates of effect sizes,"But if the replication study article makes a convincing argument that (a) the methodology of the original study was flawed, (b) the selection of study participants or contexts in the original raises questions about the findings, or (c) the results are counter to other established results, this provides a more persuasive argument in the article in favor of replication."
MAKING REPLICATION MAINSTREAM,Article in Press,Replication studies play an important role in providing more accurate estimates of effect sizes,"Researcher degrees of freedom and publication bias that favors statistically signi?cant results have produced overestimations ofeffect sizes in the literature, given that the studies with nonsigni?cant effects and smaller effect sizes have been relegated to the ?le drawer (Rosenthal 1979). Ifa replication study is carried out in a ?eld characterized by such practices, then it is likely to obtain a smaller effect size, often so small as to not be distinguishable from zero when using sample sizes typical of the literature. Replication thus has an important role in providing more accurate estimates ofeffect sizes."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Replication studies play an important role in providing more accurate estimates of effect sizes,"When discussing our results, we should ‘‘bring many contextual factors into play to derive scientific inferences, including the design ofa study, the quality ofthe measurements, the external evidence for the phenomenon under study, and the validity ofassumptions that underlie the data analysis’’ (ASA statement; Wasserstein & Lazar, 2016). For example, results from exploratory studies are usually less reliable than from confirmatory (replication) studies also if their p-values were the same, because exploratory research offers more degrees of freedom in data collection and analysis (Gelman & Loken, 2014; Higginson & Munafo, 2016; Lew, 2016)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Replication studies suffer less from publication bias than original studies,"Small, early, and highly cited studies tend to overestimate effects (Fanelli, Costas & Ioannidis, 2017). Pioneer studies with inflated effects often appear in higher-impact journals, while studies in lower-impact journals apparently tend to report more accurate estimates of effect sizes (Ioannidis, 2005; Munafo, Stothart & Flint, 2009; Munafo & Flint, 2010; Siontis, Evangelou & Ioannidis, 2011; Brembs, Button & Munafo, 2013). The problem is likely publication bias towards significant and inflated effects particularly in the early stages ofa potential discovery. At a later time, authors ofreplication studies might then want, or be allowed by editors, to report results also ifthey found only negligible or contradictory effects, because such results find a receptive audience in a critical scientific discussion (Jennions & Møller, 2002). Replications therefore tend to suffer less from publication bias than original studies (Open Science Collaboration, 2015)."
The possibility and desirability of replication in the humanities,Note,Reporting guidelines allow experiment details and variations to be captured,"The development and use of reporting guidelines for study protocols, publications and data sets will most likely also be important for the humanities."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Reporting guidelines allow experiment details and variations to be captured,"It is noteworthy that in the clinical trial domain, most journals, review boards, and funding agencies now enforce mandatory registration of clinical trials, with obligatory reporting requirements, on Clinicaltrials.gov. This allows for assessment of the design, pre-speci?ed outcomes, and, ideally, results of a clinical trial, with information on results sometimes provided in advance of a peer-reviewed publication."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reporting guidelines allow experiment details and variations to be captured,"The use of the template should facilitate provision of explicit details about exercise interventions in clinical trials as a basic standard and is likely to be an important adjunct to the CONSORT,14 SPIRIT15 and TIDieR7 templates."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reporting guidelines allow experiment details and variations to be captured,"The number of checklist items that are reported in a manuscript is improved when journals required completion as part of the submission process.59 An evidence synthesis of systematic review methods has demonstrated that clear procedural details are required for the ?ndings of clinical trials to be implemented into practice,60 and the CERT would assist the completeness of systematic reviews of exercise ef?cacy."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reporting guidelines allow experiment details and variations to be captured,"The uptake of the CERT will ultimately lead to better reporting of exercise interventions in clinical trials, and enable replication in clinical practice."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Reporting guidelines allow experiment details and variations to be captured,"The value of MISEV2014 is supported by a key finding of van Deun and Hendrix reporting on the release of the EV-TRACK knowledgebase. The EV-TRACK project includes an “EV-METRIC” that can be calculated to gauge the quality of reporting of each EV-related paper. van Deun et al. found that, in the year after the release of MISEV2014, the EV-METRICs of publications overall did not improve significantly. However, publications that cited MISEV2014 had significantly higher EV-METRICs. Those who engaged withMISEV2014, then, seemed to be better equipped to report important experimental parameters."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Reporting guidelines allow experiment details and variations to be captured,"When the question was phrased, “Do you think that minimal reporting and/or methodologic requirements are important for the EV field?”, only one respondent answered “no” . There was thus near-unanimity among respondents that requirements are needed."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Reporting guidelines are increasingly mandated by journals and should be even more,"Authors should follow accepted standards for reporting methods (such as the COBIDAS standard for MRI studies), and journals should require adherence to these standards. Every major claim in a paper should be directly supported by appropriate statistical evidence, including specific tests for significance across conditions and relevant tests for interactions."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reporting guidelines are increasingly mandated by journals and should be even more,"We encourage healthcare journals and editorial groups, such as the World Association of Medical Editors and the International Committee of Medical Journal Editors, to endorse the routine use of the CERT to accompany manuscript submission and for use by reviewers in assessing trial and systematic review manuscripts submitted for publication. Journal endorsement of the CONSORT Statement has demonstrated a bene?cial effect on the completeness of reporting of the trials that they publish."
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reporting guidelines are increasingly mandated by journals and should be even more,An increasing number of journals have editorial policies stating that they will not publish trials unless detailed intervention protocols or full details are available on stable and enduring electronic and digital links.
Using the mouse to model human disease: Increasing validity and reproducibility,Review,Reporting guidelines are increasingly mandated by journals and should be even more,"As a result, journals including Disease Models &Mechanisms (DMM) recently introduced a compulsory submission checklist that asks authors to verify that they have followed best practice guidelines regarding experimental subjects, data reporting and statistics (http://dmm. biologists.org/sites/default/files/Checklist.pdf)."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Reporting guidelines are increasingly mandated by journals and should be even more,"Regulatory assessments are often hampered by a lack of reliable (eco)toxicity studies, such as for nanoparticles, pharmaceuticals, and industrial chemicals. Moreover, evaluations of recently published (eco)toxicity studies show incomplete and inadequate reporting, regarding both description of methodology and presentation of results. Promotion of proper reporting by scienti?c journals has been suggested as a solution to this problem."
How we can make ecotoxicology more valuable to environmental protection,Note,Reporting guidelines can help journals to distinguish themselves from predatory publishers,"By demonstrating a commitment to the best ecotoxicology, journals can distinguish themselves from predatory publishers (Bohannon, 2013; Kolata, 2013). Below, we provide advice and guidance to journal publishers, editors-in-chief, associate editors, reviewers and authors."
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,"Reporting guidelines do not ensure reproducibility, but ensure that reproducibility can be assessed","ISEV strongly encourages all authors to submit their experimental protocols on EV isolation and characterization to the EV-TRACK website (evtrack.org), and to consider applying additional steps if they or reviewers/editors feel that the calculated metric is low. The important consideration is not obtaining a particular metric, which after all may vary widely between basic and clinical studies; instead, the level of detail required for approved entries in EV-TRACK ensures that the transparency and reproducibility of procedures can be assessed. Furthermore, the knowledgebase can be revised and expanded as technologies and techniques develop, with input and assistance from the community."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Reporting guidelines do not ensure reproducibility, but ensure that reproducibility can be assessed",Journal submission guidelines can increase transparent research practices by authors (Giofrè et al. 2017; Nuijten et al. 2017).
Reproducibility literature analysis - a federal information professional perspective,Article,Standards contribute to improved research practices,"Standards comprise the fundamental reference for a system of weights and measures, against which all other measuring devices are compared. Standards contribute to improved research practices and promote positive change (Capes-Davis and Neve, 2016)."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Standards contribute to improved research practices,"There is evidence that completion of checklists can improve the comprehensiveness and quality of reporting in research articles. The completion of a journal-mandated checklist has been reported to improve reporting quality of key methodological aspects of studies in preclinical biomedical research. Similarly, in psychology, the implementation of journal guidelines for reporting of statistics was found to be linked to improved statistical reporting, although in both cases significant room for improvement remains."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Standards contribute to improved research practices,Reporting guidelines to help deliver publications that contain the right sort of information to allow other researchers to reproduce results.
"Introducing Registered Reports at Language Learning: Promoting Transparency, Replication, and a Synthetic Ethic in the Language Sciences",Editorial,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Over recent years, Language Learning has been promoting several open science practices, for example, by requiring the reporting of effect sizes (Ellis, 2000); encouraging authors to make materials and data fully transparent by holding them in a publicly accessible repository, such as IRIS (Marsden, Mackey, & Plonsky, 2016; https://www.iris-database.org) or other publicly accessible databases, including the Open Science Framework (OSF; https://osf.io) and Dataverse (https://dataverse.org); producing guidelines for transparent reporting ofquantitative studies (Norris, Plonsky, Ross, & Schoonen, 2015);"
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","The first step to recover EVs is to harvest an EV-containing matrix, such as fluid from tissue culture or from an organismal compartment. During this pre-analytical phase, an extended constellation of factors, including characteristics of the source, how the source material is manipulated and stored, and experimental conditions, can affect EV recovery. Therefore, it is crucial to plan collection and experimental procedures to maximize the number of known, reportable parameters, and then to report as many preanalytical parameters as are known."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","New procedures by journals to enhance quality of manuscripts and reviews Such changes might include rigorous checklists to promote appropriate design features, enhanced statistical assessment by journals, and encouragement or requirement that raw data be provided during submission to be available online."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Journals also have an important role to play, Nosek added, by providing incentives for replications. He highlighted the TOP Guidelines (http://cos.io/top/) from the Centre for Open Science, which specifies replication as one of its key eight standards for advancing transparency in research and publication. “TOP is gaining traction across research communities with about 3,000 journal signatories so far”, said Nosek. “COS also offers free training services for doing reproducible research, and fosters adoption of incentives that can make research more open and reproducible, such as badges to acknowledge open practices”, Another COS initiative called Registered Reports (http://cos.io/rr/) promotes a publishing model where peer review is conducted prior to the outcomes of the research being known."
Using the mouse to model human disease: Increasing validity and reproducibility,Review,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","In summary, setting rigorous standards for carrying out and reporting mouse work will help to improve the likelihood of reproducibility. Validation of the model, proper use ofcontrols and attention to rigorous experimentation and statistics are fundamental to increase the translational impact of animal experiments."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)",Revealing and centralizing methodologic details are part of successful standardization.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Scholarly publishers can help to increase data quality and reproducible research by promoting transparency and openness. Increasing transparency can be achieved by publishers in six key areas: (1) understanding researchers’ problems and motivations, by conducting and responding to the ?ndings of surveys; (2) raising awareness of issues and encouraging behavioural and cultural change, by introducing consistent journal policies on sharing research data, code and materials; (3) improving the quality and objectivity of the peer-review process by implementing reporting guidelines and checklists and using technology to identify misconduct; (4) improving scholarly communication infrastructure with journals that publish all scienti?cally sound research, promoting study registration, partnering with data repositories and providing services that improve data sharing and data curation; (5) increasing incentives for practising open research with data journals and software journals and implementing data citation and badges for transparency; and (6) making research communication more open and accessible, with open-access publishing options, permitting text and data mining and sharing publisher data and metadata and through industry and community collaboration. This chapter describes practical approaches being taken by publishers, in these six areas, their progress and effectiveness and the implications for researchers publishing their work."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)",Scholarly publishers have a responsibility to promote reproducible research (Hrynaszkiewicz et al. 2014) but are more able to in?uence the reporting of research than the conduct of research. Transparency is a precursor to reproducibility and can be supported by journals and publishers (Nature 2018).
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Implementation of greater transparency in, reporting and reuse potential of, research by publishers can be achieved in several ways: 1. Understanding researchers’ problems and motivations 2. Raising awareness and changing behaviours 3. Improving the quality, transparency and objectivity of the peer-review process 4. Better scholarly communication infrastructure and innovation 5. Enhancing incentives 6. Making research publishing more open and accessible."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Journal policies and guides to authors include large amounts of information covering topics from manuscript formatting, research ethics and con?icts of interest. Many journals and publishers have, since 2015, endorsed – and are beginning to implement – the Transparency and Openness Promotion (TOP) guidelines. The TOP guidelines are a comprehensive but aspirational set of journal policies and include eight modular standards, each with three levels of increasing stringency including transparency in data, code and protocols (Nosek et al. 2014)."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","Using more formal, data sharing (data use) agreements can improve authors’ willingness to share data on request (Polanin and Terzian 2018), and guidelines on depositing clinical data in controlled-access repositories have been de?nedbyeditors andpublishers, as a practical alternative to public data sharing (Hrynaszkiewicz et al. 2016). Publishers are also supporting editors to improve policy effectiveness and consistency of implementation (Graf 2018)."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","But, as described earlier, even if the necessary knowledge onhowto enhance transparency is readily available, authors need to be motivated to use that knowledge. So, to improve authors’ motivation to be transparent, Table 8 includes recommendations for journals and publishers, editors, and reviewers on howtomake transparencyamore salient requirement for publication. Paraphrasing Steve Kerr’s(1975) famous article, it would be na¨?ve to hope that authors will be transparent if editors, reviewers, and journals do not reward transparency—even if authors know and have the ability to be more transparent."
Reproducibility and Research Integrity,Note,"Journals and societies should ensure rigorous standards for carrying out and reporting work (e.g., TOP guidelines)","While required disclosures depend on the nature of research one is conducting, some general types of information which should be disclosed include: the research design (e.g. controlled trial, prospective cohort study), methods (e.g. blinding, randomization), procedures, techniques, materials, equipment, data analysis methods and tools (including computer programs or codes), study population (for animals or humans), exclusion and inclusion criteria (for animals and humans), ethics committee approvals (if appropriate), theoretical assumptions and potential biases, sources of funding, and conflicts of interest (Landis et al 2012, Nature 2014b, McNutt 2014, Nature 2015, Elliott and Resnik 2015, Rooney et al 2016, Morgan et al 2016). Additional disclosures may need to occur after the research is published to allow independent scientists to obtain information needed to reproduce experiments, reanalyze data, or develop new hypotheses or theories related to the research."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Transparency does not mean less exploration and tests of methods, but just more honesty about the processes used","Our discussion of transparency, or lack thereof, does not mean that we wish to discourage discovery- and trial-and-error-oriented research. To the contrary, epistemological approaches other than the pervasive hypothetico-deductive model, which has dominated management and related fields since before World War II (Cortina, Aguinis, & DeShon, 2017a), are indeed useful and even necessary. For example, inductive and abductive approaches can lead to important theory advancements and discoveries (Fisher & Aguinis, 2017; Hollenbeck & Wright, 2017; Murphy & Aguinis, 2017). Sharing our perspective, Hollenbeck and Wright (2017) defined “tharking” as “clearly and transparently presenting new hypotheses that were derived from post hoc results in the Discussion section of an article. The emphasis here is on how (transparently) and where (in the Discussion section) these actions took place” (p. 7). So, we are not advocating a rigid adherence to a hypotheticodeductive approach but, rather, epistemological and methodological plurality that has high methodological transparency."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Transparency in research management helps the issue of biased reporting,"Transparency of the research management process helps the reporting of the practices and methods that are then not necessarily documented in the publication of results, and in
the best case may also help document, report and then publish negative results – an elusive but highly valuable asset of the research process. There is great epistemic value in negative results, both for guidance and for proofing future studies."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Transparency reduces the need for trust,"Transparency is especially needed for research conducted across different sites and institutions, that rely heavily on materials, expensive machinery and complex protocols to generate data, exposed to unforeseen circumstances and with large staff turnover, among other things."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Improving reporting standards to enhance transparency of published research,"Additional efforts have also tried to improve the disclosure and visible indexing of information related to transparency and reproducibility. In 2017, PubMed, which is run by the United States National Library ofMedicine (NLM) at the National Institutes ofHealth (NIH), started including funding and conflicts ofinterest statements with study abstracts. Although this information is often disclosed in the full text ofjournal articles, many research consumers do not have a subscription to all ofthe journals catalogued in PubMed. To our knowledge, it is unknown whether information about key transparency indicators is easily accessible to the general public on PubMed and whether this information was available prior to 2017. These and other recent open science initiatives, or even simply the wider sensitization of the scientific community over the past 20 years, may have improved the reproducibility and transparency ofthe biomedical research over the last few years."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Improving reporting standards to enhance transparency of published research,"There are numerous causes of irreproducibility and suboptimal data quality. Some of these causes relate to how research is conducted and supervised, and others relate to how well or completely research is reported. Data quality and reproducibility cannot be assessed without complete, transparent reporting of research and the availability of research outputs which can be reused."
Reproducibility literature analysis - a federal information professional perspective,Article,Improving reporting standards to enhance transparency of published research,"Simply put, transparency means ‘provable to the outside’ (Bartling and Fecher, 2015)."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Improving reporting standards to enhance transparency of published research,"A prerequisite to enable a thorough evaluation is proper reporting of the methods used and the results obtained. However, transparent reporting is not in itself critical for evaluating reliability but does allow for fast and easy review of the data."
What you see is what you get? Enhancing methodological transparency in management research,Review,Improving reporting standards to enhance transparency of published research,"We conceptualize lack oftransparency as a “research performance problem” because it masks fraudulent acts, serious errors, and questionable research practices, and therefore precludes inferential and results reproducibility."
What you see is what you get? Enhancing methodological transparency in management research,Review,Improving reporting standards to enhance transparency of published research,"We focus on the relative lack of methodological transparency because it masks outright fraudulent acts (as committed by, for example,Hunton&Rose, 2011and Stapel & Semin, 2007), serious errors (as committed by, for example, Min & Mitsuhashi, 2012; Walumbwa, Luthans, Avey, & Oke, 2011), and questionable research practices (as described by Banks, et al., 2016a)."
What you see is what you get? Enhancing methodological transparency in management research,Review,Improving reporting standards to enhance transparency of published research,"Without a clear discussion of the changes made, readers may doubt conclusions, as it might appear that authors changed the scales to obtain the desired results, thereby reducing inferential reproducibility."
"Best Practices for Transparent, Reproducible, and Ethical Research","Report, policy document or website",Improving reporting standards to enhance transparency of published research,"Transparent research enables correct (null) hypothesis testing, increases visibility and discoverability of research, and shifts attention away from statistically significant results to the quality and relevancy of the research itself."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Improving reporting standards to enhance transparency of published research,"Greater openness and transparency – in terms of both methods and data, including publication of null or negative results."
How we can make ecotoxicology more valuable to environmental protection,Note,Transparent reporting allows readers to fairly judge the quality of the paper,"Only when studies are reported in a transparent and detailed way is it possible for a reader (e.g., a regulator who wants to use the results in the publication to ensure adequate protection of the environment) to judge the quality/reliability of the paper."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,"Science tends to adopt an adversarial style where only one truth exists, but it should in fact adopt a constructive criticism approach","There are two main strategies with which this problem of conflicting evidence can be approached. The first one assumes that one side is right and, therefore, the other one must be necessarily wrong. Accordingly, either all the positive or all the negative results must be a product of bias, self-deception, unsound statistical analysis, faulty logic or some other type of error. Such an approach has much in common with the adversarial style dominating legal systems and political life in many countries. It has many advantages: it forces the opponents to formulate clear hypothesis, sharp arguments and convincing justifications. It focuses the debates on the most relevant points and can be more entertaining to readers, viewers and listeners than the sometimes rather dull consensus. However, in science, as in politics, the adversarial approach has also its downsides, often just the other side of what we could justly consider as its advantage. The focused nature of the arguments can mean a narrowing of the perspective, leaving out the sheer diversity of the phenomena in question."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,"Science tends to adopt an adversarial style where only one truth exists, but it should in fact adopt a constructive criticism approach","An alternative approach is constructive scepticism. It is constructive in assuming that both positive and negative results are genuine, but reflect possible differences in experimental designs, definitions of the phenomena in question, examined populations and the environment in which the research is being conducted. As true scepticism, it critically examines the evidence and the arguments on both sides. It sees as its main objective not to prove one’s point but to try to understand and explain current puzzles and contradictions. It does not deny opposing points of view but tries to transcend them. It can build on a long tradition of dialectics, from Ancient Greece (and indeed Ancient India), through medieval universities up to the present day. In the dialectical method, the thesis and antithesis is followed by a synthesis, integrating both and, hopefully, coming closer to the truth."
How to Make More Published Research True,Article,Adopting reproducibility approaches that have worked in specific disciplines may help in any subject area,"To make more published research true, practices that have improved credibility and efficiency in specific fields may be transplanted to others which would benefit from them—possibilities include the adoption of large-scale collaborative research; replication culture; registration; sharing; reproducibility practices; better statistical methods; standardization of definitions and analyses; more appropriate (usually more stringent) statistical thresholds; and improvement in study design standards, peer review, reporting and dissemination of research, and training of the scientific workforce."
Reproducibility literature analysis - a federal information professional perspective,Article,Adopting reproducibility approaches that have worked in specific disciplines may help in any subject area,"Common past practice may have seen librarians contributing to the scientific endeavor in very limited ways, such as assisting with initial literature access and reviews, or as cataloging and preserving reported scientific results. However, the evolution of modern scientific research has, as discussed above, opened up a number of roles for library, information, and data professionals throughout the entire scientific research lifecycle. Our participation can positively impact scientific reproducibility and replicability. First, we must understand the issues, and this paper is one contribution to develop that understanding. Next we should apply that comprehension to aid our colleagues across research disciplines."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"In qualitative research, the specific details of the research setting should be noted","Future qualitative research should provide detailed information regarding contextual issues regarding the research setting (e.g., power structure, norms, heuristics, culture, economic conditions)."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"In qualitative research, the specific details of the research setting should be noted","Position of researcher along the insider-outsider continuum. Future qualitative research should provide detailed information regarding the researcher's position along the insider-outsider continuum (e.g., existence of a pre-existing relationship with study participants, the development of close relationships during the course of data collection)."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"In qualitative research, the specific details of the research setting should be noted","Documenting interactions with participants. Future qualitative research should document interactions with participants (e.g., specify which types of interactions led to the development of a theme)."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"In qualitative research, the specific details of the research setting should be noted",Management of power imbalance. Future qualitative research should report and describe whether power imbalance exists between the researcher and the participants and how it has been addressed
How to Make More Published Research True,Article,Any solutions to the reproducibility crisis should be regualrly reviewed or evaluated,"Whatever solutions are proposed should be pragmatic, applicable, and ideally, amenable to reliable testing of their performance."
Digital tools and services to support research replicability and verifiability,"Report, policy document or website",Collaboration and input from peers,"Collaboration and data sharing across research groups is one way to address issues of low power, and this can be supported by digital platforms."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Collaboration and input from peers,"Better use of input and advice from other experts, for example through collaboration on projects, or on parts of projects."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Collaboration and input from peers,Advice from experts in statistics and experimental design being made more widely available and sought at the beginning of a project.
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Collaboration and input from peers,"The need for a global approach, in all senses: funding bodies, research institutions, publishers, editors, professional bodies, and individual researchers must act together to identify and deliver solutions – and they will need to do so at an international level. Cultural change will take time."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Collaboration and input from peers,"The environment in which research is carried out is a critical factor in addressing reproducibility. Collaborative working can help address some problems that affect reproducibility, such as underpowered studies."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Communicate openly and honestly about reproducibility issues,"Talking openly within the research community about challenges of delivering reproducible results. Scientists and science communicators, including press officers, have a duty to portray research results accurately."
Raising research quality will require collective action,"Report, policy document or website","Doctoral schools need to introduce training on transparency, and supervisors should discuss reproducibility with their students","Consider the productivity gains if competence in the programming language R — which can be used to run statistical analyses across a broad range of areas — was a given. Research would become more efficient, like the railways did after adopting a common standard for track gauge."
Research integrity nine ways to move from talk to walk,"Report, policy document or website","Doctoral schools need to introduce training on transparency, and supervisors should discuss reproducibility with their students","The largest universities in Denmark now mandate integrity training for PhD students, and offer access to designated counsellors across career stages. Both junior and senior researchers have dedicated people they can talk to."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Doctoral schools need to introduce training on transparency, and supervisors should discuss reproducibility with their students","Our article can be used as a resource to address the “research performance problem” regarding low methodological transparency. Specifically, information in Tables 3–7 can be used for doctoral student training and also for researchers as checklists for how to be more transparent regarding judgment calls and decisions in the theory, design, measurement, analysis, and reporting of results stages of the empirical research process."
Reproducibility and Research Integrity,Note,"Doctoral schools need to introduce training on transparency, and supervisors should discuss reproducibility with their students","To help students and trainees better understand how to promote reproducibility in their work, courses in research methodology and RCR should include sections devoted to the importance of reproducibility and issues that can impact it, such as experimental design, recordkeeping, biological variability, data analysis, and transparency (Titus et al 2008, Shamoo and Resnik 2015, National Institutes of Health 2016). The NIH has developed some online reproducibility training modules which are available to the public (National Institutes of Health 2015). The modules address topics such as blinding, randomization, transparency, recordkeeping, bias, sample size, and data analysis. Several World Conferences on Research Integrity (2016) have provided forums for international discussion of ethics issues in research and science education, including those that impact reproducibility."
Reproducibility and Research Integrity,Note,"Doctoral schools need to introduce training on transparency, and supervisors should discuss reproducibility with their students","Researchers should also informally discuss reproducibility issues with their students and trainees as part of the mentoring process. For example, if a student or trainee is having difficulty repeating an experiment, the mentor should help him or her to understand what may have gone wrong and how to fix the problem. The mentor may also be able to share stories of his or her own experimental successes and failures with the student to illustrate reproducibility concepts. Failure to replicate an experiment need not be a total loss but can be an opportunity to teach students about principles of good science (Firestein 2016)."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website",Researchers should be able to discuss sloppy science with their supervisors without risking thier careers,"Other organizations have decided to do publication checks internally. After the Beatson Institute in Glasgow, UK, had to deal with a retraction in 2012, it hired a dedicated integrity offer, former molecular biologist Catherine Winchester, to check all papers destined for publication by eye. “It took only a short time for the more junior scientists to shed their fear that they were being policed, but there was immediate buy-in from senior PIs,” she says. “Now everyone is really grateful for the service.”"
"Data management plans, the missing perspective",Note,Reprioritising data curation in the research process,The low percentage of funders requiring DMPs versus data shar-
"Data management plans, the missing perspective",Note,Reprioritising data curation in the research process,"ing plans indicates a greater emphasis on sharing and reuse of data than on data collection and processing. This disproportionate emphasis should be reexamined and equal weight given to upstream processes that determine data quality and support research results. The least-required items in a data management plan were aspects about data collection and processing. This is unfortunate, because data collection including the original observation and processing are not only large determinants of data quality, they are the determinants for which the opportunity to intervene is lost as time moves on."
"Trust, but Verify II: A Practical Guide to Chemogenomics Data Curation",Review,Reprioritising data curation in the research process,Data curation is especially critical for computational modelers because their success depends inherently on the accuracy of the data used for model development.
Our path to better science in less time using open data science tools,Article,Reprioritising data curation in the research process,"Our team developed conventions to standardize the structure and names of files to improve consistency and organization. Along with the GitHub workflow (see ‘Collaboration’), having a structured approach to file organization and naming has helped those within and outside our team navigate our methods more easily. We organize parts of the project in folders that are both RStudio ‘projects’ and GitHub ‘repositories’, which has also helped us collaborate using shared conventions rather than each team member spending time duplicating and organizing files."
"Most computational hydrology is not reproducible, so is it really science?",Note,Reprioritising data curation in the research process,"Second, the complexity of many hydrologic models and data analysis codes used today makes it simply infeasible to report all settings that can be adjusted (e.g., initial conditions and parameters) in publications—a point recognized recently in a joint editorial published in ?ve hydrology journals [Bl€oschl et al., 2014]. Transparency across hydrology is especially important given research builds on previous research. For example, being able to evaluate how ‘‘tidied up’’ data sets have been created by explicitly showing all of the assumptions made will lead to bene?ts in interpreting where and why subsequent models that are built upon such data sets fail."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,Data exclusion decisions and data analyses must be documented,"In summary, outliers should be included in a data set unless a prede?ned and defensible set of exclusion criteria can be generated and applied."
What you see is what you get? Enhancing methodological transparency in management research,Review,Data exclusion decisions and data analyses must be documented,"While missing data and nonresponses are rarely the central focus of a study, they usually affect conclusions drawn from the analysis (Schafer & Graham, 2002). Moreover, given the variety of techniques available for dealing with missing data (e.g., deletion, imputation), without precise reporting of results of missing data analysis and the analytical technique used, others are unable to judge whether certain data points were excluded because authors did not have sufficient information or because excluding the incomplete responses supported the authors’ preferred hypotheses (Baruch & Holtom, 2008). In short, low inferential reproducibility is virtually guaranteed if this information is absent."
What you see is what you get? Enhancing methodological transparency in management research,Review,Data exclusion decisions and data analyses must be documented,"All analytic techniques include assumptions (e.g., linearity, normality, homoscedasticity, additivity), and many available software packages produce results of assumptions tests without the need for additional calculations on the part of researchers. While the issue of assumptions might be seen as a basic concept that researchers learn as a foundation in many methodology courses, most published articles do not report whether assumptions were assessed (Weinzimmer,Mone, &Alwan, 1994)."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Data exclusion decisions and data analyses must be documented,"Saturation point. Future qualitative research should identify the saturation point and describe the judgment calls the researcher made in defining and measuring it. The saturation point occurs when there are no new insights or themes in the process of collecting data and drawing conclusions (Bowen, 2008; Strauss & Corbin, 1998). Authors should therefore report how they defined the saturation point and how they decided that it was reached."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Data shared should be formatted according to community standards and shared in open repositories,"Unthresholded statistical maps and the raw data would be shared via appropriate community repositories, and the shared raw data would be formatted according to a community standard, such as the Brain Imaging Data Structure (BIDS)73, and annotated using an appropriate ontology to enable automated meta-analysis."
"Most computational hydrology is not reproducible, so is it really science?",Note,Data shared should be formatted according to community standards and shared in open repositories,"We argue that in order to advance and make more robust the process of knowledge creation and hypothesis testing within the computational hydrological community, we need to adopt common standards and infrastructures to: (1) make code readable and reuseable; (2) create well-documented work?ows that combine reuseable code together with data to enable published scienti?c ?ndings to be reproduced; (3) make code and work?ows available and easy to ?nd through the use of code repositories and creation of code metadata; (4) use unique persistent identi?ers (e.g., DOIs) to reference reuseable code and work?ows, thereby clearly showing the provenance of published scienti?c ?ndings."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Encouraging data and code sharing,"Journal data sharing policies do not guarantee that investigators will actually share their data [11]. In PLOS ONE, in which the policy states that the preferred methods ofdata sharing is deposition in a repository, only 20% ofthe data availability statements indicate that data are deposited in a repository. In another empirical evaluation of data from randomized trials published in PLOS Medicine and the BMJ (both ofwhich require full data availability as a prerequisite to publication), only 46% ofthe data sets could be retrieved. These findings suggest that more stringent policies or new incentives may be necessary to increase data sharing practices."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Encouraging data and code sharing,"Similarly, protocol or dataset registration and deposition is likely to become widely adopted only with similar incentives—eg, if a prerequisite for funding and publication of research reports."
Open is not enough,"Report, policy document or website",Encouraging data and code sharing,"Embrace openness whenever possible Identify materials that can be shared publicly, publish them in trusted repositories, link and reference them to contextualize them. Consider using embargo periods for sensitive datasets or materials. Share restricted data or work in progress among your collaborators."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Encouraging data and code sharing,"Publishers can help enable repositories to be used, more visible, and valued, in scholarly communication. This is bene?cial for researchers and publishers, as connecting research papers with their underlying data has been associated with increased citations to papers (Dorch et al. 2015; Colavizza et al. 2019)"
How we can make ecotoxicology more valuable to environmental protection,Note,Encouraging data and code sharing,"We acknowledge that the vast majority of the ecotoxicology that is done is a result of funding, whether from government, industry, or other sources, such as NGOs. As bodies that decide which work will be performed, it is vital they ensure that they strive to support the highest quality science, and that it is reported properly. The bene?tto funders will be the creation of data that will allow for the widest reach by all users, enhancing the value of limited ?nancial resources. To facilitate this, funders should:
1. Create minimum requirements for conducting studies and reporting of data prior to funding approval.
2. Provide funding so that open source publishing and repositories for raw data can be maintained.
3. Promote ethics and integrity among grantees."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Encouraging data and code sharing,"A different way to facilitate dissemination of SCRD studies that do not produce expected effects is to create contingencies that encourage the open sharing of research protocols, software, and data, including data that reflect absence of experimental control. Uploading research protocols, software, and data to open access repositories may promote visibility of studies that might otherwise be subjected to publication bias or the file drawer effect."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Encouraging data and code sharing,"Greater openness and transparency is critical to addressing irreproducibility in research. Changes in policies by funders and publishers could drive notable changes in data-sharing practices. Pre-registration of protocols prevents one source of irreproducibility, i.e. post-hoc cherry-picking of data, which could be important in some fields."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Datasets and results should be version-numbered,All data sets and results would be assigned version numbers to enable explicit tracking of provenance. Automated quality control would assess the analysis at each stage to detect potential error.
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Datasets and results should be version-numbered,"The paper would be distributed along with the full codebase to perform the analyses and the data necessary to reproduce the analyses, preferably in a container or virtual machine to enable direct reproducibility."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Effect sizes must be justified and calculated correctly,"Many researchers, such as Cohen (1962), and Gelman and Carlin (2014), have pointed out that a prospective power analysis should be conducted before we run a study; after all, why would one want to spend money and time running an experiment where the probability of detecting an effect is 30% or less?"
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Effect sizes must be justified and calculated correctly,"It is of course possible to publish more informative studies by simply running higher-power experiments. But how can we decide what constitutes a higher-powered study? Frequentist statistics has several proposals for sequential testing (e.g., Frick, 1998), which avoid running unnecessarily large numbers of participants."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Effect sizes must be justified and calculated correctly,"Whenever researchers use NHST they should justify its use, and publish pre-study power calculations and effect sizes, including negative ?ndings. Hypothesis-testing studies should be pre-registered and optimally raw data published. The current statistics lite educational approach for students that has sustained the widespread, spurious use of NHST should be phased out."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Effect sizes must be justified and calculated correctly,"Guideline 4: Re?ections on effect sizes are included, reporting and discussing whether the Journal of International Business Studies effects (the coef?cients and, if appropriate, marginal effects) are substantive in terms of the research question at hand."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Effect sizes must be justified and calculated correctly,"Guideline 4a: When discussing effect size, authors should take the con?dence interval associated with the estimated coef?cient into account as well as the minimum and maximum effect (not just one standard deviation above and below the mean), thus providing a range of the strength of a particular relationship. This may be done graphically for more complex models Guideline 4b: When discussing effect sizes, where possible and relevant, authors should compare the range of the effect size of the variable of interest with other variables included in the regression mod"
Reproducibility and Research Integrity,Note,Effect sizes must be justified and calculated correctly,"For example, statistical power analysis can help to ensure that one’s sample is large enough to detect a significant effect in biomedical, physicochemical, or behavioral research; randomization and blinding can control for bias in clinical trials or animal experiments; and auditing of data and other research records can help reduce errors and inconsistencies in many fields of science (Shamoo and Resnik 2015, National Institutes of Health 2016, Shamoo 2016)."
Open is not enough,"Report, policy document or website",Enable reuse with open licences,"Enable liberal and fair reuse Licensing and crediting is crucial. All our open scholarly materials (data, code, documentation, papers) are accompanied by liberal licenses and data/software citation recommendations and we do encourage other scientific disciplines to adopt the same principle."
The science institutions hiring integrity inspectors to vet their papers,"Report, policy document or website",Establish pre-submission checks,"It’s possible that pre-submission checks could be an optional item on these certificates. For some Leibniz institutes, he adds, it could be a way “to protect scientists from being in danger of scientific misconduct”."
CRED: Criteria for reporting and evaluating ecotoxicity data,Article,Peer review process can be facilitated by clear pre-defined reporting standards,"Finally, we stress the following general recommendations for risk assessors: evaluate reliability and relevance in a systematic way, such as by using the checklist provided in the Supplemental Data; document your choices and the rationale behind them, highlight uncertainties, and stay critical regarding bias coming from the background of the authors; contact the authors if more details are needed; derive other effect values (e.g., EC10) when data are available; make sure all derivations are consistent; use an expert group for review if possible; and let an agency from another country review your derivation."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,External validation of research is essential,"External validation by independent teams is essential, yet infrequent in many specialties."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Full protocols for analysis should be developed, followed by researchers and shared publicly","Make publicly available the full protocols, analysis plans or sequence of analytical choices, and raw data for all designed and undertaken biomedical research. Monitoring—proportion of reported studies with publicly available (ideally preregistered) protocol and analysis plans, and proportion with raw data and analytical algorithms publicly available within 6 months after publication of a study report."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Full protocols for analysis should be developed, followed by researchers and shared publicly","The extent to which research is done on the basis of a rudimentary protocol or no protocol at all is unknown, because even when protocols are written, they are often not publicly available. Consequently, researchers might improvise during the conduct of their studies, and place undue emphasis on chance findings. Although some improvisation is unavoidable because of unanticipated events during a study (eg, an unexpectedly high dropout rate, or unpredicted adverse events), changes in the research plan are often poorly documented and not present in formal data analyses (eg, non-response and refusal data might be neither reported nor used to adjust formal uncertainty measures)."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Full protocols for analysis should be developed, followed by researchers and shared publicly","Clinical trials and other studies that are not exploratory research should be done in accordance with detailed written plans, with advance public deposition of the protocol."
Open is not enough,"Report, policy document or website","Full protocols for analysis should be developed, followed by researchers and shared publicly","Capture your content. What are the core elements that need to be included in a reproducible analysis package? What needs to be documented? Capture your analysis assets if they are located in volatile places or the location of your assets if they are stored safely. Think about input data, configurations and parameters, as well as analysis software and its dependencies. Make sure to preserve the computational environment and runtime external dependencies. Use established community platforms or talk to teams in your institution to check how you can safeguard your research. Capture your workflows How did you arrive at the results? Preserve your computational workflow steps. Automate your analysis and make it scriptable instead of using interactive user interfaces. Use a structured computational workflow engine that can run your analysis in a suitable computational environment."
Reboot undergraduate courses for reproducibility,"Report, policy document or website","Full protocols for analysis should be developed, followed by researchers and shared publicly","It works best if a PhD student or postdoc develops the primary research question for the undergraduates to tackle, drafts a ‘bare-bones’ study protocol and manages the study. Over the UK summer break, this protocol is circulated to undergraduate students (usually from two to five students at each institution), and each of them plans a secondary research question and suitable method."
"Data management plans, the missing perspective",Note,Funders should require the maintenance of DMPs,"Instead, when conceptualized and operationalized as comprehensive documentation of the data lifecycle for a study, a data management plan is a powerful tool and an integral component of the data management quality system. The best practice would be for research funders to require maintenance of the data management plan following award and during the active phase of a study."
Detecting and avoiding likely false-positive findings – a practical guide,Article,Get researchers to replicate their own work,"A partial solution to this dilemma could come from researchers replicating their own ?ndings. This eliminates the quality issue as well as issues related to dissimilarity inmaterials or methods. A simple and cheap way ofgetting this started was suggested by one ofour colleagues, JarrodHad?eld (Had?eld, 2015). He proposed that researchers running long-term studies could publish addenda to their previous publications, declaring in a one-page publication that their original ?nding did or did not hold up in the data ofthe following years (after the publication), and comparing the effect sizes between the original data and the newer data. This would be a quick way ofproducing another publication, and it would be enormously helpful for the scienti?c ?eld. This may also relax the feeling of stigma when something does not hold up to future evaluation. Admitting a failure to replicate could actually be perceived as a signal of a researcher’s integrity and be praised as a contribution to the scienti?c community. For grant applications, funding agencies could even speci?cally ask for visible signs of such integrity rather than exclusively focussing on metrics ofproductivity and impact."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Get researchers to replicate their own work,"Reproducibility checks are nearly non-existent, with few exceptions, and so any improvement would be positive."
Reproducible Research A Retrospective,"Report, policy document or website",Get researchers to replicate their own work,"He noted in particular the bene?ts of reproducibility to the original authors: ”It may seem strange to put the author’s own name at the top of the list to whom we wish to provide the reproducible research, but it often seems that the greatest bene?ciary of preparing the work in a reproducible form is the original author!” It is equally notable that the public was listed last; all of the other constituencies mentioned would likely exist within the small orbit of an individual investigator. In Claerbout’s discussion, the primary focus is on improving the transparency and productivity of the lab itself, given that much time can be lost attempting to re-create past ?ndings for the sole purpose of understanding what was previously done."
Reproducibility literature analysis - a federal information professional perspective,Article,Labs should conduct regular audits and spot checks,"Implementing quality assurance may involve a variety of checks for data completeness, validity, consistency, precision, and accuracy, among other aspects of the data (Wiggins et al., 2011). Research units often take an ad-hoc approach to methods and workflow, but standardizing operations and following certified protocols increases confidence in research results (Baker, 2016b)"
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Labs should conduct regular audits and spot checks,Due diligence by research funders can improve research standards.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Labs should conduct regular audits and spot checks,Conduct audits to ensure maintenance of record keeping and good research practice.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",There should be harsher punishment for genuine academic misconduct,"Require compulsory ex-ante disclosure of conflict of interest; Make undeclared conflict of interest sanctionable, e.g. no access to grants, other resources and publication."
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,"If logistics do not allow full reproducibility, authors (and peer reviewers) should ensure that alternative interpretations are explicit","Multiple samples might be pooled to establish the reliability of the separation method(s) and characterize EVs before further characterization or functional studies are performed with individuals samples. If even this solution is impractical, authors should indicate the limit of detection of each applied EV characterization technology and demonstrate that the available material falls below this limit. However, applying this “escape clause” means that EVs cannot be rigorously demonstrated, requiring that authors mention (and reviewers insist on) the caveats of alternative interpretations, i.e. that EVs may contribute, but not necessarily exclusively, to an observed phenomenon or molecular signature."
Minimal information for studies of extracellular vesicles 2018 (MISEV2018): a position statement of the International Society for Extracellular Vesicles and update of the MISEV2014 guidelines,Scopus item - Unclassified,"If logistics do not allow full reproducibility, authors (and peer reviewers) should ensure that alternative interpretations are explicit","Finally, there are exceptions to every rule. MISEV2018 is meant to guide and improve the field, not stifle it. If MISEV recommendations and requirements cannot be met, authors will then need to explain their unique situation and describe their attempts to meet the guidelines and the reason for failure. These guidelines will also continue to evolve."
Promises and pitfalls of data sharing in qualitative research,Note,Thematic coding could be shared in lieu of full interview transcripts,"To minimize the risk of deductive disclosure, a data sharing policy might, in lieu of obliging the release of interview transcripts, require investigators to implement procedures to enhance transparency. Many aspects of the qualitative analysis (e.g., transcription rules, data segmentation, coding units, process for code development, ?nalized codes) could be shared with minimal risk to study participants. Taking transparency a step further, investigators could export coding queries and make these available to external investigators. Because coding queries consist of excerpted and possibly disembodied interview text, they may offer greater anonymity compared with full transcripts. Depending on the interview content, investigators may still need to redact some of the text to preserve anonymity – which would entail added burden – but the risk of deductive disclosures would be reduced."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Developing incentives for rigorous, transparent, and reproducible research","First among these would be to focus more attention on linking the respect we accord scientists to our assessment of the integrity of their work, attention to detail, the durability of their discoveries, and their ability to mentor junior colleagues along these dimensions."
A manifesto for reproducible science,Review,"Developing incentives for rigorous, transparent, and reproducible research","However, there can also be incentives for efficiency and effectiveness — those who conduct rigorous, transparent and reproducible research could be rewarded more than those who do not. There are promising examples of effective interventions for nudging incentives."
How to Make More Published Research True,Article,"Developing incentives for rigorous, transparent, and reproducible research","Optimal interventions need to understand and harness the motives of various stakeholders who operate in scientific research and who differ on the extent to which they are interested in promoting publishable, fundable, translatable, or profitable results."
How to Make More Published Research True,Article,"Developing incentives for rigorous, transparent, and reproducible research","Modifications need to be made in the reward system for science, affecting the exchange rates for currencies (e.g., publications and grants) and purchased academic goods (e.g., promotion and other academic or administrative power) and introducing currencies that are better aligned with translatable and reproducible research."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Developing incentives for rigorous, transparent, and reproducible research","Reward (with funding, and academic or other recognition) reproducibility practices and reproducible research, and enable an efficient culture for replication of research."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,"Developing incentives for rigorous, transparent, and reproducible research","Reward mechanisms (eg, prestigious publications, funding, and promotion) often focus on the statistical significance and newsworthiness of results rather than the quality of the design, conduct, analysis, documentation, and reproducibility of a study. Similarly, statistically significant results, prestigious authors or journals, and well connected research groups attract more citations than do studies without these factors, creating citation bias."
Raising research quality will require collective action,"Report, policy document or website","Developing incentives for rigorous, transparent, and reproducible research","Funding, appointments, promotions, tenure, prizes and so on emphasize individual achievement and overlook deeds that benefit everyone and should be valued explicitly, such as producing usable tools or sharing code. If we want to move towards a transparent model of research, we need to reward open-research practices. If we want researchers to work well in large collaborations, we need to train them in communication skills and collective self-scrutiny."
Raising research quality will require collective action,"Report, policy document or website","Developing incentives for rigorous, transparent, and reproducible research","Earlier this year, the University of Bristol, where I work, made the use of data sharing and other open-research practices an explicit criterion for promotion."
Reproducibility literature analysis - a federal information professional perspective,Article,"Developing incentives for rigorous, transparent, and reproducible research","Rewards for publishing, often tied to showcasing certain results, comprise the biggest challenge to widespread adoption of open data. Conversely, well-conceived incentives may also provide solutions for increased reproducibility. If journals in particular regulate and highlight incentives for research practices promoting reproducibility, researchers will more widely adopt these positive practices (Gezelter, 2015) (Begley and Ioannidis, 2015)."
"Most computational hydrology is not reproducible, so is it really science?",Note,"Developing incentives for rigorous, transparent, and reproducible research","Making code reuseable is more likely to lead to citation and reuse of an individual’s work, which provides an incentive within the current publication system that can be built upon to move toward reproducibility, and gain ef?ciencies across the hydrology community to advance scienti?c understanding across catchments."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,"Developing incentives for rigorous, transparent, and reproducible research","A problem as thorny as the publication bias will require multiple interventions to resolve. However, a central aim must be to re-align incentives for career progression with those for conducting high-quality rigorous research."
On Replication in Communication Science,Editorial,"Developing incentives for rigorous, transparent, and reproducible research","We call on researchers, reviewers, hiring committees, tenure committees, grant reviewers, and administrators to help us continue this work. Use your roles to rework the incentive structure for the work that is conducted by communication scientists. While there is, of course, a place for novelty, to mature as a field and truly have something to contribute to the knowledge marketplace, we also make room for work that adheres to the core of scientific principles, including, and especially, replication studies."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website","Developing incentives for rigorous, transparent, and reproducible research",The environment and culture of biomedical research. Robust science and the validity of research findings must be the primary objective of the incentive structure. These should be valued above novel findings and publications in high-impact journals.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","Developing incentives for rigorous, transparent, and reproducible research","Foster a reproducibility culture by providing incentives and rewards, and invest in research integrity."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","Developing incentives for rigorous, transparent, and reproducible research","Career and promotion
· Make rewards less focused on sheer number of high-impact publications, and more focused on methodological rigour, sharing of results, quality of reports, impact of research;
· Review and tweak research and researcher’s assessment to valorise data reuse;
· Create career incentives (credit and recognition) for data-management;
· Evaluate positively registered reports (acceptance pre-research based on research question and methods proposed to answer them);"
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","From a policy perspective, reproducibility should be discussed as part of research integrity","Finally, ‘research integrity’ is also closely related to reproducibility. Many of the underlying topics to reproducibility are often discussed under research integrity in academia, and funders and policy-makers use research integrity to frame reproducibility in science policy. ‘Research integrity’ refers to the process of good research management practices and to the truthfulness of results, which is also the focus of reproducibility, as well as to the behaviour of individual scientists and to the ethical principles of science and society. In other words, reproducibility is a well-defined complement of integrity, and it may be a clearer policy target for the issues discussed above."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Funders should make efforts to promote high quality research,"Finally, funders could help to promote high-quality research. Many expert panels note the large number of excellent applications that they receive, but this perception is not consistent with the quality of reported research (figure). Members of funding panels, often drawn from the research community itself, might be reluctant to impose a high quality threshold that could disadvantage many investigators. The scientific and administrative leadership of funding agencies could clarify the great importance that they attach to study quality and the minimum standards that they require to reduce the effect-to-bias threshold to an acceptable level."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Funders should make efforts to promote high quality research,"Suggestions to improve reproducibility and reward systems—Designs could be supported and rewarded (at funding or publication level, or both) that foster careful documentation and allow testing of repeatability and external validation efforts, including datasets being made available to research groups that are independent of the original group.86–90 It is important to reward scientists on the basis of good quality of research and documentation, and reproducibility of results, rather than statistical significance.91"
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Funders should make efforts to promote high quality research,Set specific grant requirements.
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Include reproducibility sessions in all scientific meetings,"It is rare to see sessions dedicated to reproducibility issues in research. Imagine if most scienti?c meetings held a regular panel discussion with representation from scientists, funding agencies, and journal editors devoted to their recent experiences with fostering research reproducibility. Common problems could be highlighted and solutions proposed"
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Increasing awareness, not shame and blame","As this signalling (Gintis, Smith & Bowles, 2001) becomes more widespread, it will be harder for others to cut corners and present results that are likely wrong."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Increasing awareness, not shame and blame","If we recognize our cognitive biases as fundamental to our nature rather than as character ?aws to be ashamed of, we can structure our scienti?c endeavours in ways to minimize their effects."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Increasing awareness, not shame and blame","Fortunately, unscienti?c practices like ?shing for signi?cance, HARKing, and many others are on their way out, as a growing community becomes more aware of the risks and better able to recognize the signs of bad practices."
Detecting and avoiding likely false-positive findings – a practical guide,Article,"Increasing awareness, not shame and blame","Further, as preregistration became the norm, exploratory studies would become more transparent and studies without preregistration would come to be viewed as more provisional. If funding bodies rewarded preregistration and unbiased reporting practices along with intellectual merit rather than rewarding only success in attracting citations, then we would rapidly have a transition from a fairly dysfunctional to a much more objective science."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Increasing awareness, not shame and blame","The increasingly frank public discussion of problematic antibodies, coupled with publicly available databases documenting inadequate or appropriate antibody validation (Baker, 2015), represents important steps in energizing the community to pay more attention to the quality of antibodies."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Increasing awareness, not shame and blame","The rising tide of angst and discussions related to the ?delity and reproducibility of preclinical research is likely to foster greater awareness of methodological pitfalls and challenges in experimental design and execution. A growing chorus of scienti?c societies, independent funding agencies and journals, and independent investigators have taken up the rallying cry, and thoughtful recommendations and consensus statements continue to accumulate."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,"Increasing awareness, not shame and blame","excitement and daring with professional fear of error. The key question, therefore, is to de?ne an optimal balance, surely weighted in the direction of reliability, but appropriately tolerant of tentative conclusions and honest errors, while continuously seeking to reduce the latter."
On the Reproducibility of Psychological Science,Article,"Increasing awareness, not shame and blame","Furthermore, as these baselines begin appearing in research papers, there will be further incentive to keep the code up to date. However, only time will tell if we succeed in the long term."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,"Increasing awareness, not shame and blame","Although there are hundreds of papers arguing against null hypothesis significance testing, we see more and more p-values (Chavalarias et al., 2016) and the ASA feeling obliged to tell us how to use them properly (Goodman, 2016; Wasserstein & Lazar, 2016). Apparently, bashing or banning p-values does not work. We need a smaller incremental step that at the same time is highly efficient. There are not many easy ways to improve scientific inference, but together with Higgs (2013) and others, we believe that removing significance thresholds is one of them."
Open is not enough,"Report, policy document or website","Increasing awareness, not shame and blame","Raise awareness Care about the longevity of scientific results. Whether you are a professor, funding body, research associate or a graduate student, ask and discuss with your collaborators if your and their results are preserved and reusable in the long term. Can the next generation of PhD students build on top of your work? Think about publishing code, data and workflow recipes in trusted repositories."
"Most computational hydrology is not reproducible, so is it really science?",Note,"Increasing awareness, not shame and blame","Thus, instead of seeing the need to make their work reproducibile as an inconvenient after-thought, it will be an integral part of their research process."
How we can make ecotoxicology more valuable to environmental protection,Note,"Increasing awareness, not shame and blame","1. Better and ongoing training ofscientists/practitioners (e.g., free short courses on best practices for the conducting and reporting of studies at annual meetings).
2. Promoting ethics and integrity for students and supervisors in their research activities.
3. Advocating for a set of consistent toxicity test methods across jurisdictions that will be the agreed initial screen characterization of the toxicity of a compound to a particular organism.
4. Promoting civil and open discussion/critique of papers and mechanisms for discussion, such as special sessions at annual meetings.
5. Ensuring society journals are working with publishers, authors, and reviewers to improve the reporting of new and negative data."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website","Increasing awareness, not shame and blame",The need to raise awareness among researchers about the importance of reproducibility and how to achieve it.
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Initial authors should be invited to participate in replication studies,"Inviting the initial author to review replication studies can help reduce this, and (in the case of a Registered Report) the initial authors can be invited to provide a Stage 1 review before data collection (see Marsden, Morgan-Short, Tro?movich, & Ellis, 2018). Even more transparent practices that may promote more and higher quality replication, reduce publication bias, and reduce perceptions ofbullying include (a) publishing open reviews and authors’ responses to reviews (e.g., in BMC Psychology; Laws, 2016); (b) giving initial authors an automatic right to a peer-reviewed published commentary (e.g., in Perspectives in Psychological Science; in our sample, we found one such example, Kanno, 2000); and (c) adversarial collaborations (Coyne, 2016; Kahneman, 2014; Koole & Lakens, 2012; Mellers, Hertwig, & Kahneman, 2001), where researchers who account for phenomena differently agree to work together following a single protocol."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Initial authors should be invited to participate in replication studies,"Recommendation: When possible, ensure that replication studies are conducted by researchers independently of the initial study’s authors but that the initial authors are invited to be involved at some stage of the review process, preferably prior to data collection."
Making sense of replications,Article,Initial authors should be invited to participate in replication studies,"The authors of the original papers were contacted in advance for details of the research methodology that may not have appeared in their paper, and were asked to share any original reagents, protocols and data in order to maximize the quality and fidelity of the replication designs."
The challenges of replication,Editorial,Initial authors should be invited to participate in replication studies,"For every paper the team performing the replication contacted the corresponding author of the original paper for additional information to help prepare the Registered Report. The corresponding author was also asked to comment on this report during the peer review process and some, but not all, availed of this."
A manifesto for reproducible science,Review,Reproducibility initiatives will need continued improvement and evaluation,"These cautions are not a rationale for inaction. Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and efficient knowledge accumulation is a scientific question; the most effective solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few. True understanding of how best to structure and incentivize science will emerge slowly and will never be finished. That is how science works. The key to fostering a robust metascience that evaluates and improves practices is that the stakeholders of science must not embrace the status quo, but instead pursue self-examination continuously for improvement and self-correction of the scientific process itself. As Richard Feynman said, “The first principle is that you must not fool yourself – and you are the easiest person to fool.”"
Consensus on Exercise Reporting Template (CERT): Explanation and Elaboration Statement,Article,Reproducibility initiatives will need continued improvement and evaluation,"We anticipate that the CERTwill be an evolving document, as have been other reporting guidelines, with a requirement for review and re?nement as new evidence and critical comments accumulate."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,Reproducibility initiatives will need continued improvement and evaluation,"We will revisit the guidance in 2020 but will also conduct six-monthly audits, in order to monitor its effects, and will introduce new guidance as appropriate."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,Reproducibility initiatives will need continued improvement and evaluation,"Future efforts could examine benefits, and also potential pitfalls, regarding these and other initiatives such as posting datasets online and various changes in policies by journals, professional organizations, and funding agencies aimed at reducing impediments to transparency"
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Reproducibility initiatives will need continued improvement and evaluation,"Measures to improve reproducibility should be developed in consultation with the biomedical research community and evaluated to ensure that they achieve the desired effects. They should not unnecessarily inhibit research, stifle creativity, or increase bureaucracy."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Issues with scientific reproducibility should be communicted clearly with the general public so as not to cause mistrust in science,"• There is high public trust in scientists, but this must continue to be earned. Insofar as lack of reproducibility is a problem, the scientific community should talk about plans to tackle it, rather than trying to hide it from public view.
• Greater effort is needed to raise awareness of the complexity of the scientific method, noting that conflict is a natural part of the process. Scientists should not underestimate the general public’s scientific understanding, but communicate the complexity of the issues rather than reducing the problem to a simple narrative that may be misleading.
• There is a need to emphasise that irreproducible research can occur for many legitimate reasons and this does not necessarily mean that research is fraudulent or poorly conducted."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Issues with scientific reproducibility should be communicted clearly with the general public so as not to cause mistrust in science,"• Care should be taken in communicating issues of reproducibility without undermining public trust. Science is one of the most powerful drivers of improvements in people’s quality of life, and we must be clear that the goal is to make it even better, rather than to disparage scientists. Scientists and science communicators (including journalists and press office staff) have a duty to accurately portray research results."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Journals should also have a clear way of identifying retracted material,"Clarify use of retractions Develop an agreed upon and consistent approach for journals to manage retractions. This would include clear criteria for retractions versus corrections, how they are linked to published papers, and how the reason for the retraction or correction is explained, whether due to misconduct or a wide array of more innocent explanations."
Reproducibility and Research Integrity,Note,Journals should also have a clear way of identifying retracted material,Retraction notices should provide a clear explanation of the reasons for the retraction and should be linked to the original article. Retracted articles should be clearly identified in electronic versions of the journal and bibliographic databases (Committee on Publication Ethics 2009: 2).”
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Journals should encourage and directly invite replications or have a dedicated article type for this,"Another possibility is for journal editors to invite replications of particular studies, as is occasionally done by the editors of the Registered Replication Reports in Perspectives in Psychological Science. This approach exerts strong editorial in?uence over the types of studies that are replicated and how they are replicated and demands a heavy editorial role (D. Simons, personal communication, September 16, 2016)."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Journals should encourage and directly invite replications or have a dedicated article type for this,"Leading journals could trigger a positive change by requiring data and code release for all articles, and introducing a special article type (e.g., a pre-registered Replication Report) for direct replication attempts. Currently, direct replications are not considered to be novel enough to be worth publishing, and novelty of results is given disproportionate weight. However, replication is an important tool for establishing reliability. This is something that a p-value, especially a pvalue computed from an underpowered study, cannot ever deliver. Increasing precision and conducting direct replications are vital for any empirically rigorous science."
The possibility and desirability of replication in the humanities,Note,Journals should encourage and directly invite replications or have a dedicated article type for this,"Journals in the humanities should encourage replication studies and publish them, irrespective of their results."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Journals should encourage and directly invite replications or have a dedicated article type for this,"However, the need to promote the publication ofreplications does not mean that these should be diverted into separate, distinct journals, which may give the false impression ofbeing second-rate science. In fact, mathematical modeling suggests that replications often are more important than discoveries [35]. We should also acknowledge that an increase in the proportion of articles that use replication-related language may either mean that more replications are performed and/or studies with replication elements are now becoming more readily disclosed as such, while in the past investigators would have tried to sell them as entirely novel."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,Journals should encourage and directly invite replications or have a dedicated article type for this,"To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research."
Replication in strategic management,Editorial,Journals should encourage and directly invite replications or have a dedicated article type for this,SMJ is interested in replications that accord with prior findings as well as those that do not.
On Replication in Communication Science,Editorial,Journals should encourage and directly invite replications or have a dedicated article type for this,"Within this issue of Communication Studies, our editorial team (McEwan, Westerman, and Carpenter) were granted the opportunity by Editor-in-Chief Ken Lachlan to do more than simply lament the lack of replication in our field but to provide publishing space and incentive to researchers to pursue replication studies."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Machine readable papers would allow us to better capture science,"Perhaps in the future, natural language processing and machine learning algorithms will permit the under-examined literature to be more effectively explored."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",More research on research is needed to identify strategies to improve reproducibility,"Wherever possible, strategies to address these issues should be supported by evidence of the effectiveness of these strategies, and that will require more ‘research on research’."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Delivering training at all levels of seniority or experience,"6.1. Enhance training in experimental design, statistics, proper use of reagents, data management, research and publication ethics, and clarify expectations for how to respond to concerns about these That this is necessary is obvious e such knowledge is at the core of what is required to do quality research. Most institutions have research training programs designed to address these issues, but it seems clear that additional focused efforts are needed. Institutions should consider how best to ensure that skilled educators and appropriate materials are available and effectively employed. Education is needed at all levels of training, including for faculty, who may have de?ciencies in key realms of which they are unaware. In the end, faculty bear the greatest responsibility for both applying proper methodologies and educating their trainees. Leaders of institutions that conduct substantial research programs must make this area a priority."
How to Make More Published Research True,Article,Delivering training at all levels of seniority or experience,Better training of scientific workforce in methods and statistical literacy
How to Make More Published Research True,Article,Delivering training at all levels of seniority or experience,"Finally, proper training and continuing education of scientists in research methods and statistical literacy are also important."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Delivering training at all levels of seniority or experience,"Expectations for continued professional development, reflective practice, and validation of investigative skills should be reconsidered. The medical profession recognises the need for continued medical education, and even revalidation, to ensure the highest quality of clinical practice. Clinical and laboratory researchers might also benefit from an opportunity to update their skills in view of newer methodological developments, perhaps through short courses and novel approaches to continued methodological education."
Reboot undergraduate courses for reproducibility,"Report, policy document or website",Delivering training at all levels of seniority or experience,Early training in collaboration might also bring comfort and creativity with regards to similar approaches later in students’ research careers.
Reproducibility literature analysis - a federal information professional perspective,Article,Delivering training at all levels of seniority or experience,"Open Science, the movement to make scientific products and processes accessible to, and reusable by all, relies on culture and knowledge as much as it does on technologies and services. Convincing researchers of the benefits of changing their practices, and equipping them with the skills and knowledge needed to do so can happen through training and education. A recommendation from citizen science states that “Training, in particular, has been shown elsewhere to enhance accuracy and credibility (Freitag, 2016, Kosmala, 2016)."
Our path to better science in less time using open data science tools,Article,Delivering training at all levels of seniority or experience,"There is ongoing and important work by the informatics community on the architecture and systems for data management and archiving, as well as efforts to enable scientists to publish the code that they do have. This work is critical, but comes with the a priori assumption that scientists are already thinking about data and coding in a way that they would seek out further resources. In reality, this is not always the case, and without visible examples of how to use these tools within their scientific fields, common stumbling blocks will be continually combatted with individual workarounds instead of addressed with intention. These workarounds can greatly delay focusing on actual scientific research, particularly when scientific questions that may not yet have answers—for example, how the behavior of X changes with Y—are conflated with data science questions that have many existing answers—for example, how to operate on only criteria X and Y."
"Most computational hydrology is not reproducible, so is it really science?",Note,Delivering training at all levels of seniority or experience,"A key step to change this culture is to ensure that computational science training (e.g., http://software-carpentry.org) is properly embedded within hydrological science curriculums, so that future generations of hydrologists have the skills to build readable, version controlled and unit-tested software [McConnell, 2004], allowing them to engage more fully in an open scienti?c community by reproducing and reusing each other’s research outputs."
How we can make ecotoxicology more valuable to environmental protection,Note,Delivering training at all levels of seniority or experience,"1. Work towards enhancing the training of all practitioners prior to the conduction of any ecotoxicology study.
2. Prior to starting the study, draw in appropriate expertise to ensure greatest possible quality (e.g., statisticians, chemists)."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Delivering training at all levels of seniority or experience,Teach Alternative Approaches Seriously
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Delivering training at all levels of seniority or experience,The role for continuing education and training that improves research methods and statistical knowledge; this should be targeted at individuals across career stages.
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Delivering training at all levels of seniority or experience,Providing further education in research methods may be a starting point in ensuring scientists are up-to-date with new developments and making them aware of issues surrounding irreproducibility.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Delivering training at all levels of seniority or experience,"Training and careers: Provide accredited training on integrity, basic statistics principles/methods, publication process, reproducibility culture, etc.; Provide training for peer-review panel chairs; · Foster specific career tracks (specialised data stewards)."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Focus specifically on training new researchers to have good practice from the start,"While senior grant applicants are still following an old logic, juniors are more invested in good data management. It is necessary to act at the earliest possible stage, in graduate school (MSc), to ensure that old hoarding practices do not propagate to new generations of scientists. Competition to publish at PhD level makes young researchers vulnerable: they will pressurise themselves to conform to the current race to the bottom. There is a need for rewards for good practice from an early career stage to guide PhD and young scholars to share data, findings, methods, to engage with the public, to focus on real impact vs. the impact factor."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Focus specifically on training new researchers to have good practice from the start,"Training in statistics, data management and intellectual property rights (IPR) are critical, and career and cultural rewards are needed. Indeed, young scholars face a greater pressure to obtain positive results, and not to be seen as having ‘failed’ when achieving no positive results or new breakthroughs in spite of good research, than scholars with an established standing."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Need more research on failures of integrity to understand the trigger,"Much more needs to be done to explore this issue. We must understand how and why scientists sometimes cross the line between rigorous objectivity and practices that violate it, tempted by what they may see as short term bene?ts, though in the end these decisions are self-defeating. A second line that may be crossed is hard to delineate in some cases: the point where “questionable practices” transform into research misconduct, against which strong sanctions are required."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Need more research on failures of integrity to understand the trigger,"Frequently, the question is: where do sloppiness, wishful thinking, and, perhaps, morally innocent selectivity in data presentation transition into falsi?cation? A key element in making the distinction is whether there is intention to deceive, as opposed to errors, self-deception, or honest differences of opinion. It should be obvious that such distinctions are very dif?cult to make in individual cases. Those individuals required by circumstance and institutional role to make these dif?cult distinctions face an onerous task, since intention is often hard to assess, and careers typically hang in the balance."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Need more research on the why so few reproduction studies exist,"We also emphasize that data are needed about the causes of low published replication rates to inform our efforts, including those recommended in the following sections, in empirically grounded ways. For example, the publication of replication studies that had null ?ndings or that did not support the initial ?ndings may have been adversely affected by publication bias and so may be one cause of the overall low rate of published replications."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,Need more research on the why so few reproduction studies exist,Recommendation: Make systematic inquiry into the causes of low rates of published replication studies and provide (more) empirical evidence about the extent and causes of publication bias in the ?eld.
MAKING REPLICATION MAINSTREAM,Article in Press,Need to develop clear definitions and methodologies for different types of replication studies,"Conceptual replications do not serve the same purposes as direct replications. Therefore, we encourage researchers to adopt different terminology when describing conceptual replications in the future. This will yield a clearer distinction between studies that use the same procedures as the original studies and studies that use different procedures."
MAKING REPLICATION MAINSTREAM,Article in Press,Need to develop clear definitions and methodologies for different types of replication studies,Using alternative test in place of the term conceptual replication might help clear up confusion in the literature that occurs when researchers disagree as to whether or not an effect has been replicated.
Reproducibility of Published Research,"Report, policy document or website",Need to develop clear definitions and methodologies for different types of replication studies,"The role ofAcademies and Learned Societies can be very important both for research integrity in general, and for questions of reproducibility in particular. Many Science Academies represent a broad range of scienti?c disciplines and are thus a very appropriate forum to, for example, agree on questions of terminology regarding reproducibility. More importantly, they should be able to identify for what type of research one should reasonably expect that research results need to be reproducible to be recognised."
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards",Recommendation: Encourage more journals to give more and stronger incentives to their authors for systematically making materials and data openly available.
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","A very simple strategy for enabling research to be more easily reproduced is the provision of suf?cient experimental detail, including careful description of and source of reagents, cell lines, and animals used in each experiment. While this sounds obvious, assessment of the extent to which publications routinely provide suf?cient information to facilitate reproducibility reveals major gaps in our collective communication styles."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Openly and fully report our detailed methods, materials, procedures, data, and analysis scripts."
A manifesto for reproducible science,Review,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Poor usability reflects difficulty in evaluating what was done, in reusing the methodology to assess reproducibility, and in incorporating the evidence into systematic reviews and metaanalyses. Improving the quality and transparency in the reporting of research is necessary to address this."
A manifesto for reproducible science,Review,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Benchmarking intrinsically brings the concept of comparisons. For this reason, a standardized way of reporting should be preferred."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards",Approaches used to reduce unwanted sources of variation by data normalization (which means the correction of test values to baseline or control group values) or to generate normal (Gaussian) data (e.g. by logtransformation) must be justi?able and explained.
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Selection of a method should be guided by the relevant scientific question and downstream applications, and all details of the method should be provided to ensure reproducibility."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Transparency regarding the level of theory, measurement, andanalysis allows others to recognizepotential influences of such decisions on the research question and themeaning ofresults, such aswhether constructs and results differ when conceptualized and tested at different levels, or whether variables at different levels may affect the substantive conclusions reached (Dionne et al., 2014;Hitt, Beamish, Jackson &Mathieu, 2007; Mathieu & Chen, 2011; Schriesheim, Castro, Zhou, & Yammarino, 2002)."
What you see is what you get? Enhancing methodological transparency in management research,Review,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","First, as mentioned previously, enhanced transparency also improves results reproducibility—the ability of others to reach the same results as the original paper using the data provided by the authors. This allows reviewers and editors to check for errors and inconsistencies in results before articles are accepted for publication, thereby reducing the chances of a later retraction. Second, enhanced transparency can contribute to producing higher-quality studies and in quality control (Chenail, 2009)"
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","If replication is a desirable goal, then transparency is a required step."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Also, improved replicability is likely to lead to improvements in quality because manuscripts that are more transparent allow for a more trustworthy assessment of a study's contributions for theory and practice (Brutus, Aguinis, & Wassmer, 2013)."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","The higher the transparency level across the 12 criteria, the more the study becomes trustworthy and reproducible, and the higher the likelihood that future replication will be possible. In other words, the transparency criteria have a cumulative effect in terms of the trustworthiness and replicability of results. Also, we offer recommendations on what features or information to include."
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"Need to improve the overall quality and transparency of reporting, to rely less on trust, and to build reporting standards","Data coding and first-order codes. Future qualitative research should be clear about the type of coding strategies adopted (e.g., structural, in vivo, open/initial, emotional, and vs.)."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,"Need to report deviations from protocol, mistakes, outliers, and other variation","What if scientists could not follow all guidelines? In this case, they were encouraged to explain why certain experiments could not be performed. The example of low-abundance, precious samples was given explicitly. That is, guidance was accompanied by acknowledgement of the realities of the field"
What you see is what you get? Enhancing methodological transparency in management research,Review,"Need to report deviations from protocol, mistakes, outliers, and other variation","While missing data and nonresponses are rarely the central focus of a study, they usually affect conclusions drawn from the analysis (Schafer & Graham, 2002). Moreover, given the variety of techniques available for dealing with missing data (e.g., deletion, imputation), without precise reporting of results of missing data analysis and the analytical technique used, others are unable to judge whether certain data points were excluded because authors did not have sufficient information or because excluding the incomplete responses supported the authors’ preferred hypotheses (Baruch & Holtom, 2008). In short, low inferential reproducibility is virtually guaranteed if this information is absent."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,"Need to report deviations from protocol, mistakes, outliers, and other variation","Guideline: Outlier observations are discussed carefully, especially when they have been eliminated from the sample (e.g., through technical practices such as ‘winzorizing’)"
Transparency and replicability in qualitative research: The case of interviews with elite informants,Article,"Need to report deviations from protocol, mistakes, outliers, and other variation","Unexpected opportunities, challenges, and other events. Future qualitative research should report what unexpected opportunities, challenges, and other events occurred during the study and how they were handled (e.g., participants dropped out of the study, a new theoretical framework was necessary). Because these unexpected events may affect data accessibility and substantive conclusions, researchers should report and describe any unexpected events and highlight whether they had an impact on the data collection and data analysis."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,New codes and softwares should be tested appropriately,"With regard to point 2, we are currently working on developing an infrastructure, based on work?ow programming language, that allows users to connect to the mentioned repository and run their feature extraction software. This infrastructure can be expanded by introducing in the work?ow a “benchmarking module,” where users can easily “test” their software on common data sets, directly choosing them according to the modality and/or disease population of interest."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Non replication can serve to generate useful discussions,"Can we explain satisfactorily such a consistent profile through random variation? Or can it lead us to the possible reason for the conflicting evidence? If we follow the metaphor of a submerged mountain chain, we can predict which parts are likely to appear first when the water levels begin to fall. The task is to identify what determines the changes in water level."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Non replication can serve to generate useful discussions,"This paper argues that the conflicting evidence in the area of bilingualism and cognition and the ensuing debate are neither surprising, nor worrying, nor exceptional compared with other areas of science."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Non replication can serve to generate useful discussions,"Finally, the discrepancy in results between different studies is not a curse but a blessing for the advancement of science. It can allow us to identify new factors, which would otherwise have gone unnoticed. It can help us to produce new theories, with a much stronger claim to universality than those based on a small sample of world’s countries and cultures."
"Cooking pasta in la Paz: Bilingualism, bias and the replication crisis",Article,Non replication can serve to generate useful discussions,"Doing bilingualism research in Toronto and San Francisco, Edinburgh and Barcelona, Hyderabad and Hong Kong might not be the same either. But in every place we can learn something new and putting together these bits of the puzzle can help us to get the larger picture. We should embrace the complexity of the results we are facing rather than try desperately to reduce them to an enticingly simple but ultimately misleading “yes” or “no” question (Bak, 2015; Woumans & Duyck, 2015)."
The possibility and desirability of replication in the humanities,Note,Non replication can serve to generate useful discussions,When the primary study and its replication attempt lead to different conclusions it is important to scrutinize the details of both studies. That may lead to the conclusion that one of them is superior and should be trusted more. Or the differences between both studies may explain the differences in results by showing that these are conditional. And in some instances another replication attempt may be needed.
Making sense of replications,Article,Non replication can serve to generate useful discussions,"Indeed, a failed replication can lead to a better understanding of a phenomenon if it results in the generation of new hypotheses to explain how the original and replication methodologies produced different results and, critically, leads to follow-up experiments to test these hypotheses (Ebersole et al., 2017)."
Making sense of replications,Article,Non replication can serve to generate useful discussions,"When a replication ""fails"" it can spur productive theorizing about the source of that irreproducibility. For example, it could be that the experimental model did not behave as expected (for example, the rate of tumor onset observed in the replication might be higher than the rate observed in the original research in both the control and experimental conditions). In such circumstances, the original hypothesis and finding may not have been evaluated directly because the experimental circumstances necessary to test them did not recur."
The reproducibility “crisis”: Reaction to replication crisis should not stifle innovation,Note,Non replication can serve to generate useful discussions,"Nonetheless, discrepancies between original and replication studies could indeed enrich research. “Provided all studies were done well, different outcomes of replicate studies would be informative in telling us that conditions matter, and that we need to search further to establish the range of conditions under which a given treatment works”, Wurbel said."
On Replication in Communication Science,Editorial,Non replication can serve to generate useful discussions,"However, a failure to replicate a previous finding under different conditions may not invalidate the finding overall, but it may help to alert to potential boundary conditions of where a proposed relationship between variables can be expected to occur or not occur."
Reproducibility literature analysis - a federal information professional perspective,Article,Plans to improve research integrity must be co-created by all stakeholders,"An overarching theme with regard to reproducibility solutions boils down to a culture shift throughout the research endeavor and data gathering. Culture shift encompasses changing beliefs, behaviors, and outcomes. Industries must broadly address their practices at all stages of data collection, processing, publication, dissemination, and preservation to make reproducibility commonplace (Baker, 2016b)."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Plans to improve research integrity must be co-created by all stakeholders,"They need to be involved in analysing the problem, devising solutions, and maintaining and updating plans to implement those solutions. Differing perceptions must be explored and negotiated, and solutions crafted for each institution."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Plans to improve research integrity must be co-created by all stakeholders,"“Hierarchical, top-down implementation is doomed to fail.”"
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Plans to improve research integrity must be co-created by all stakeholders,"Measures to improve reproducibility should be developed in consultation with the biomedical research community and evaluated to ensure that they achieve the desired effects. They should not unnecessarily inhibit research, stifle creativity, or increase bureaucracy"
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",Plans to improve research integrity must be co-created by all stakeholders,Irreproducibility of biomedical research is a global issue – tackling it requires a global approach with multi-stakeholder involvement.
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Local champions can help,"It is difficult to assess how much these projects increased research integrity, let alone compare them in terms of time and effort. We know that they required local champions. The fact that we were able to identify dozens of these projects suggests that people can be convinced that such internally driven efforts are worthwhile."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website","Policy intervention to increase reproducibility should focus on reproduction and replication, and wider open science policies would assist with re-use",Policy intervention to increase reproducibility should focus on reproduction and replication; and wider open science policies would assist with re-use.
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Pre-registration should become the norm,"This transparency would likely prove sobering and for some would be an unwelcome obligation, yet it might frame the more promising results presented in a more realistic light and broader context."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Pre-registration should become the norm,"Planning. The sample size for the study would be determined in advance using formal statistical power analysis. The entire analysis plan, including exclusion and inclusion criteria, software workflows (including contrasts and multiple-comparison methods) and specific definitions for all planned regions of interest, would be formally pre-registered."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Pre-registration should become the norm,"Pre-registration In our view one of the most important and virtually costfree (to researchers) improvement would be to pre-register hypotheses and analysis parameters and approaches (in line with Section 5.2 in Nichols et al., 2016; p11; Gelman and Loken, 2014). Pre-registration can easily be done for example, at the website of the Open Science Foundation (osf.org), also in a manner that it does not immediately become public. Hence, competitors will not be able to scoop good ideas before the study is published."
Recommendations for open data science,Note,Provide or cite data and code alongside papers,Provide or cite source code for tools in public repositories. Software packages are critical in data-driven studies. Other researchers can only fully utilize results if the details of the computational methods used are completely transparent. Provide source code for all original software in public repositories such as Github with a license permitting as free use as possible. Cite precise versions and command line prompts for all previously published tools. Do not use proprietary software that cannot be accessed by other researchers.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Provide or cite data and code alongside papers,"Publishing platforms can promote reproducibility and provenance tracking by improving the connections between research papers and data and materials in repositories. Ensuring links between journal articles and datasets are present, functional and accurate is technologically simple, but can be procedurally challenging to implement when multiple databases are involved. Connecting published articles and research data in a standardised manner across multiple publishing platforms and data repositories, in a dynamic and universally adoptable manner, is highly desirable."
"Most computational hydrology is not reproducible, so is it really science?",Note,Provide or cite data and code alongside papers,"Reproducibility is a foundational principle in scienti?c research. Yet in computational hydrology the code and data that actually produces published results are not regularly made available, inhibiting the ability of the community to reproduce and verify previous ?ndings. In order to overcome this problemwe recommend that reuseable code and formal work?ows, which unambiguously reproduce published scienti?c results, are made available for the community alongside data, so that we can verify previous ?ndings, and build directly from previous work. In cases where reproducing large-scale hydrologic studies is computationally very expensive and time-consuming, new processes are required to ensure scienti?c rigor. Such changes will strongly improve the transparency of hydrological research, and thus provide a more credible foundation for scienti?c advancement and policy support."
"Most computational hydrology is not reproducible, so is it really science?",Note,Provide or cite data and code alongside papers,"In order to advance scienti?c progress in hydrology, reproducibility is required in computational hydrology for several key reasons. First, the reliability of scienti?c computer code is often unclear. From our own experience, it is often very dif?cult to spot errors unless they manifest themselves in very obvious errors in model outputs. Thus, code needs to be transparent to allow the legitimacy of published results to be veri?ed."
"Most computational hydrology is not reproducible, so is it really science?",Note,Provide or cite data and code alongside papers,"To help move toward reproducible computational hydrology, we recommend the following:
1. code needs to be made readable and reuseable for the community; 2. work?ows that tie together data and reuseable code need to be created to document, unambiguously, the full provenance of published scienti?c results; 3. reuseable code and work?ows need to be made available and easy to ?nd through consistent use of repositories and creation of code metadata;
4. reuseable and reproducible code needs to be cited in publications using unique persistent identi?ers (e.g., DOIs) to clearly show the provenance of published scienti?c ?ndings; and
5. new procedures need to be developed that ensure scienti?c rigor in circumstances where reproducing large-scale studies is computationally very expensive and time consuming."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Provide or cite data and code alongside papers,"Publish All Analysis Scripts with Analysis Settings Another cost-free improvement is to publish all analysis scripts with the ability to regenerate all ?gures and tables (Laine et al., 2007; Peng, 2009, 2011; Diggle and Zeger, 2010; Keiding, 2010; Doshi et al., 2013). This does not require large storage space and can also be done in Supplementary Material. If researchers keep this expectation in mind from the start of a project then implementing it becomes relatively straightforward. Program code will often provide information which is missing from papers."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Provide or cite data and code alongside papers,"Open data: Enable liberal and fair re-use of project data; Encourage peer-review of data; Extend the concept of ‘open’: open data, open protocols, open software, open research tools, open computational workflows, ...; Capture structured information about the research data analysis and document workflows, both computational and lab-based; Support the production of dedicated software and workflows that enable reproducibility."
Open is not enough,"Report, policy document or website",Clearly describe techniques behind analysis,Describe: adequately describe and structure the knowledge behind a physics analysis in view of its future reuse. Describe all the assets of an analysis and track data provenance. Ensure sufficient documentation and capture associated links.
Open is not enough,"Report, policy document or website",Clearly describe techniques behind analysis,"Capture: store information about the analysis input data, the analysis code and its dependencies, the runtime computational environment and the analysis workflow steps, and any other necessary dependencies in a trusted digital repository.
Reuse: instantiate preserved analysis assets and computational workflows on the compute clouds to allow their validation or execution with new sets of parameters to test new hypotheses."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,Data sharing standards need to vary across disciplines,"Providing several options for journal data policy is necessary because, across multiple research disciplines, some research communities and their journals are more able to introduce strong data sharing requirements than others."
The statistical significance filter leads to overoptimistic expectations of replicability,Article,Journals should ask that codes be included with manuscripts,Data and code to be released mandatorily along with the published paper.
Jupyter Notebooks—a publishing format for reproducible computational workflows,Conference Paper,Journals should ask that codes be included with manuscripts,"Others provide code separately as supplementary material, but it may be difficult for readers to cross reference between code and prose, and there is a risk that the two become inconsistent as the author works on them."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Journals should ask that codes be included with manuscripts,"Custom analysis code should always be shared on manuscript submission. It may be unrealistic to expect reviewers to evaluate code in addition to the manuscript itself, although this is standard in some journals such as the Journal of Statistical Software. However, reviewers should request that the code be made available publicly (so others can evaluate it) and, in the case of methodological papers, that the code is accompanied with a set of automated software tests."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Journals should ask that codes be included with manuscripts,"Because the computer code is often necessary to understand exactly how a data set has been analysed, releasing the analysis code is particularly useful and should be standard practice."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Journals should ask that codes be included with manuscripts,All code for data collection and analysis would be stored in a version-control system and would include software tests to detect common problems. The repository would use a continuous integration system to ensure that each revision of the code passes appropriate software tests. The entire analysis workflow (including both successful and failed analyses) would be completely automated in a workflow engine and packaged in a software container or virtual machine to ensure computational reproducibility.
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,"p-values should be declared in full, and negative findings need to be reported","When reporting statistics, give their precise values rather than mere inequalities; for example, if we are reporting a P-value and it is 0.03, report “p = 0.03,” not “p < 0.05.”"
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,"p-values should be declared in full, and negative findings need to be reported",Guideline: Authors should refer to the actual p-value rather than the threshold p-value when assessing the evidence for and against their hypothesis. Guideline: Authors should not report asterisks to signal p-value thresholds.
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,"p-values should be declared in full, and negative findings need to be reported","Guideline: Null and negative ?ndings are equally interesting as are positives, and hence are honestly reported, including a discussion of what this implies for theory."
Experimental design and analysis and their reporting II: updated and simplified guidance for authors and peer reviewers,Editorial,Some journals allow authors to explain whether a statistical analysis result may be a false negative or positive,"Statistical analysis does not guarantee that a ?nding is the data necessarily correct, and we will allow an author the right to argue that a false positive or a false negative ?nding may have been generated. This issue is particularly relevant to variables of secondary interest. Clearly, group size should be determined apriori such that an expected effect on the variable of primary interest can easily be detected using the prede?ned P threshold (we refer readers to our advice on determining group sizes). However, such group sizes may be insuf?cient for reliable detection of effects on secondary and subsidiary variables. It is the responsibility of the author to explain this in the paper, especially if they wish to argue that an apparent lack of effect was due to a type 2 error (false negative)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"We review how confusion about interpretation of larger pvalues can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"In the following, we start with reviewing what p-values can tell about replicability and reliability of results. That this will not be very encouraging should not be taken as another advice to stop using p-values. Rather, we want to stress that reliable information about reliability ofresults cannot be obtained from p-values nor from any other statistic calculated in individual studies. Instead, we should design, execute, and interpret our research as a ‘prospective meta-analysis’ (Ioannidis, 2010), to allow combining knowledge from multiple independent studies, each producing results that are as unbiased as possible. Our aim is to show that not p-values, but significance thresholds are a serious obstacle in this regard."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"If necessary, we should then focus on the p-value as a continuous measure of compatibility (Greenland et al., 2016), and interpret larger p-values as perhaps less convincing but generally ‘positive’ evidence against the null hypothesis, instead ofevidence that is either ‘negative’ or uninterpretable or that only shows we did not collect enough data. In short, we should develop a critical but positive attitude towards larger p-values. This alone could lead to less proofs of the null hypothesis, to less significance chasing, less data dredging, less p-hacking, and ultimately to less publication bias, less inflated effect sizes and more reliable research. And removing significance thresholds is one of the smallest steps that we could imagine to address issues of replicability. Using p-values as graded evidence would not require a change in statistical methods. It would require a slight change in interpretation of results that would be consistent with the recommendations by the late Ronald Fisher and thus with a neoFisherian paradigm described by Hurlbert& Lombardi (2009)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"A first step would be to stop using the word ‘significant’ (Higgs, 2013; Colquhoun, 2014). Indeed, often null hypotheses about zero effects are automatically chosen only to be rejected (Gelman, 2013b), and null hypotheses on effect sizes other than zero, or on ranges of effect sizes, would be more appropriate (Cohen, 1994; Greenland et al., 2016). The way to become aware of our zero effect automatism is to ‘‘replace the word significant with a concise and defensible description of what we actually mean to evoke by using it’’ (Higgs, 2013)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"We do not need to publicly register our study protocol before we collect data, we do not need to learn new statistics, we do not even need to use confidence intervals if we prefer to use standard errors. Ofcourse all ofthose measures would be helpful (e.g., Cumming, 2014; Academy ofMedical Sciences, 2015), but they usually require an active change of research practices. And many researchers seem to hesitate changing statistical procedures that have been standard for nearly a century (Thompson, 1999; Sharpe, 2013)."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Remove fixed significance thresholds e.g. p values,"Further, we recommend choosing not only from the available null hypothesis tests but also from the toolbox provided by Bayesian statistics (Korner-Nievergelt et al., 2015). But we agree that we should not ‘‘look for a magic alternative to NHST [null hypothesis significance testing], some other objective mechanical ritual to replace it. It doesn’t exist’’ (Cohen, 1994)."
How Bayes factors change scientific practice,Article,Remove fixed significance thresholds e.g. p values,Bayes factors partly solve the problem by allowing the evidence to go both ways. This means you can tell when there is evidence for the null hypothesis and against the alternative. You can tell when there is good evidence against there being a treatment side effect (and when the evidence is just weak); you can tell when the data count against a theory (and when they count for nothing).
How Bayes factors change scientific practice,Article,Remove fixed significance thresholds e.g. p values,"Power and replication. Replications are hard to evaluate by reference to p-values. If an original result was significant, and a direct replication non-significant, it might feel like a failure to replicate. But as p-values cannot indicate whether the null hypothesis is supported, a non-significant replication tells one nothing in itself."
When null hypothesis significance testing is unsuitable for research: A reassessment,Review,Remove fixed significance thresholds e.g. p values,"Null hypothesis signi?cance testing (NHST) has several shortcomings that are likely contributing factors behind the widely debated replication crisis of (cognitive) neuroscience, psychology, and biomedical science in general. We review these shortcomings and suggest that, after sustained negative experience, NHST should no longer be the default, dominant statistical practice of all biomedical and psychological research. If theoretical predictions are weak we should not rely on all or nothing hypothesis tests. Different inferential methods may be most suitable for different types of research questions."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Reporting items should be machine-readable,"Perhaps in the future, natural language processing and machine learning algorithms will permit the under-examined literature to be more effectively explored."
Repeatability and Reproducibility of Radiomic Features: A Systematic Review,Article,Reporting items should be machine-readable,"To facilitate this process, we are working on providing users with standard template tables that need to be ?lled in by the users and that include all the information mentioned earlier. In our view, this represents the ?rst step toward homogenizing and increasing the quality of reporting. However, to facilitate feature comparison, we recommend using ontology techniques combined with Semantic Web to transform template tables into semantically linked data that can easily be queried by means of universal concepts de?ned by the ontology."
Open is not enough,"Report, policy document or website",Reporting items should be machine-readable,Structure your knowledge to be both human and machine readable. Using descriptive ‘readme’ files is good; using a structured JSON format to describe knowledge and make it searchable is even better. Use standard vocabularies existing in your community.
"Most computational hydrology is not reproducible, so is it really science?",Note,Reporting items should be machine-readable,"To help move toward reproducible computational hydrology, we recommend the following: code needs to be made readable and reuseable for the community."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Reproducibility index (both at researcher and journal level),"Rather than only quantifying citations, papers in top journals, and research funding, why not quantify reproducibility? Imagine if each scientist was associated with a Rs (ReproducibilityScientist) index, re?ecting the number of times the key scienti?c ?ndings in a paper had been reproduced by at least one other independent research group. Of course, one would have to carefully debate and re?ne the meaning of ‘‘reproducible’’ (Goodman et al., 2016), but perhaps one could start simply by requiring that (a) the key ?ndings and (b) at least 50% of the experimental data, from a single paper, were independently reproduced by at least one other research group. So a senior scientist with a Rs index of 40 would have published 40 research papers with ?ndings found to be independently reproduced by others."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Reproducibility index (both at researcher and journal level),"Adjudication of the reproducibility of each paper would have to be carefully tracked over time, ideally by an independent body, which would require funding to sustain its activities. This type of undertaking presents major feasibility challenges, but also opportunities for independent organizations dedicated to the reproducibility of published research."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Reproducibility index (both at researcher and journal level),"The reproducibility index should not be restricted to scientists. Each journal should also have an associated RJ (ReproducibilityJournal) index, similarly re?ecting the number of papers it publishes that are ultimately found to be reproducible. The R indexes could also be divided by the total number of publications (per scientist or journal) to yield R%s and R%J indices, re?ecting the proportion of total papers and output ultimately found to be reproducible."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Reproducibility index (both at researcher and journal level),"A potential advantage of this index is that it does not matter whether one regularly publishes only in high impact journals or simply does careful meaningful science published in subspecialty or mid-tier general science journals. Although no metric is likely to be without ?aws or critics, pursuing high-quality reproducible science, without formally measuring reproducibility, is not likely to be successful. As Begley and Ioannidis have noted, ‘‘We get what we incentivize’’ (Begley and Ioannidis, 2015), and if we fail to measure and incentivize careful reproducible science, it is unlikely we will change the landscape of our current problematic scienti?c enterprise."
How to Make More Published Research True,Article,Reproducibility should be built into study design upfront,The prospect of replication needs to be considered and incorporated up front in designing the research agenda in a given field.
How to Make More Published Research True,Article,Reproducibility should be built into study design upfront,"Improvements in study design standards could improve the reliability of results. For example, for animal studies of interventions, this would include randomization and blinding of investigators. There is increasing interest in proposing checklists for the conduct of studies to be approved, making it vital to ensure both that checklist items are indeed essential and that claims of adherence to them are verifiable."
Open is not enough,"Report, policy document or website",Reproducibility should be built into study design upfront,"Our own experience from opening up vast volumes of data is that openness cannot simply be tacked on as an afterthought at the end of the scientific endeavour. In addition, openness alone does not guarantee reproducibility or reusability, so it should not be pursued as a goal in itself. Focusing on data is also not enough: it needs to be accompanied by software, workflows and explanations, all of which need to be captured throughout the usual iterative and closed research lifecycle, ready for a timely open release with the results."
Open is not enough,"Report, policy document or website",Reproducibility should be built into study design upfront,"We argue that physics analyses ideally should be automated from inception in such a way that they can be executed with a single command. Automating the whole analysis while it is still in its active phase permits to both easily run the ‘live’ analysis process on demand as well as to preserve it completely and seamlessly once it is over and the results are ready for publication. Thinking of restructuring a finished analysis for eventual reuse after its publication is often too late. Facilitating future reuse starts with the first commit of the analysis code. This is the purpose served by the Reusable Analyses service, REANA: a standalone component of the framework dedicated to instantiating preserved research data analyses on the cloud. While REANA was born from the need to rerun analyses preserved in the CERN Analysis Preservation framework, it can be used to run ‘active’ analyses before they are published and preserved."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Reproducibility should be built into study design upfront,"Design and methods: Incorporate best reproducibility practices early in the research design; Endorse compliance with established methodological guidelines; Reinforce standardised study design, protocols, etc.; Foster the use of better statistics, value of samples, methodologies; Pre-registration of protocols, analysis plans, etc."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,The connectivity of scholarly communication infrastructure with researchers' workflow and research tools can be improved,"Better connecting scholarly communication infrastructure with researchers’ work?ow and research tools is recognised by publishers as a way to promote transparency and reproducibility, and publishers are increasingly working more closely with research work?ow tools (Hrynaszkiewicz et al. 2014)."
Reproducibility and reliability of biomedical research - improving research practice,"Report, policy document or website",The connectivity of scholarly communication infrastructure with researchers' workflow and research tools can be improved,Technology and infrastructure to help deliver better reproducibility. This might include shared virtual lab environments and new tools for data capture and sharing.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Concept of 'pre-reproducibility',"Best practices in the reproducibility continuum start far before publication of scientific results to make them ‘reasonably available’. The concept of ‘pre-producibility’ is sometimes used to describe a set of measures that aim to ensure accountability and quality at the earliest possible stage. The ‘pre’ phase is crucial for the success of policy action to increase reproducibility. The ‘pre’ includes documenting the scientific process at the earliest stage of research before results, including by pre-prints, preregistration, data management plans (DMPs), journal and funder guidelines, dedicated grant support, investment in human resources (reproducibility experts, statistics training, expert evaluators), computational reproducibility practices, and universal technical tools (e.g. persistent identifiers)."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Concept of 'pre-reproducibility',"Not all parts of the research process need to be or will be perfectly accounted for; even ex-ante mechanisms such as study pre-registration can only ensure integrity but not complete accountability. There are limits to prescription even in the best case scenario: while one can describe the scientific process accurately and completely, circumstances do change during studies and make it impractical to report every single change."
Open is not enough,"Report, policy document or website",Researchers should define their reproducibility aims early on,"The definition of reproducibility goals early on is essential for ensuring future reusability of scientific results. Questions to consider are: what do you produce? What is the amount of collaborative work and personnel turnover? Would you like to achieve reproducibility and reuse internally or even externally? Choosing an appropriate and balanced reproducibility strategy for an analysis can involve a number of considerations, such as the available resources, the required level of detail, the reuse value of the processed data, the analysis results and so on. Many funding agencies tend to demand data management plans and it is worthwhile if we can use this requirement to our advantage by including it in our daily routine."
Research integrity nine ways to move from talk to walk,"Report, policy document or website",Support for reproducibility should be available across all stages of research,The University of Luxembourg has research-integrity coaches available for consultation at all stages of planning and publishing a project.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Support for reproducibility should be available across all stages of research,Expand the perception of integrity to include methodological aspects that guard against cognitive biases.
Towards reproducibility in recommender-systems research,Article,Building clear guidance and practical instructions for researchers (e.g. guidelines),"We discuss these ?ndings and conclude that to ensure reproducibility, the recommender-system community needs to (1) survey other research ?elds and learn from them, (2) ?nd a common understanding of reproducibility, (3) identify and understand the determinants that affect reproducibility, (4) conduct more comprehensive experiments, (5) modernize publication practices, (6) foster the development and use of recommendation frameworks, and (7) establish best-practice guidelines for recommender-systems research."
Reproducible Research Practices and Transparency across the Biomedical Literature,Article,Building clear guidance and practical instructions for researchers (e.g. guidelines),"It is likely that most scientists are now aware ofthe need to respond to the calls to improve research transparency and reproducibility. However, it is possible that many are unsure as to what they need to do or change in concrete and practical terms. This confusion exists in spite ofthe presence ofmany reviews and commentaries on the problems related to research transparency. We hope that our report can provide a straight-forward “to-do” list about indicators that are worth improving."
The earth is flat (p > 0:05): Significance thresholds and the crisis of unreplicable research,Article,Building clear guidance and practical instructions for researchers (e.g. guidelines),"Although there are hundreds of papers arguing against null hypothesis significance testing, we see more and more p-values (Chavalarias et al., 2016) and the ASA feeling obliged to tell us how to use them properly (Goodman, 2016; Wasserstein & Lazar, 2016). Apparently, bashing or banning p-values does not work. We need a smaller incremental step that at the same time is highly efficient. There are not many easy ways to improve scientific inference, but together with Higgs (2013) and others, we believe that removing significance thresholds is one of them."
Open is not enough,"Report, policy document or website",Building clear guidance and practical instructions for researchers (e.g. guidelines),"Incorporate best practices early in your research Adopting preservation and reproducibility practices and tools early in the research development process benefits the project and the research proponents. Invest time at the beginning of a project to do the groundwork, and document the planned outputs and how they could be organized, preserved and shared in order to support your reproducibility goals. Mentor good practice and demonstrate its usefulness. For example, use your bespoke preservation and reproducibility practices to familiarize new people with ongoing and past analyses in your team. Ensure verification and validation of your code before running new analyses. Use a version control system and continuous integration. Follow the reproducibility manifesto and similar guidelines."
Reproducibility literature analysis - a federal information professional perspective,Article,Building clear guidance and practical instructions for researchers (e.g. guidelines),"A framework for a systematic process to guide researchers and reviewers in assessing, documenting, and mitigating the sources of uncertainty in a study enhance comparability and reproducibility (Plant et al., 2018). Experimental design features should enhance, or facilitate inference about, the reproducibility and generalizability of the expected results (Würbel, 2017). [Related terms: pre-registration of results; case study]"
How we can make ecotoxicology more valuable to environmental protection,Note,Building clear guidance and practical instructions for researchers (e.g. guidelines),Create a checklist of a good quality study prior to commencing experimental work (see Section 12.1. A.: What Journals Can Do; Example in Table 1) and have a plan to meet those requirements.
How we can make ecotoxicology more valuable to environmental protection,Note,Building clear guidance and practical instructions for researchers (e.g. guidelines),"Become familiar with testing standards standards (e.g., Japanese Ministry of Agriculture, Forestry and Fisheries (JMAFF), Of?ce of Chemical Safety and Pollution Prevention (OCSPP), Organisation for Economic Co-operation and Development (OECD) and the American Society for Testing and Materials (ASTM)) for your test organisms, so that you are aware of performance requirements and minimal expectations around experimental design. Investigate whether or not regulatory bodies have criteria for evaluating published literature information (e.g., ECA 2012, EFSA 2013, US EPA 2011) and report this information in your study."
How we can make ecotoxicology more valuable to environmental protection,Note,Building clear guidance and practical instructions for researchers (e.g. guidelines),Advocating for a set of consistent toxicity test methods across jurisdictions that will be the agreed initial screen characterization of the toxicity of a compound to a particular organism.
Open is not enough,"Report, policy document or website",Good research practices to enable reproducibility should be specific to each discipline,"We argue that having the reuse of research results as a goal requires the adoption of new research practices during the data analysis process. Such practices need to be tailored to the needs of each given discipline with its particular research environment, culture and idiosyncrasies."
Research integrity nine ways to move from talk to walk,"Report, policy document or website","Institutions should have comprehensive reproducibility plans outlining how policies will be implemented, maintained and evaluated","To ensure that new procedures and policies work as intended, institutions need a comprehensive plan that makes sure the broad goals don’t get lost. It should specify how policies will be implemented, maintained and evaluated."
Recommendations for open data science,Note,"Checklists or guidelines, including their technological implementation, may help authors, reviewers, editors and journals","Reviewers also have a responsibility to ensure that computational methods are adequately reported. One suggestion is that journals provide authors and reviewers with a ‘computational reproducibility checklist’, similar to checklists already provided for statistical analyses. This checklist may ask whether the source code and data for all tools and analyses are available, whether new tools have been deposited in public package managers, and whether sufficient documentation and user manuals are available. Failure to comply with this checklist should disqualify a study from publication."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Checklists or guidelines, including their technological implementation, may help authors, reviewers, editors and journals",Improving the quality and objectivity of the peer-review process by implementing reporting guidelines and checklists and using technology to identify misconduct.
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Checklists or guidelines, including their technological implementation, may help authors, reviewers, editors and journals","An alternative approach to this problem taken by the multidisciplinary science journal Nature was to introduce a standardised editorial checklist to promote transparent reporting that could be applied to many different study designs and research disciplines. The checklist was developed by the journal in collaboration with researchers and funding agencies (Anon 2013) and is implemented by professional editors, who require that all authors complete it. The checklist elements focus on experimental and analytical design elements that are crucial for the interpretation of research results."
How we can make ecotoxicology more valuable to environmental protection,Note,"Checklists or guidelines, including their technological implementation, may help authors, reviewers, editors and journals","Journals should work with all stakeholders to create clear minimal standards for publishing ecotoxicology studies in concert with the peer review process. This could be implemented through a formal checklist for authors prior to submission. Ideally, all journals would implement the same standards. Journals could screen submissions using their checklist and if their criteria are not met, the paper will not be reviewed."
Recommendations for open data science,Note,Researchers need training in data science,"The primary source of training often comes from the laboratory environment itself. For wet-lab work, students are usually trained in practices such as keeping a lab notebook and sharing protocols. Similar efforts must now be invested for computational work. For instance, tools and analyses should be reviewed within the lab for quality and correctness. One suggestion is to maintain a lab Github repository where analyses are discussed as ‘issues’ and peer-reviewed via ‘pull-requests’. Another suggestion is to hold lab-wide or institute-wide data science tutorials. Finally, trainees can take advantage ofonline learning platforms (e.g. Software Carpentry) that provide freely available training in tools and practices for data-driven research."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Researchers should (want to) take part in the building and evaluation of new processes e.g. reporting guidelines,"One unambiguous outcome ofthe survey was that ISEV board members and JEV editors are not encouraged to update MISEV2014 alone. This approach was endorsed by only 4% of respondents. Thirty-five percent of participants asked the ISEV board to include ISEV members who are not on the board, and 62% felt that the entire ISEV community should be invited to participate. Of the latter group, approximately three in four said that community voting would not be necessary, whereas one in four favoured community-wide voting on specific points."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Researchers should (want to) take part in the building and evaluation of new processes e.g. reporting guidelines,"Guidelines published in 2008 and updated in 2012 and 2016, now include several thousand authors who reviewed the text and offered changes in a carefully guided manner. Klionsky’s staged writing process and a distributed author invitation plan ensure efficient progress but broad embrace of the global research body. In our view, the type of community consensus and buy-in fostered by this approach is exactly what we feel would be most valuable to EV researchers in an upcomingmanifestation ofthe MISEV guidelines and also consistent with what the"
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Researchers should (want to) take part in the building and evaluation of new processes e.g. reporting guidelines,MISEV survey respondents indicated about their willingness and desire to be involved.
Our path to better science in less time using open data science tools,Article,Researchers should (want to) take part in the building and evaluation of new processes e.g. reporting guidelines,Engagement may best be approached as an evolution rather than as a revolution that may never come.
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,Results should be hidden from editors and reviewes so that acceptance is based on scientific rigour alone,"The purpose of scientific enquiry is to estimate the presence and size of causal associations, and results from studies designed and conducted to the highest standards of scientific rigour will provide the most reliable and informative estimates. Thus, for the optimal advancement of science it seems logical that decisions regarding what to publish would be better based on judging quality, rather than results [11]. One way to achieve this would be ‘results-free’ review, where results are hidden from editors and reviewers, forcing reviewer reports and editorial decisions to be based on the scientific rigour of the study design alone."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Reviewers should ask surprising findings to be tested with available data,"In addition, in cases of especially surprising findings, findings that could have influence on public health policy or medical treatment decisions, or findings that could be tested using data from another existing data set, reviewers should consider requesting replication of the finding by the group before accepting the manuscript."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Reviewers should ask surprising findings to be tested with available data,"Validation. For empirical papers, all exploratory results would be validated against an independent validation data set that was not examined before validation. For methodological papers, the approach would follow best practices for reducing overly optimistic results method would be validated against benchmark data sets and compared with other state-of-the-art methods."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Funders should support infrastructure to shift the financial burden of reproducibility,"Funders need to fund and incentivise reproducibility to assist the efforts of results producers for the benefit of users by shifting costs from producers via intermediaries; by ensuring that there are incentives for re-use of the results, rather than continuous re-doing; by supporting basic infrastructures for the preservation and sharing of underlying data and method, among other actions."
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Simplicity and versatility (e.g., include multiple versions of tools) of softwares to increase transparency is important","Over the last year, we have increased the breadth and quality of tools available on the Public Galaxy server by adding 400 new tools to the server. More than 650 tools now enable users to analyze a wide variety of different genomic data. Very simple tools perform text manipulation and statistical operations, but the majority of tools on the server are for analysis ofhigh-throughput genomic datasets from current DNA sequencers."
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Simplicity and versatility (e.g., include multiple versions of tools) of softwares to increase transparency is important",We now use the Tool Shed as the source for all new tools and version updates; the Tool Shed greatly simplifies tool deployment and ensures versioning and reproducibility.
"The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2016 update",Article,"Simplicity and versatility (e.g., include multiple versions of tools) of softwares to increase transparency is important","All Galaxy tools use a single interface to simplify tool integration and maximize usability. In the past year, we have rewritten the tool interface to be significantly more responsive and dynamic, and we believe that these changes have substantially increased the interface’s usability. We have recently extended this interface to include citations so that scientists can reference the methods used in their analysis. To ensure reproducibility, multiple versions oftools can be installed and users can run any installed version."
"Increasing value and reducing waste in research design, conduct, and analysis",Article,Statisticians should be involved across all stages of research,"Statisticians and methodologists should be involved in all stages of research. This recommendation has been repeatedly discussed, mostly for clinical trials, but it applies to all types of studies. Enhancement of communication between methodologists and other health scientists is also important. Medical and public health schools and graduate programmes"
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Support open research including OA and publication of negative results,Support full OA publication.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Support open research including OA and publication of negative results,"Establish a quality assurance system for Open Access journals, to avoid predatory journals and non-peer-reviewed / low quality journals."
Preventing the ends from justifying the means: Withholding results to address publication bias in peer-review,Editorial,The criteria determining publication must be aligned with those for conducting rigorous scientific practice,"In order to improve the quality and reliability of published research, the criteria determining publication must be aligned with those for conducting rigorous scientific practice."
Detecting and avoiding likely false-positive findings – a practical guide,Article,The number of analyses performed should be known,"When the total number of statistical tests conducted is known (e.g. 10 tests), then it is possible to calculate the probability of obtaining at least one signi?cant result by chance alone (1?0.9510 =40%), and it is possible to adjust ?-levels (0.05/10=0.005) for each test to ensure that the probability of making one or several Type I errors remains at about 5% (1?0.99510 =4.9%). This adjustment is known as the classical Bonferroni correction (Dunn, 1961)."
Detecting and avoiding likely false-positive findings – a practical guide,Article,The number of analyses performed should be known,"We believe that this practice could be stopped most effectively by adequate measures from funding bodies. In an ideal world, all measures of effect size would be reported (not necessarily in peer-reviewed journals) and the respective raw data would be made openly available (see also Morey et al., 2016)."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,The paper should have the data embedded for figure generation,The paper would be written using a literate programming technique in which the code for figure generation is embedded within the paper and the data depicted in figures are transparently accessible.
Replication in Second Language Research: Narrative and Systematic Reviews and Recommendations for the Field,Review,The value of replication studies should be promoted and data should be available to run them,Recommendation: Increase funding from institutional through to international levels to promote replication as an integral part of the research process.
The possibility and desirability of replication in the humanities,Note,The value of replication studies should be promoted and data should be available to run them,Funding agencies need to make proposals for humanistic replication studies eligible and must demand that funded primary studies are replicable.
The possibility and desirability of replication in the humanities,Note,The value of replication studies should be promoted and data should be available to run them,Probably funding agencies are essential in incentivizing the changes we advocate: they can simply demand pre-registration and making the data available mandatory by adding this to their conditions for studies they sponsor. And journals publishing humanistic research can contribute meaningfully by adopting registered reports.
Detecting and avoiding likely false-positive findings – a practical guide,Article,The value of replication studies should be promoted and data should be available to run them,"Funding agencies and journal editors focus on novelty. This is particularly hard to justify on the part of funders since failing to invest in replication means failing to seek robust answers to questions they already have made a commitment to answering. If the answer truly was worth paying for, then the replication should also be worth paying for (Nakagawa & Parker, 2015). Promoting the funding of replication studies would be relatively straightforward. Most obviously, agencies could set aside funds for important and well-justi?ed replications. Agencies could also incentivize replication by preferentially funding novel studies when those studies rest on well-replicated foundations (Parker, 2013). They could also preferentially fund researchers whose prior work has often been successfully replicated."
Detecting and avoiding likely false-positive findings – a practical guide,Article,The value of replication studies should be promoted and data should be available to run them,"Hence, researchers should recognize the value of unbiased reporting and funding bodies should reward such practice during the review process. The latter will have to think ofways ofassessing researcher performance in terms of scienti?c rigour and integrity, because the current assessment in terms ofproductivity and impact causes unwanted natural selection pressure in favour of bad science (Smaldino & McElreath, 2016)."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,The value of replication studies should be promoted and data should be available to run them,"Second, outstanding replication study articles make a strong argument about what the field learns from the replication.1 In particular, the authors need to be persuasive that the replication will allow us to learn something new—not merely to confirm what we essentially already know. A replication study article needs to make a strong case that the replication advances our understanding of the phenomena under investigation, such as by answering previously unanswered questions or suggesting new and important issues that merit further exploration."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,The value of replication studies should be promoted and data should be available to run them,"Articles published in top-tier journals should inform the field in substantial and significant ways, even when they report the results of replication studies."
When and why replication studies should be published: Guidelines for mathematics education journals,Note,The value of replication studies should be promoted and data should be available to run them,"But if the replication study article makes a convincing argument that (a) the methodology of the original study was flawed, (b) the selection of study participants or contexts in the original raises questions about the findings, or (c) the results are counter to other established results, this provides a more persuasive argument in the article in favor of replication."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,The value of replication studies should be promoted and data should be available to run them,"The availability, evaluation, and consideration of the entire body of evidence, including rigorous studies that detected reduced or no effects, is critical for the scientific advancement of knowledge in the respective field."
"Replication Research, Publication Bias, and Applied Behavior Analysis",Note,The value of replication studies should be promoted and data should be available to run them,"There is evidence to suggest that systematic reviews that do not incorporate gray literature may obtain inflated intervention effect sizes that likely are a function of the file drawer effect (Gage et al., 2017;Sham&Smith, 2014). We therefore believe it is critical that future systematic reviews of ABA studies include gray literature to provide the most conservative and complete estimation ofeffect sizes for ABA interventions. Gray literature should be included in systematic reviews and held to the same standards of methodological rigor as published research."
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Provide dedicated funding for replication studies,Provide dedicated funding for the reproduction/replication of studies.
Reproducibility of Scientific Results in the EU - Scoping report,"Report, policy document or website",Provide dedicated funding for replication studies,"Systemic incentives: Finance meta reviews and systematic checks on reproducibility of funded research; Fund meta-research to improve the research process; Establish and maintain quality data infrastructures (curation, archiving, etc.); Establish reporting systems for witnessed malpractice."
Detecting and avoiding likely false-positive findings – a practical guide,Article,The value of reporting multiple analyses results need to be accepted by reviewers and editors,"Justi?ed skepticism from reviewers creates an incentive for reduced transparency in scienti?c publications, thereby lowering the overall utility of the reported work. This problem could be mitigated if reviewers and editors would acknowledge and appreciate the greater scienti?c value of a paper that comprehensively reports all outcomes of a study compared to the minimalistic presentation ofa single ?nding."
On the issue of transparency and reproducibility in nanomedicine,Letter,There should be consistency in reporting guidelines across all journals in a particular discipline,"Finally, standardization of the terminology used in MIRIBEL would also be welcomed."
How we can make ecotoxicology more valuable to environmental protection,Note,There should be consistency in reporting guidelines across all journals in a particular discipline,"Journals should work with all stakeholders to create clear minimal standards for publishing ecotoxicology studies in concert with the peer review process. This could be implemented through a formal checklist for authors prior to submission. Ideally, all journals would implement the same standards. Journals could screen submissions using their checklist and if their criteria are not met, the paper will not be reviewed."
Updating the MISEV minimal requirements for extracellular vesicle studies: building bridges to reproducibility,Editorial,Triangulation - Researchers should validate their findings with different analyses and technologies,"For characterization of single vesicles, as a means to assess population heterogeneity, MISEV2014 called for the use of at least two different but complementary technologies. For example, electron or atomic force microscopy could be paired with one of the single particle tracking methods. Furthermore, close-up images should be accompanied by wide-field views to allow assessment of heterogeneity."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Triangulation - Researchers should validate their findings with different analyses and technologies,"First, they may conduct multiple studies to test the same hypothesis, thus providing not only evidence of validity under different conditions, but also reducing the opportunities for HARKing. In academic disciplines investigating behaviors of individuals, such as organizational psychology, organizational behavior, and human resource management, it is established good practice to include multiple studies to test a new hypothesis (see, e.g., the Journal of Applied Psychology)."
What's in a p? Reassessing best practices for conducting and reporting hypothesis-testing research,Review,Triangulation - Researchers should validate their findings with different analyses and technologies,"Guideline: Authors are expected to conduct a variety of robustness tests to show that the signi?cant ?nding is not due to an idiosyncrasy of the selected empirical measures, model speci?cations and/or estimation strategy."
Scanning the horizon: Towards transparent and reproducible neuroimaging research,Article,Encourage validation rather than constrain methods and diversity of analysis,"Although there are concerns regarding the degree to which flexibility in data analysis may result in inflated error rates, we do not believe that the solution is to constrain researchers by specifying a particular set of methods that must be used. Many of the most interesting findings in fMRI have come from the use of novel analysis methods, and we do not believe that there will be a single best workflow for all studies; in fact, there is direct evidence that different studies or individuals will probably benefit from different workflows. We believe that the best solution is to allow flexibility but require that all exploratory analyses be clearly labelled as such, and strongly encourage validation of exploratory results (for example, through the use of a separate validation data set)."
Minimum statistical standards for submissions to Neuroimage: Clinical,Editorial,Encourage validation rather than constrain methods and diversity of analysis,"Our intention here is not to be unduly prescriptive, and we recognise that different techniques will be appropriate for different experiments. However, manuscripts that do not meet even a bare minimum of statistical rigour, as outlined above, will normally be returned to authors without review."
What you see is what you get? Enhancing methodological transparency in management research,Review,Encourage validation rather than constrain methods and diversity of analysis,"As noted by Freese (2007b), while researchers today have more degrees of freedom regarding data-analytic choices than ever before, decisions made during analysis are rarely disclosed in a transparent manner."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,We should become more acctpting of uncertainty and modest in interpreting results,"One way to reduce selective reporting and attention is to maintain that all results are uncertain. If we obtain a small Pvalue, a large effect estimate, or a narrow interval estimate— or even all three—we should not be confident about textbook inferences from these results. In one of the next replications, p will be large, the effect estimate will be small, or the interval estimate wide, and thus the textbook inferences will shift dramatically due to random variation or to assumptions we have notmodeled. Because ofthis uncertainty, there is simplynoneed to selectively report studies based on statistical results. We should thus “move toward a greater acceptance of uncertainty and embracing of variation” (Gelman 2016) and focus on describing accurately how the study was conducted, what problems occurred (e.g., nonresponse of some subjects, missing data), and what analysis methods were used, with detailed data tabulation and graphs, and complete reporting of results. The advent ofonline supplements and preprint servers eliminate the common excuse that space limitations prevent reporting such detail."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,We should become more acctpting of uncertainty and modest in interpreting results,"Acknowledge that our statistical results describe relations between assumptions and the data in our study,and that scientific generalization from a single study is unwarranted."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,We should become more acctpting of uncertainty and modest in interpreting results,"We should thus communicate our limited conclusions about our data, not our generalized inferences about some illdefined universal population. And decisions to communicate and interpret a result should not be based on P-values, nor on any other statistic. Presentations that start with analysis plans that were formulated before the analysis (pre-analysis protocols) can help strengthen both the validity and credibility of our inferences. The reported description ofour results will be a good description ifit is complete and honest. If we think we did a good study, we should thus be modest about our conclusions, but be proud about our painfully honest and thorough description and discussion ofour methods and ofour data."
Inferential Statistics as Descriptive Statistics: There Is No Replication Crisis if We Don’t Expect Replication,Article,We should become more acctpting of uncertainty and modest in interpreting results,"Clear signs of overconfidence are formulations like “we proved” or “we disproved” or “we rejected” a hypothesis, or “we have shown” or “demonstrated” a relation exists or is explained in some manner. So are “there was no effect/no association/no difference” (which almost always would be an impossible proof of the null hypothesis), and “our study confirms/validates/invalidates/refutes previous results” (because a single study has nothing definitive, it can only add one further data point to the larger picture; at most it can be
“consistent/inconsistent with previous results”). If we find any of those or related phrases, we should question the interpretations being offered in the paper and search for arguments provided by the authors. If the main argument for a conclusion is that the results were “significant” or “not significant,” this does not automatically mean that the study is bad. But it does flag the paper as likely providing an unreliable interpretation of the reported results."
Never Waste a Good Crisis: Confronting Reproducibility in Translational Research,Note,Awareness alone won't solve the problem,"While accepting the notion that sunlight is the best disinfectant for many problems, simply highlighting existing challenges in an anecdotal way, or publishing guidelines, commentaries, or position papers, while perhaps helpful, seems unlikely to move the needle in a meaningful way."
Open is not enough,"Report, policy document or website",Awareness alone won't solve the problem,"Open science and reproducible research have become pervasive goals across research communities, political circles and funding bodies. The understanding is that open and reproducible research practices enable scientific reuse, accelerating future projects and discoveries in any discipline. In the struggle to take concrete steps in pursuit of these aims there has been much discussion and awareness-raising, often accompanied by a push to make research products and scientific results open quickly. Although these are laudable and necessary first steps, they
are not sufficient to bring about the transformation that would allow us to reap the benefits of open and reproducible research. It is time to move beyond the rhetoric and the trust in quick fixes and start designing and implementing tools to power a more profound change."
A manifesto for reproducible science,Review,Changing incentives requires a coordinated effort involving different stakeholders,"Funders, publishers, societies, institutions, editors, reviewers and authors all contribute to the cultural norms that create and sustain dysfunctional incentives. Changing the incentives is therefore a problem that requires a coordinated effort by all stakeholders to alter reward structures. There will always be incentives for innovative outcomes — those who discover new things will be rewarded more than those who do not."
How to Make More Published Research True,Article,"Data sharing may give rise to significant issues around Data ownership, Data sensitivity, Ethics and confidentiality, Intellectual property, Misuse of shared data","The potential of multiple analysts performing contradicting analyses, difficulties with de-identification of participants, and the potential for parties to introduce uncertainty for results that hurt their interests, as in the case of diesel exhaust and cancer risk."
Publishers’ Responsibilities in Promoting Data Quality and Reproducibility,Article,"Data sharing may give rise to significant issues around Data ownership, Data sensitivity, Ethics and confidentiality, Intellectual property, Misuse of shared data",Medical science researchers reported that copyright and licencing (data ownership) issues were their biggest challenge.
Promises and pitfalls of data sharing in qualitative research,Note,"Data sharing may give rise to significant issues around Data ownership, Data sensitivity, Ethics and confidentiality, Intellectual property, Misuse of shared data","The movement to promote reproducible research in the medical and public health literature has lagged, perhaps for myriad reasons. First, concerns are frequently voiced about intellectual property protections and/or the potential hazard of disclosing protected health information (Hrynaszkiewicz et al., 2010; Mello et al., 2013; Tudur Smith et al., 2015). Second, because medical and public health research can often carry enormous ?nancial implications for speci?c products (Rennie, 1997; Shuchman, 2005) or entire industries (Kaiser, 1997; Michaels and Monforton, 2005; Muggli et al., 2001) that are implicated in the ?ndings, requests for data may be driven by ?nancial motivations that extend well beyond any disinterested concerns about science for science's sake."
Promises and pitfalls of data sharing in qualitative research,Note,"Data sharing may give rise to significant issues around Data ownership, Data sensitivity, Ethics and confidentiality, Intellectual property, Misuse of shared data","Because qualitative study designs often lend themselves to the in-depth study of highly sensitive subject material (Kelly et al., 2011; King et al., 2013; Parkinson, 2013; Wade et al., 2005), ?eld notes and interview transcripts would need to be anonymized prior to dissemination in order to conform with prevailing legal and ethical guidelines. Institutional Review Board concerns about participant anonymity, discussed in the PLOS policy (Bloom et al., 2014), have been identi?ed as a leading barrier to data sharing. Consequently, investigators lacking proper guidance on how to comply with data sharing guidelines in a way that provides adequate anonymity protections may simply default to data withholding."
Promises and pitfalls of data sharing in qualitative research,Note,"Data sharing may give rise to significant issues around Data ownership, Data sensitivity, Ethics and confidentiality, Intellectual property, Misuse of shared data","Because elites are more empowered to articulate concerns about con?dentiality and disclosure, data sharing could unintentionally perpetuate power differentials in which health program bene?ciaries endure as research subjects while health program funders and implementers remain understudied (Schneider and Aguiar, 2012)."
Practical Computational Reproducibility in the Life Sciences,Note,"Even computing notebooks do not guarantee reproducibility (unavailable softwares, different results for different computers, etc.)","Yet most still fall short from achieving full reproducibility because they fail to preserve the full computing environment in which analyses have been performed. For example, consider an analysis executed on Galaxy, a popular web-based scienti?c workbench. An analysis executed on a particular Galaxy server might include tools not found elsewhere and therefore cannot be reproduced outside that server. Another example is a Jupyter notebook that includes tools speci?c to a particular platform and a distinct set of software libraries. There is no guarantee that this notebook will produce the same results on a different computer."
"Irreproducibility of published bioscience research: Diagnosis, pathogenesis and therapy",Note,Post-publication peer review may open the door to anonymous harrassment,"second new element relates to new venues for “post publication peer review”, such as PubMed Commons [55], PubPeer [56] and other sites, where participants discuss published data in variably moderated online communities. The potential ability of such venues to enhance scienti?c communication seems obvious. Although many discussions on PubPeer have questioned the validity of published data and some have even led to retractions, the fact that most discussants are anonymous has been challenged and is a topic of ongoing debate [57]. Overall, it seems likely that a robust capacity for extended online discussion of published research will eventually advance scienti?c progress, and may hasten discovery of problems with some papers, while creating unfortunate opportunities for anonymous and misdirected harassment in some cases."
Promises and pitfalls of data sharing in qualitative research,Note,Proprietary data often cannot be shared,"The movement to promote reproducible research in the medical and public health literature has lagged, perhaps for myriad reasons. First, concerns are frequently voiced about intellectual property protections and/or the potential hazard of disclosing protected health information (Hrynaszkiewicz et al., 2010; Mello et al., 2013; Tudur Smith et al., 2015). Second, because medical and public health research can often carry enormous ?nancial implications for speci?c products (Rennie, 1997; Shuchman, 2005) or entire industries (Kaiser, 1997; Michaels and Monforton, 2005; Muggli et al., 2001) that are implicated in the ?ndings, requests for data may be driven by ?nancial motivations that extend well beyond any disinterested concerns about science for science's sake."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"The substantive ways in which qualitative and quantitative data differ should be considered when assessing the extent to which qualitative and mixed methods researchers should be expected to adhere to data sharing policies developed with purely quantitative studies in mind. We outline several of the most critical concerns below, while also suggesting possible modi?cations that may help to reduce the probability of unintended adverse consequences and to ensure that the sharing ofqualitative data is consistent with ethical standards in research."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"The data collected in qualitative studies are typically obtained through in-depth interviews, focus groups, direct observation, document review, and audio recording review. These data, while typically not aimed at establishing generalizability, lend themselves to generating new theoretical insights about certain phenomena in greater depth and detail than is possible through quantitative designs (Patton, 2002). While complementary to other forms of social measurement, these data are also neither collected nor analyzed in as linear a manner, and it has been argued that the concept of reliability does not directly translate from the quantitative (rationalistic) to the qualitative (naturalistic) paradigm (Guba and Lincoln, 1981)."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"Veri?cation does not translate well to a data sharing policy for qualitative studies. Given the inherently intersubjective nature of qualitative data collection, the iterative nature of qualitative data analysis, and the unique importance of interpretation as part of the core contribution of qualitative work, veri?cation is likely to be impossible in the setting of qualitative research. We discuss two principal reasons below."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"First, some scholars have argued that interview transcripts, even when accompanied by detailed ?eld notes, cannot represent with suf?cient ?delity the actual interview that took place. Even audio and video recordings, which are generally considered the most complete observational data that can be captured, cannot convey valuable tactile and/or olfactory data obtained in the ?eld (Bernard and Ryan, 2009). Drawing on focus groups conducted with qualitative researchers, Broom et al. (2009) showed that many of them were of the immoderate opinion that their transcript data were “an encoded account only decipherable to the individual who collected it” (p.1170). According to this understanding, we should question the extent to which interview transcripts may be considered “raw data” for external investigators to use in the same manner as a dataset taken from a randomized controlled trial of the latest unoriginal antidepressant medication"
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"The interview transcripts disseminated to external investigators are unlikely to be the data they would have collected had they conducted the study themselves. A qualitative study guided by the method of grounded theory, for example, follows an inductive process with concurrent review of the data being collected, ?ltering of the data for relevance and meaningfulness, and grouping and naming of patterns observed in the data (Glaser and Strauss, 1967)."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"Applying these standards uncritically, one might presume that data sharing involves providing the following in an online supplementary appendix: interview guides and interview transcripts, in the original language and in the translated language of the investigators (if different from the original); ?eld notes; data used, if any, to establish inter-coder reliability; full code books; and documents, if any, describing the process of open coding, selection of codes for inclusion in the ?nal codebook, and category construction. The “audit trail” supports reliability and validity, so even if it is recognized that no two groups would conduct identical qualitative studies, the information available to external investigators would enable them to understand how the study authors arrived at the published conclusions. Most computer-assisted qualitative data analysis software packages offer export functions that enable users to save an entire “project” (e.g., raw data, codebook, coding links, and memos), which could facilitate dissemination. While these types of maneuvers might be consistent with a data sharing policy, there are a number of challenges that could hamper their implementation in practice. Below we highlight the most signi?cant challenges facing data sharing in qualitative research.
Data sharing policies should carefully consider the potential effects of data sharing on study participants. Most qualitative researchers use respondent validation (e.g., reviewing emerging themes and analyses with study participants or key informants) to ensure rigor, and the practice is highlighted as a key process component of qualitative research in most reporting checklists (Clark, 2003; O'Brien et al., 2014; Tong et al., 2007). This method of data sharing through member-checking of interim ?ndings is carefully supervised. In contrast, data sharing policies that make interview transcripts available to study participants in a completely unstructured fashion may have negative effects. Chiefamong these are the potential psychosocial consequences of compromising study participant anonymity."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"While Institutional Review Board restrictions are commonly cited to justify withholding of quantitative data (Campbell et al., 2002), in fact it may be possible to release de-identi?ed versions of transcripts that preserve the anonymity of qualitative study participants. The nature of any anonymization procedures would depend on the nature of the data collected and the extent to which the data can be linked with publicly available information to reveal speci?c identities. At a minimum, the anonymization procedures would entail redaction or alteration ofprotected health information and any speci?c encounter details that reveal, however indirectly, the identity of any of the parties to the encounter, with obfuscated information shown in brackets."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"Because interview transcripts contain verbatim quotations, it is likely that some transcripts cannot be suf?ciently anonymized to prevent deductive disclosure, or what Tolich (2004) has called violations of “internal con?dentiality.” That is, study participants could recognize themselves, their communities, or other study participants (if they belong to the same community) (Larossa et al., 1981). van den Hoonaard (2003) holds that anonymity is “a virtual impossibility in ethnographic research” (p.141). Depending on the sensitivity of the subject matter, deductive disclosure could result in harm to study participants and their relationships with others in the community. Ellis (1995), Scheper-Hughes (2000), and Stein (2010) have famously written about being angrily received by study participants over deductive disclosures following the publication of their celebrated books (Ellis, 1986; Scheper-Hughes, 1977; Stein, 2001)."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"small community must be ?rst secured from highly networked research gatekeepers, such as village leaders or community advisory boards. In these settings, a minor, idiosyncratic detail – such as a manner of speaking or a speci?c phrase – that is ofunknown signi?cance to the investigator (and therefore likely to go unredacted) could result in deductive disclosure and potential harm."
Promises and pitfalls of data sharing in qualitative research,Note,Qualitative data is not as simple to share and can be of limited value for replication,"Moreover, there are no standards in the ?eld for systematically documenting the hours of conversations, conference calls, and email exchanges required for code selection and category construction. Guidelines would need to be developed so that documentation of these procedures is uniform across studies. Larger qualitative and mixed methods studies would entail an even greater documentation burden."
How to Make More Published Research True,Article,Too much focus on replication can leave fewer resources for new research,"To give an extreme example, one could easily eliminate all false positives simply by discarding all studies with even minimal bias, by making the research questions so bland that nobody cares about (or has a conflict with) the results, and by waiting for all scientists in each field to join forces on a single standardized protocol."