Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
- University of Ottawa, Canada
- Federal University of Pelotas, Brazil
- University of Toronto, Canada
- McGill University, Canada
- Stanford University, United States
- Ottawa Hospital Research Institute, Canada
Abstract
Introduction: Reproducibility is a central tenant of research. We aimed to synthesize the literature on reproducibility and describe its epidemiological characteristics, including how reproducibility is defined and assessed. We also aimed to determine and compare estimates for reproducibility across different fields.
Methods: We conducted a scoping review to identify English language replication studies published between 2018-2019 in economics, education, psychology, health sciences and biomedicine. We searched Medline, Embase, PsycINFO, Cumulative Index of Nursing and Allied Health Literature – CINAHL, Education Source via EBSCOHost, ERIC, EconPapers, International Bibliography of the Social Sciences (IBSS), and EconLit. Documents retrieved were screened in duplicate against our inclusion criteria. We extracted year of publication, number of authors, country of affiliation of the corresponding author, and whether the study was funded. For the individual replication studies, we recorded whether a registered protocol for the replication study was used, whether there was contact between the reproducing team and the original authors, what study design was used, and what the primary outcome was. Finally, we recorded how reproducibilty was defined by the authors, and whether the assessed study(ies) successfully reproduced based on this definition. Extraction was done by a single reviewer and quality controlled by a second reviewer.
Results: Our search identified 11,224 unique documents, of which 47 were included in this review. Most studies were related to either psychology (48.6%) or health sciences (23.7%). Among these 47 documents, 36 described a single reproducibility study while the remaining 11 reported at least two reproducibility studies in the same paper. Less than the half of the studies referred to a registered protocol. There was variability in the definitions of reproduciblity success. In total, across the 47 documents 177 studies were reported. Based on the definition used by the author of each study, 95 of 177 (53.7%) studies reproduced.
Conclusion: This study gives an overview of research across five disciplines that explicitly set out to reproduce previous research. Such reproducibility studies are extremely scarce, the definition of a successfully reproduced study is ambiguous, and the reproducibility rate is overall modest.
Funding: No external funding was received for this work.
Data availability
Data and materials are available on the Open Science Framework (https://osf.io/wn7gm/).
-
Scoping review data on reproducibility of research in a 2018-2019 multi-discipline sampleOpen Science FrameworkDOI 10.17605/OSF.IO/WN7GM.
Article and author information
Author details
Funding
The authors declare that there was no funding for this work.
Copyright
© 2023, Cobey et al.
This article is distributed under the terms of the Creative Commons Attribution License permitting unrestricted use and redistribution provided that the original author and source are credited.
Metrics
-
- 1,169
- views
-
- 113
- downloads
-
- 5
- citations
Views, downloads and citations are aggregated across all versions of this paper published by eLife.
Download links
A two-part list of links to download the article, or parts of the article, in various formats.
Downloads (link to download the article as PDF)
Open citations (links to open the citations from this article in various online reference manager services)
Cite this article (links to download the citations from this article in formats compatible with various reference manager tools)
Epidemiological characteristics and prevalence rates of research reproducibility across disciplines: a scoping review of articles published in 2018-2019
eLife 12:e78518.
https://doi.org/10.7554/eLife.78518
Further reading
-
- Medicine
- Neuroscience
Neurodegenerative diseases are age-related disorders characterized by the cerebral accumulation of amyloidogenic proteins, and cellular senescence underlies their pathogenesis. Thus, it is necessary for preventing these diseases to remove toxic proteins, repair damaged neurons, and suppress cellular senescence. As a source for such prophylactic agents, we selected zizyphi spinosi semen (ZSS), a medicinal herb used in traditional Chinese medicine. Oral administration of ZSS hot water extract ameliorated Aβ and tau pathology and cognitive impairment in mouse models of Alzheimer’s disease and frontotemporal dementia. Non-extracted ZSS simple crush powder showed stronger effects than the extract and improved α-synuclein pathology and cognitive/motor function in Parkinson’s disease model mice. Furthermore, when administered to normal aged mice, the ZSS powder suppressed cellular senescence, reduced DNA oxidation, promoted brain-derived neurotrophic factor expression and neurogenesis, and enhanced cognition to levels similar to those in young mice. The quantity of known active ingredients of ZSS, jujuboside A, jujuboside B, and spinosin was not proportional to the nootropic activity of ZSS. These results suggest that ZSS simple crush powder is a promising dietary material for the prevention of neurodegenerative diseases and brain aging.
-
- Medicine
Inherited retinal degenerations (IRDs) constitute a group of clinically and genetically diverse vision-impairing disorders. Retinitis pigmentosa (RP), the most common form of IRD, is characterized by gradual dysfunction and degeneration of rod photoreceptors, followed by the loss of cone photoreceptors. Recently, we identified reserpine as a lead molecule for maintaining rod survival in mouse and human retinal organoids as well as in the rd16 mouse, which phenocopy Leber congenital amaurosis caused by mutations in the cilia-centrosomal gene CEP290 (Chen et al., 2023). Here, we show the therapeutic potential of reserpine in a rhodopsin P23H rat model of autosomal dominant RP. At postnatal day (P) 68, when males and females are analyzed together, the reserpine-treated rats exhibit higher rod-derived scotopic b-wave amplitudes compared to the controls with little or no change in scotopic a-wave or cone-derived photopic b-wave. Interestingly, the reserpine-treated female rats display enhanced scotopic a- and b-waves and photopic b-wave responses at P68, along with a better contrast threshold and increased outer nuclear layer thickness. The female rats demonstrate better preservation of both rod and cone photoreceptors following reserpine treatment. Retinal transcriptome analysis reveals sex-specific responses to reserpine, with significant upregulation of phototransduction genes and proteostasis-related pathways, and notably, genes associated with stress response. This study builds upon our previously reported results reaffirming the potential of reserpine for gene-agnostic treatment of IRDs and emphasizes the importance of biological sex in retinal disease research and therapy development.
