Translation from preclinical research to patients is challenging for many reasons (Denayer et al., 2014; Pound and Ritskes-Hoitinga, 2018). Biological complexity and the disparity between animal models of disease and humans accounts for some failures in translation, but not all (Kimmelman and London, 2011; Mullane and Williams, 2019; van der Worp et al., 2010). Other reasons include the fact that the evidence generated in preclinical efficacy studies is often weak due to the low numbers of experimental units or entities receiving a treatment (Bonapersona et al., 2019; Carneiro et al., 2018; Howells et al., 2014). Moreover, analyses are often not reported in full, leading to the selective reporting of outcomes and the exploitation of researcher degrees of freedom (Motulsky, 2014). False positives abound in such studies, and reported effects are often inflated (Dirnagl, 2020; Kimmelman et al., 2014; Turner and Barbee, 2019). Moreover, a strong bias against the publication of non-significant results augments this problem (Sena et al., 2010) and makes meta-analytic assessment of preclinical evidence difficult (Sena et al., 2014).

An obvious way to address some of these problem would be for other research groups to reproduce and then replicate preclinical studies before starting experiments on humans. By reproduce we mean to be in principle able to repeat the original study through in depth understanding of methods, protocols, and analytical pipelines used by the original research group. By replicate we mean to actually perform a study to see if the findings of the original study still hold. This potentially involves adapting some methods, protocols, and analytical pipelines (Nosek and Errington, 2020a; Patil et al., 2016): for example, when different animal strains are used or when environmental factors are changed (Voelkl et al., 2020). Reproducibility is thus a prerequisite for engaging in replications that will increase our confidence in a finding through its wider validity.

Here, we consider the role of replications in the context of the preclinical research trajectory for a potential treatment: such a trajectory is a series of experiments designed to generate evidence that will inform any decision about testing the treatment in humans. The experiments in a preclinical research trajectory typically include exploratory studies, toxicity studies, positive and negative controls, pharmacodynamics and kinetics, and are intended to generate evidence to support an inferential claim and refute possible alternatives. They can be performed on animal models (including invertebrates, zebrafish, nonhuman primates and, quite often, rodents) or with replacement methods (such as cell cultures and organoids).

Within this framework, replications strengthen two key characteristics of preclinical experimental evidence: validity and reliability. Validity refers to the degree to which an inference is true, and reliability refers to the quality and accuracy of the data supporting an inferential claim. In this article we describe strategies for preclinical research trajectories in which replications balance reliability and validity to foster preclinical and translational research in a way that is ethical and efficient. For example, consider a researcher who hypothesizes that a disease is caused by a metabolic product. A potential drug candidate will inhibit an enzyme that is involved in the relevant metabolic process. In an exploratory study, applying the drug in a knockout mouse model of the disease reduced the metabolic product and the health condition of the animals improved. A within-lab replication confirms these initial findings. The findings are reliable as initial data and replication support the inferential claim that the drug improves health conditions. However, this does not necessarily mean that the inference is valid, particularly when extrapolated to humans: as we will outline below, the validity of such an inference can be threatened on several levels during the preclinical research trajectory.

A number of large-scale replication projects have been conducted in psychology and social sciences (Camerer et al., 2016; Camerer et al., 2018; Open Science Collaboration, 2015). Those studies were performed with healthy subjects with little to no harm anticipated through participation. Consequently, increasing the number of tested subjects was not ethically problematic or overly expensive. The same is not true for projects that involve animals, so there have been relatively few large-scale replication projects in biomedical research. Moreover, the projects that have been started – such as the Reproducibility Project: Cancer Biology, the Brazilian Reproducibility Initiative, and the Confirmatory Preclinical Studies project – all take different approaches to identifying the studies to be replicated and to performing the replications (see Table 1 and Box 1).

Based on the results from the study being replicated and the stage in the preclinical research trajectory, the question is: what additional evidence is needed to ultimately decide to start trials in human subjects? Throughout a sequence of preclinical experiments, validity and reliability have to be adapted at each stage, and criteria are set so that we know whether to continue, to revise, or even completely break off the experiments (Figure 1). How, for example, could the previous experiments be improved? How many animals should be tested? Are additional controls needed? Should additional labs be involved? At all stages the aim should be to increase the validity of replications and test the reliability of the initial finding (Dirnagl, 2020; Kimmelman et al., 2014; Piper et al., 2019).

Figure 1

Download asset Open asset

Researchers usually start off with a study in exploratory mode. As not all details and confounders in such a study can be known upfront, the reliability and validity are potentially not fully optimized. However, even at these early stages researchers should implement strategies to mitigate risks of bias (Figure 1). After an initial phase, the question is: should we continue to test a particular claim? The criteria used to answer this question should be lenient: standard p-value criteria are potentially too strict and identifying the range of possible effect sizes is a viable goal in early phases. Particularly in fields with a low prevalence of true hypotheses, this will prevent discarding promising treatments too early (Albers and Lakens, 2018; Lakens, 2014). More stringent criteria should be applied in later experiments. Standard p-values will then identify true effects in high powered replications, which will reduce Type I errors and increase the predictive value of a set of experiments.

As we move along the research trajectory it also becomes helpful to think in terms of three types of validity: internal validity, external validity, and translation validity (Figure 1). To illustrate this, we return to the example of a disease caused by a metabolic product. An experiment in which the inhibition of a metabolic pathway leads to an improvement in animal health is internally valid if the measured effect (improved health) is caused by the experimental manipulation (administration of the inhibitor). Such an experiment is also externally valid if the effect is observed in other animal models and/or can be replicated in other labs. And if the effect is also observed in humans it is translationally valid. As we shall discuss, different strategies are needed to improve these three types of validity.

To optimize evidence from experiments on animals it is necessary to balance the different types of validity and reliability of experiments and replications. Early on, internal validity needs to be established with high priority. As knowledge about the animal model and disease mechanisms increases, external validity needs to be strengthened through within-lab replications and, for core results, to multiple centres. Systematic heterogeneity (additional strains, similar animal models, different sexes) will further strengthen external validity, and such heterogeneity should be introduced at the early stages of the work if this is feasible.

Replications at such later stages should also include secondary outcomes that directly link to clinically relevant parameters. However, even in high-validity experiments, reliability can be low when the number of experimental units is not sufficient to detect existing effects. For replications at this stage, reliability should be increased by increasing sample sizes or refining measurement procedures. As true preclinical effect sizes are frequently small and associated with considerable variance between experimental units, increased numbers of experimental units are needed to obtain reliable results (Bonapersona et al., 2020; Carneiro et al., 2018). According to the 3R principles (Russell and Burch, 1959), the number of animals tested needs to reflect the current stage in the preclinical trajectory (Sneddon et al., 2017; Strech and Dirnagl, 2019).

There is, however, no consensus yet on how to balance ethical and statistical power considerations in replications in animal experiments. Standard approaches where power calculations are based on the point estimate of the initial study will often yield too small animal numbers (Albers and Lakens, 2018; Piper et al., 2019). This potentially inflates false negatives, running the risk of missing important effects and wrongfully failing a replication. Alternatives like safeguard power analysis (Perugini et al., 2014), sceptical p-value (Held, 2020), or adjusting for uncertainty (Anderson and Maxwell, 2017) have been proposed mainly for psychological experiments with human subjects. These approaches will often yield high enough sample sizes to ensure sufficient power for replications. Due to ethical and resource constraints, preclinical replications are seldom able to test such high numbers, which may be one reason why such approaches have not yet been implemented widely in preclinical research design. Here, we see clear room for improvement and research opportunities.

Regarding the number of experimental units, the RPCB aimed at achieving at least 80% statistical power, based on the effect size measured in the original study. However, the 'winner's curse' means that published effect sizes (p<0.05) tend to be larger due to random sample variability. So basing the design of a replication on the effect size reported in the original study could result in the replication being underpowered (Colquhoun, 2014).

The BRI team calculated sample sizes to achieve a statistical power of 95% to detect the original effect in each of the three replications (which will be conducted in different labs). Through this, the BRI team tried to compensate for a possible inflation of the original results due to publication bias and winner’s curse. Furthermore, they planned to include additional positive and/or negative controls to ensure interpretation of the outcomes (Neves and Amaral, 2020). The different approaches taken by RPCB and BRI also confirm that there is, as yet, no consensus on how to calculate animal numbers towards an ethical conduct of replications.

The goal of a preclinical research trajectory is to enable the decision to engage in clinical studies. In a simplified scheme, decision options include to discontinue experiments because of futility (hypothesis apparently not true or effects not biologically significant), to gather more evidence to resolve ambiguity (with increased validity and reliability), or to engage in a clinical study (when enough evidence is collected). Most studies in preclinical research, however, are targeted at initial findings that are often exploratory (Howells et al., 2014). Systematic replication efforts that are decision-enabling are rare in academic preclinical research and have only recently begun to be conducted (Kimmelman et al., 2014). For the decision to finally engage in a clinical trial, a systematic review of all experiments is needed. In such a review, evidence should be judged on the validity and reliability criteria discussed here. Ideally, this will form the basis for informative investigator brochures that are currently lacking such decision enabling information (Wieschowski et al., 2018).

Our proposed framework is of course not applicable in all cases. If prior knowledge about a mechanism is already available and models are established, internal validity may already be high from the onset. Moreover, our simplified proposal may not generalise across all fields, or to academic and industry settings alike. Nonetheless, replications (and ideally every preclinical experiment) need to be framed in terms of validity and reliability. This is even more pressing as replications constitute an important foundation for successful translation. To enable the establishment of preclinical research trajectories, we see a need for action for funders and researchers.

The surprising lack of systematic replication in preclinical research also stems from lack of funding opportunities. Contrary to clinical trials, across-lab multi-centre replications are rare in preclinical research. Funders should thus design specific calls aimed at replications in the broad sense described here. Funding schemes should take the structure of preclinical research trajectories into account and may specifically be tailored towards the different experimental stages.

For replications, researchers need to specify how validity is improved by the replication compared to an initial study. Detailing how internal, external and translational validity increase in replications will emphasise the new evidence that is generated beyond the initial study and provide an ethical justification for the replication. Sample-size calculations should consider how reliability and validity are balanced against each other and define clear criteria for decisions to advance to the next stage in the trajectory. This will require a deviation from standard one-size-fits-all sample-size calculations. Researchers need guidance on how to adjust sample-size calculations and decision criteria, starting from an initial exploratory study that will serve as a proof of concept and gradually moving towards decision-enabling studies that will define whether a clinical trial is warranted.

The scientific endeavour is not limited to a single study and simple null-hypothesis testing. Even though the prevailing statistical test framework may suggest this, researchers are operating in a larger framework where evidence is accumulated over several levels with several competing alternative hypotheses (Platt, 1964). Replications are an important building block, where research priorities are transparently updated according to current knowledge. This includes proper reporting at all stages and registration of research at critical stages to avoid biases (Strech and Dirnagl, 2019). Currently, the literature on preclinical research and associated decisions is scarce (see, for example, van der Staay et al., 2010). Research on successful – and also on less successful – research lines will inform about best practices and yield important insights how biases potentially distort evidence collection (Kiwanuka et al., 2018).

In conclusion, systematically improving scientific validity in replications will improve trustworthiness and usefulness of preclinical studies and thus allow for a responsible conduct of animal experiments.

No data was generated.

1. 1. Albers C
   2. Lakens D

   (2018) When power analyses based on pilot data are biased: inaccurate effect size estimators and follow-up bias

   Journal of Experimental Social Psychology 74:187–195.

   https://doi.org/10.1016/j.jesp.2017.09.004

   • Google Scholar
2. 1. Amaral OB
   2. Neves K
   3. Wasilewska-Sampaio AP
   4. Carneiro CF

   (2019) The Brazilian Reproducibility Initiative

   eLife 8:e41602.

   https://doi.org/10.7554/eLife.41602

   • PubMed
   • Google Scholar
3. 1. Anderson SF
   2. Maxwell SE

   (2017) Addressing the "Replication Crisis": Using original studies to design replication studies with appropriate statistical power

   Multivariate Behavioral Research 52:305–324.

   https://doi.org/10.1080/00273171.2017.1289361

   • PubMed
   • Google Scholar
4. 1. Bailoo JD
   2. Reichlin TS
   3. Würbel H

   (2014) Refinement of experimental design and conduct in laboratory animal research

   ILAR Journal 55:383–391.

   https://doi.org/10.1093/ilar/ilu037

   • PubMed
   • Google Scholar
5. Website

   1. BMBF-DLR

   (2018) Confirmatory preclinical studies (Förderung von konfirmatorischen präklinischen studien)

   German Federal Ministry of Education and Research. Accessed January 14, 2021.

   https://www.gesundheitsforschung-bmbf.de/de/8344.php
6. Preprint

   1. Bonapersona V
   2. Hoijtink H
   3. Relacs S

   (2019) RePAIR: a power solution to animal experimentation

   bioRxiv.

   https://doi.org/10.1101/864652

   • Google Scholar
7. 1. Bonapersona V
   2. Hoijtink H
   3. Joëls M
   4. Sarabdjitsingh RA

   (2020) P.201 reduction by prior animal informed research (RePAIR): a power solution to animal experimentation

   European Neuropsychopharmacology 31:S19–S20.

   https://doi.org/10.1016/j.euroneuro.2019.12.027

   • Google Scholar
8. 1. Camerer CF
   2. Dreber A
   3. Forsell E
   4. Ho TH
   5. Huber J
   6. Johannesson M
   7. Kirchler M
   8. Almenberg J
   9. Altmejd A
   10. Chan T
   11. Heikensten E
   12. Holzmeister F
   13. Imai T
   14. Isaksson S
   15. Nave G
   16. Pfeiffer T
   17. Razen M
   18. Wu H

   (2016) Evaluating replicability of laboratory experiments in economics

   Science 351:1433–1436.

   https://doi.org/10.1126/science.aaf0918

   • PubMed
   • Google Scholar
9. 1. Camerer CF
   2. Dreber A
   3. Holzmeister F
   4. Ho TH
   5. Huber J
   6. Johannesson M
   7. Kirchler M
   8. Nave G
   9. Nosek BA
   10. Pfeiffer T
   11. Altmejd A
   12. Buttrick N
   13. Chan T
   14. Chen Y
   15. Forsell E
   16. Gampa A
   17. Heikensten E
   18. Hummer L
   19. Imai T
   20. Isaksson S
   21. Manfredi D
   22. Rose J
   23. Wagenmakers EJ
   24. Wu H

   (2018) Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

   Nature Human Behaviour 2:637–644.

   https://doi.org/10.1038/s41562-018-0399-z

   • PubMed
   • Google Scholar
10. 1. Carneiro CFD
    2. Moulin TC
    3. Macleod MR
    4. Amaral OB

    (2018) Effect size and statistical power in the rodent fear conditioning literature - A systematic review

    PLOS ONE 13:e0196258.

    https://doi.org/10.1371/journal.pone.0196258

    • PubMed
    • Google Scholar
11. 1. Colquhoun D

    (2014) An investigation of the false discovery rate and the misinterpretation of p-values

    Royal Society Open Science 1:140216.

    https://doi.org/10.1098/rsos.140216

    • PubMed
    • Google Scholar
12. 1. Dechartres A
    2. Boutron I
    3. Trinquart L
    4. Charles P
    5. Ravaud P

    (2011) Single-center trials show larger treatment effects than multicenter trials: evidence from a meta-epidemiologic study

    Annals of Internal Medicine 155:39–51.

    https://doi.org/10.7326/0003-4819-155-1-201107050-00006

    • PubMed
    • Google Scholar
13. 1. Denayer T
    2. Stöhr T
    3. Van Roy M

    (2014) Animal models in translational medicine: validation and prediction

    New Horizons in Translational Medicine 2:5–11.

    https://doi.org/10.1016/j.nhtm.2014.08.001

    • Google Scholar
14. Book

    1. Dirnagl U

    (2020) Resolving the tension between exploration and confirmation in preclinical biomedical research

    In: Bespalov A, Michel M. C, Steckler T, editors. Good Research Practice in Non-Clinical Pharmacology and Biomedicine. Springer International Publishing. pp. 71–79.

    https://doi.org/10.1007/978-3-030-33656-1

    • Google Scholar
15. 1. eLife

    (2017) The challenges of replication

    eLife 6:e23693.

    https://doi.org/10.7554/eLife.23693

    • PubMed
    • Google Scholar
16. 1. Errington TM
    2. Iorns E
    3. Gunn W
    4. Tan FE
    5. Lomax J
    6. Nosek BA

    (2014) An open investigation of the reproducibility of cancer biology research

    eLife 3:e04333.

    https://doi.org/10.7554/eLife.04333

    • Google Scholar
17. Book

    1. Friedman LM
    2. Furberg CD
    3. DeMets DL
    4. Reboussin DM
    5. Granger CB

    (2015) Multicenter Trials

    In: Friedman L. M, Furberg C. D, DeMets D. L, Reboussin D. M, Granger C. B, editors. Fundamentals of Clinical Trials. Springer. pp. 501–518.

    https://doi.org/10.1007/978-1-4419-1586-3

    • Google Scholar
18. 1. Glasgow RE
    2. Green LW
    3. Klesges LM
    4. Abrams DB
    5. Fisher EB
    6. Goldstein MG
    7. Hayman LL
    8. Ockene JK
    9. Orleans CT

    (2006) External validity: we need to do more

    Annals of Behavioral Medicine 31:105–108.

    https://doi.org/10.1207/s15324796abm3102_1

    • PubMed
    • Google Scholar
19. 1. Guerin MV
    2. Finisguerra V
    3. Van den Eynde BJ
    4. Bercovici N
    5. Trautmann A

    (2020) Preclinical murine tumor models: a structural and functional perspective

    eLife 9:e50740.

    https://doi.org/10.7554/eLife.50740

    • PubMed
    • Google Scholar
20. 1. Held L

    (2020) A new standard for the analysis and design of replication studies

    Journal of the Royal Statistical Society: Series A 183:431–448.

    https://doi.org/10.1111/rssa.12493

    • Google Scholar
21. 1. Howells DW
    2. Sena ES
    3. Macleod MR

    (2014) Bringing rigour to translational medicine

    Nature Reviews Neurology 10:37–43.

    https://doi.org/10.1038/nrneurol.2013.232

    • PubMed
    • Google Scholar
22. Preprint

    1. Hunniford VT
    2. Grudniewicz A
    3. Fergusson DA
    4. Grigor E
    5. Lansdell C
    6. Lalu MM

    (2019) Multicenter preclinical studies as an innovative method to enhance translation: a systematic review of published studies

    bioRxiv.

    https://doi.org/10.1101/591289

    • Google Scholar
23. 1. Kafkafi N
    2. Agassi J
    3. Chesler EJ
    4. Crabbe JC
    5. Crusio WE
    6. Eilam D
    7. Gerlai R
    8. Golani I
    9. Gomez-Marin A
    10. Heller R
    11. Iraqi F
    12. Jaljuli I
    13. Karp NA
    14. Morgan H
    15. Nicholson G
    16. Pfaff DW
    17. Richter SH
    18. Stark PB
    19. Stiedl O
    20. Stodden V
    21. Tarantino LM
    22. Tucci V
    23. Valdar W
    24. Williams RW
    25. Würbel H
    26. Benjamini Y

    (2018) Reproducibility and replicability of rodent phenotyping in preclinical studies

    Neuroscience & Biobehavioral Reviews 87:218–232.

    https://doi.org/10.1016/j.neubiorev.2018.01.003

    • PubMed
    • Google Scholar
24. 1. Kimmelman J
    2. Mogil JS
    3. Dirnagl U

    (2014) Distinguishing between exploratory and confirmatory preclinical research will improve translation

    PLOS Biology 12:e1001863.

    https://doi.org/10.1371/journal.pbio.1001863

    • Google Scholar
25. 1. Kimmelman J
    2. London AJ

    (2011) Predicting harms and benefits in translational trials: ethics, evidence, and uncertainty

    PLOS Medicine 8:e1001010.

    https://doi.org/10.1371/journal.pmed.1001010

    • PubMed
    • Google Scholar
26. 1. Kiwanuka O
    2. Bellander BM
    3. Hånell A

    (2018) The case for introducing pre-registered confirmatory pharmacological pre-clinical studies

    Journal of Cerebral Blood Flow & Metabolism 38:749–754.

    https://doi.org/10.1177/0271678X18760109

    • PubMed
    • Google Scholar
27. 1. Lakens D

    (2014) Performing high-powered studies efficiently with sequential analyses

    European Journal of Social Psychology 44:701–710.

    https://doi.org/10.1002/ejsp.2023

    • Google Scholar
28. 1. Llovera G
    2. Hofmann K
    3. Roth S
    4. Salas-Pérdomo A
    5. Ferrer-Ferrer M
    6. Perego C
    7. Zanier ER
    8. Mamrak U
    9. Rex A
    10. Party H
    11. Agin V
    12. Fauchon C
    13. Orset C
    14. Haelewyn B
    15. De Simoni MG
    16. Dirnagl U
    17. Grittner U
    18. Planas AM
    19. Plesnila N
    20. Vivien D
    21. Liesz A

    (2015) Results of a preclinical randomized controlled multicenter trial (pRCT): Anti-CD49d treatment for acute brain ischemia

    Science Translational Medicine 7:299ra121.

    https://doi.org/10.1126/scitranslmed.aaa9853

    • PubMed
    • Google Scholar
29. 1. Macleod MR
    2. Michie S
    3. Roberts I
    4. Dirnagl U
    5. Chalmers I
    6. Ioannidis JPA
    7. Salman RA-S
    8. Chan A-W
    9. Glasziou P

    (2014) Biomedical research: Increasing value, reducing waste

    The Lancet 383:101–104.

    https://doi.org/10.1016/S0140-6736(13)62329-6

    • Google Scholar
30. 1. Maysami S
    2. Wong R
    3. Pradillo JM
    4. Denes A
    5. Dhungana H
    6. Malm T
    7. Koistinaho J
    8. Orset C
    9. Rahman M
    10. Rubio M
    11. Schwaninger M
    12. Vivien D
    13. Bath PM
    14. Rothwell NJ
    15. Allan SM

    (2016) A cross-laboratory preclinical study on the effectiveness of interleukin-1 receptor antagonist in stroke

    Journal of Cerebral Blood Flow & Metabolism 36:596–605.

    https://doi.org/10.1177/0271678X15606714

    • PubMed
    • Google Scholar
31. 1. Metselaar JM
    2. Lammers T

    (2020) Challenges in nanomedicine clinical translation

    Drug Delivery and Translational Research 10:721–725.

    https://doi.org/10.1007/s13346-020-00740-5

    • PubMed
    • Google Scholar
32. 1. Morrison SJ

    (2014) Time to do something about reproducibility

    eLife 3:e03981.

    https://doi.org/10.7554/eLife.03981

    • Google Scholar
33. 1. Motulsky HJ

    (2014) Common misconceptions about data analysis and statistics

    Journal of Pharmacology and Experimental Therapeutics 351:200–205.

    https://doi.org/10.1124/jpet.114.219170

    • Google Scholar
34. 1. Mullane K
    2. Williams M

    (2019) Preclinical models of Alzheimer's disease: Relevance and translational validity

    Current Protocols in Pharmacology 84:e57.

    https://doi.org/10.1002/cpph.57

    • PubMed
    • Google Scholar
35. Preprint

    1. Neves K
    2. Carneiro CFD
    3. Wasilewska-Sampaio AP
    4. Abreu M
    5. Gomes BV
    6. Tan PB
    7. Amaral OB

    (2020) Two years into the Brazilian Reproducibility Initiative: Reflections on conducting a large-scale replication of Brazilian biomedical science

    MetaArXiv.

    https://doi.org/10.31222/osf.io/njs6k

    • Google Scholar
36. 1. Neves K
    2. Amaral OB

    (2020) Addressing selective reporting of experiments through predefined exclusion criteria

    eLife 9:e56626.

    https://doi.org/10.7554/eLife.56626

    • PubMed
    • Google Scholar
37. 1. Nosek BA
    2. Errington TM

    (2020a) What is replication?

    PLOS Biology 18:e3000691.

    https://doi.org/10.1371/journal.pbio.3000691

    • PubMed
    • Google Scholar
38. 1. Nosek BA
    2. Errington TM

    (2020b) The best time to argue about what a replication means? Before you do it

    Nature 583:518–520.

    https://doi.org/10.1038/d41586-020-02142-6

    • PubMed
    • Google Scholar
39. 1. Open Science Collaboration

    (2015) Estimating the reproducibility of psychological science

    Science 349:aac4716.

    https://doi.org/10.1126/science.aac4716

    • PubMed
    • Google Scholar
40. 1. Patil P
    2. Peng RD
    3. Leek JT

    (2016) What should researchers expect when they replicate studies? A statistical view of replicability in psychological science

    Perspectives on Psychological Science 11:539–544.

    https://doi.org/10.1177/1745691616646366

    • PubMed
    • Google Scholar
41. 1. Percie du Sert N
    2. Hurst V
    3. Ahluwalia A
    4. Alam S
    5. Avey MT
    6. Baker M
    7. Browne WJ
    8. Clark A
    9. Cuthill IC
    10. Dirnagl U
    11. Emerson M
    12. Garner P
    13. Holgate ST
    14. Howells DW
    15. Karp NA
    16. Lazic SE
    17. Lidster K
    18. MacCallum CJ
    19. Macleod M
    20. Pearl EJ
    21. Petersen OH
    22. Rawle F
    23. Reynolds P
    24. Rooney K
    25. Sena ES
    26. Silberberg SD
    27. Steckler T
    28. Würbel H

    (2020) The ARRIVE guidelines 2.0: updated guidelines for reporting animal research

    PLOS Biology 18:e3000410.

    https://doi.org/10.1371/journal.pbio.3000410

    • PubMed
    • Google Scholar
42. 1. Perugini M
    2. Gallucci M
    3. Costantini G

    (2014) Safeguard power as a protection against imprecise power estimates

    Perspectives on Psychological Science 9:319–332.

    https://doi.org/10.1177/1745691614528519

    • PubMed
    • Google Scholar
43. 1. Piper SK
    2. Grittner U
    3. Rex A
    4. Riedel N
    5. Fischer F
    6. Nadon R
    7. Siegerink B
    8. Dirnagl U

    (2019) Exact replication: foundation of science or game of chance?

    PLOS Biology 17:e3000188.

    https://doi.org/10.1371/journal.pbio.3000188

    • PubMed
    • Google Scholar
44. 1. Platt JR

    (1964) Strong inference: certain systematic methods of scientific thinking may produce much more rapid progress than others

    Science 146:347–353.

    https://doi.org/10.1126/science.146.3642.347

    • PubMed
    • Google Scholar
45. 1. Pound P
    2. Ritskes-Hoitinga M

    (2018) Is it possible to overcome issues of external validity in preclinical animal research? Why most animal models are bound to fail

    Journal of Translational Medicine 16:304.

    https://doi.org/10.1186/s12967-018-1678-1

    • PubMed
    • Google Scholar
46. 1. Prohaska T
    2. Etkin C

    (2010)

    External validity and translation from research to implementation

    Generations 34:59–65.

    • Google Scholar
47. 1. Richter SH
    2. Garner JP
    3. Auer C
    4. Kunert J
    5. Würbel H

    (2010) Systematic variation improves reproducibility of animal experiments

    Nature Methods 7:167–168.

    https://doi.org/10.1038/nmeth0310-167

    • PubMed
    • Google Scholar
48. Report

    1. Russell WMS
    2. Burch RL

    (1959)

    The Principles of Humane Experimental Technique

    Methuen.

    • Google Scholar
49. 1. Salvadori M
    2. Cesari N
    3. Murgia A
    4. Puccini P
    5. Riccardi B
    6. Dominici M

    (2019) Dissecting the pharmacodynamics and pharmacokinetics of MSCs to overcome limitations in their clinical translation

    Molecular Therapy - Methods & Clinical Development 14:1–15.

    https://doi.org/10.1016/j.omtm.2019.05.004

    • PubMed
    • Google Scholar
50. 1. Sasaguri H
    2. Nilsson P
    3. Hashimoto S
    4. Nagata K
    5. Saito T
    6. De Strooper B
    7. Hardy J
    8. Vassar R
    9. Winblad B
    10. Saido TC

    (2017) APP mouse models for Alzheimer’s disease preclinical studies

    The EMBO Journal 36:2473–2487.

    https://doi.org/10.15252/embj.201797397

    • PubMed
    • Google Scholar
51. 1. Sena ES
    2. van der Worp HB
    3. Bath PM
    4. Howells DW
    5. Macleod MR

    (2010) Publication bias in reports of animal stroke studies leads to major overstatement of efficacy

    PLOS Biology 8:e1000344.

    https://doi.org/10.1371/journal.pbio.1000344

    • PubMed
    • Google Scholar
52. 1. Sena ES
    2. Currie GL
    3. McCann SK
    4. Macleod MR
    5. Howells DW

    (2014) Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically

    Journal of Cerebral Blood Flow & Metabolism 34:737–742.

    https://doi.org/10.1038/jcbfm.2014.28

    • PubMed
    • Google Scholar
53. 1. Sneddon LU
    2. Halsey LG
    3. Bury NR

    (2017) Considering aspects of the 3Rs principles within experimental animal biology

    The Journal of Experimental Biology 220:3007–3016.

    https://doi.org/10.1242/jeb.147058

    • Google Scholar
54. 1. Strech D
    2. Dirnagl U

    (2019) 3Rs missing: animal research without scientific value is unethical

    BMJ Open Science 3:bmjos-2018-000048.

    https://doi.org/10.1136/bmjos-2018-000048

    • Google Scholar
55. 1. Tuntland T
    2. Ethell B
    3. Kosaka T
    4. Blasco F
    5. Zang RX
    6. Jain M
    7. Gould T
    8. Hoffmaster K

    (2014) Implementation of pharmacokinetic and pharmacodynamic strategies in early research phases of drug discovery and development at Novartis Institute of Biomedical Research

    Frontiers in Pharmacology 5:174.

    https://doi.org/10.3389/fphar.2014.00174

    • PubMed
    • Google Scholar
56. 1. Turner PV
    2. Barbee RW

    (2019) Responsible science and research animal use

    ILAR Journal 60:1–4.

    https://doi.org/10.1093/ilar/ilz020

    • PubMed
    • Google Scholar
57. 1. van der Staay FJ
    2. Arndt SS
    3. Nordquist RE

    (2010) The standardization–generalization dilemma: A way out

    Genes, Brain, and Behavior 9:849–855.

    https://doi.org/10.1111/j.1601-183X.2010.00628.x

    • PubMed
    • Google Scholar
58. 1. van der Worp HB
    2. Howells DW
    3. Sena ES
    4. Porritt MJ
    5. Rewell S
    6. O'Collins V
    7. Macleod MR

    (2010) Can animal models of disease reliably inform human studies?

    PLOS Medicine 7:e1000245.

    https://doi.org/10.1371/journal.pmed.1000245

    • PubMed
    • Google Scholar
59. 1. Voelkl B
    2. Vogt L
    3. Sena ES
    4. Würbel H

    (2018) Reproducibility of preclinical animal research improves with heterogeneity of study samples

    PLOS Biology 16:e2003693.

    https://doi.org/10.1371/journal.pbio.2003693

    • Google Scholar
60. 1. Voelkl B
    2. Altman NS
    3. Forsman A
    4. Forstmeier W
    5. Gurevitch J
    6. Jaric I
    7. Karp NA
    8. Kas MJ
    9. Schielzeth H
    10. Van de Casteele T
    11. Würbel H

    (2020) Reproducibility of animal research in light of biological variation

    Nature Reviews Neuroscience 21:384–393.

    https://doi.org/10.1038/s41583-020-0313-3

    • PubMed
    • Google Scholar
61. 1. Volk HD
    2. Stevens MM
    3. Mooney DJ
    4. Grainger DW
    5. Duda GN

    (2015) Key elements for nourishing the translational research environment

    Science Translational Medicine 7:282cm2.

    https://doi.org/10.1126/scitranslmed.aaa2049

    • PubMed
    • Google Scholar
62. 1. Vollert J
    2. Schenker E
    3. Macleod M
    4. Bespalov A
    5. Wuerbel H
    6. Michel M
    7. Dirnagl U
    8. Potschka H
    9. Waldron A-M
    10. Wever K
    11. Steckler T
    12. van de Casteele T
    13. Altevogt B
    14. Sil A
    15. Rice ASC

    (2020) Systematic review of guidelines for internal validity in the design, conduct and analysis of preclinical biomedical experiments involving laboratory animals

    BMJ Open Science 4:e100046.

    https://doi.org/10.1136/bmjos-2019-100046

    • Google Scholar
63. 1. Wieschowski S
    2. Chin WWL
    3. Federico C
    4. Sievers S
    5. Kimmelman J
    6. Strech D

    (2018) Preclinical efficacy studies in investigator brochures: do they enable risk-benefit assessment?

    PLOS Biology 16:e2004879.

    https://doi.org/10.1371/journal.pbio.2004879

    • PubMed
    • Google Scholar
64. 1. Würbel H

    (2017) More than 3Rs: the importance of scientific validity for harm-benefit analysis of animal research

    Lab Animal 46:164–166.

    https://doi.org/10.1038/laban.1220

    • PubMed
    • Google Scholar

Thank you for submitting your article "Improving the predictiveness and ethics of preclinical studies through replications" to eLife for consideration as a Feature Article. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by the eLife Features Editor (Peter Rodgers). The following individuals involved in review of your submission have agreed to reveal their identity: Catherine Winchester (Reviewer #1); Hanno Wuerbel (Reviewer #2).

The reviewers and editors have discussed the reviews and we have drafted this decision letter to help you prepare a revised submission.

Summary

This paper presents an informal framework to guide preclinical research programmes, with the aim to systematically and gradually improve the reliability and validity of the research findings. According to the authors, such a procedure would not only increase chances for translational success but also lead to a more efficient allocation of resources and ethical use of animals. While the overall message is clear – most experiments are part of larger research programmes, and experimental designs need to be adjusted to the specific questions asked at any stage of a research programme – the authors need to be more precise in their definition/use of terms like replication, and to provide more details on how their proposals would work in practice.

Essential revisions

1) Definition of replication

The definition of "replication" used in the article seems almost redundant with the term "experiment". For example, the authors state: "Here, we define replications as experiments that aim to generate evidence that supports a previously made inferential claim (Nosek and Errington, 2020a)".

First, I am not sure this is equivalent to definition given in Nosek and Errington, 2020a: "Replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research". This should be clarified.

However, even if so, I am still not convinced it would be a useful definition of "replication". Nosek and Errington conclude: "Theories make predictions; replications test those predictions. Outcomes from replications are fodder for refining, altering, or extending theory to generate new predictions. Replication is a central part of the iterative maturing cycle of description, prediction, and explanation."

This is in fact a concise description of the "scientific method". Thus, Wikipedia (sorry about that….) describes the "scientific method" as: "It [the scientific method] involves formulating hypotheses, via induction, based on such observations; experimental and measurement-based testing of deductions drawn from the hypotheses; and refinement (or elimination) of the hypotheses based on the experimental findings."

I don't think it makes sense to replace well established terms such as "scientific method" or "experiment" by "replication", which for most scientists has a much more specific meaning. The authors should clarify this point and clearly state how they distinguish their term "replication" from the term "experiment" (every experiment is a test of a claim from prior research and therefore fits Nosek and Errington's definition of replication) and how they distinguish their framework from the "scientific method" in general.

This becomes even more explicit in the following paragraph, where they state: "Besides within and between laboratory replications, experiments in such series include initial exploratory studies, toxicity studies, positive and negative controls, pharmacodynamics and kinetics that all aim to generate evidence to support an inferential claim and refute possible alternatives". Thus, according to the authors, all of these experiments (toxicity studies, pharmacodynamics, etc.) are nothing but replications – this is stretching the term replication extremely far, and I am not convinced that readers will follow them. And it would not be necessary; they could present the exact same framework by using conventional terms.

Related to this, please explain the difference/relationship between replication and reproducibility.

2) The section "What to replicate?"

The description of the three projects in the section is too long, and needs to be better integrated into the article.

Note from the Editor: since this point is largely an editorial matter, dealing with it can be deferred until later.

3) The section "How to replicate?"

The stepwise procedure presented in the text and in Figure 1 is not well developed. The trade-offs between the different dimensions of validity are not presented in any detail, and the ethical and scientific implications of different decisions at different stages in a research programme are not discussed. In particular:

– "At the start, reliability and validity are potentially low due to biased assessment of outcomes, unblinded conduct of experiments, undisclosed researcher degrees of freedom, low sample sizes, etc.". This reads like a recommendation: start your research line with quick and dirty studies and only refine methods once you have discovered something interesting. I am sure that this is not what the authors wanted to say but it is how it may come across. The fact that these problems are prevalent throughout past (and current) preclinical research does not justify them. Even at early, exploratory stages of a research programme, scientific rigour is essential for both scientific and ethical reasons. Therefore, I do not agree with the authors that internal validity may be low initially and should only be improved in the course of a research programme. Studies may be smaller (using smaller sample sizes) initially, but there is no excuse for ignoring any other measure against risks of bias (randomization, blinding, complete reporting of all measured outcome variables, etc.) as these are simply part of good research practice!

– This brings me to another point: I am not convinced that exploratory research should be conducted under highly restricted conditions. In fact, including variation of conditions (e.g. genotype, environment, measurement) right from the beginning may be much more productive a strategy for generating valuable hypotheses (see e.g. Voelkl et al., 2020). Furthermore, designing larger and more complex confirmatory or replication studies (covering a larger inference space) may reduce the need for additional studies to assess convergent and discriminant validity and external validity. Thus, there is a trade-off between more but simpler vs. fewer but more complex studies that is not at all represented in the framework presented here (and in Figure 1).

– "At this stage, criteria to decide whether to conduct additional experiments and replications should be lenient". Here it seems that the authors do make a difference between "experiments" and "replications", but as discussed above, it is unclear on what grounds.

– The definition of "translational validity" needs attention. They state that it covers "construct validity" and "predictive validity", but not "concurrent validity" and "content validity", which are the four common dimensions of test validity. Thus, it is unclear whether they actually mean "test validity" (but consider concurrent validity and content validity less important) or whether they consider "translational validity" to be different from "test validity". Importantly, however, "construct validity" is made up by "convergent validity" and "discriminant validity". This means that assessing "construct validity" requires tests of convergent and discriminant validity that cannot – according to common terminology – be considered as replications as they explicitly test and compare measures of different constructs that should or should not be related.

– "In studies with low internal validity that stand at the beginning of a new research line, e.g. in an exploratory phase, low numbers of animals may be acceptable." Again, I don't think low internal validity is ever acceptable, certainly not if animals are involved. The number of animals always needs to be justified by the specific question to be answered – that in exploratory research often low numbers of animals are used does not necessarily mean that this is acceptable.

– The recommendations in this section are rather vague. For example, the authors write: "For replications, researchers need to specify how validity is improved by the replication compared to the initial study." Yet, after this general demand, no further details or suggestions are given how this can be achieved. Similarly, they write "Researchers need guidance on how to adjust sample size calculations..." but no guidance is offered here nor are any hints given where to find such guidance (references might be useful) on what principles this guidance should be based etc.

– Please comment on how much evidence (how many replication studies/ different approaches exploring the same question) is needed? And who should collate this body of evidence – researchers, clinicians, funders, a committee?

Essential revisions

1) Definition of replication

The definition of "replication" used in the article seems almost redundant with the term "experiment". For example, the authors state: "Here, we define replications as experiments that aim to generate evidence that supports a previously made inferential claim (Nosek and Errington, 2020a)".

First, I am not sure this is equivalent to definition given in Nosek and Errington, 2020a: "Replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research". This should be clarified.

However, even if so, I am still not convinced it would be a useful definition of "replication". Nosek and Errington conclude: "Theories make predictions; replications test those predictions. Outcomes from replications are fodder for refining, altering, or extending theory to generate new predictions. Replication is a central part of the iterative maturing cycle of description, prediction, and explanation."

This is in fact a concise description of the "scientific method". Thus, Wikipedia (sorry about that….) describes the "scientific method" as: "It [the scientific method] involves formulating hypotheses, via induction, based on such observations; experimental and measurement-based testing of deductions drawn from the hypotheses; and refinement (or elimination) of the hypotheses based on the experimental findings."

I don't think it makes sense to replace well established terms such as "scientific method" or "experiment" by "replication", which for most scientists has a much more specific meaning. The authors should clarify this point and clearly state how they distinguish their term "replication" from the term "experiment" (every experiment is a test of a claim from prior research and therefore fits Nosek and Errington's definition of replication) and how they distinguish their framework from the "scientific method" in general.

This becomes even more explicit in the following paragraph, where they state: "Besides within and between laboratory replications, experiments in such series include initial exploratory studies, toxicity studies, positive and negative controls, pharmacodynamics and kinetics that all aim to generate evidence to support an inferential claim and refute possible alternatives". Thus, according to the authors, all of these experiments (toxicity studies, pharmacodynamics, etc.) are nothing but replications – this is stretching the term replication extremely far, and I am not convinced that readers will follow them. And it would not be necessary; they could present the exact same framework by using conventional terms.

Related to this, please explain the difference/relationship between replication and reproducibility.

We agree that our definition was not exactly matching the Nosek and Errington definition. We now clarify that replications are not solely about supporting evidence but works in both ways also to refute a claim. Starting from the admittedly very broad Nosek and Errington definition, we now specify that a replication is indeed based on a specific previous experiment. We thus rewrote the passage mentioned above to clarify this important issue. We now distinguish between experiments that test the same claim but with different design and approach and replications that are closely modelled after an initial experiment, but are changed to improve validity and reliability.

“Here, we define replications as experiments that are based on previous studies that aim to distinguish false from true claims (Nosek and Errington, 2020a). For this, previous experiments need to be reproducible with all methods and analytical pipelines unambiguously described. Reproducibility is thus a necessary prerequisite to engage in a contrastable replication (Patil et al., 2016; Plesser, 2018). Replications can deviate from previous experimental protocols by e.g. introducing different animal strains or changing environmental factors. Introducing such systematic heterogeneity between studies potentially strengthens generated evidence about inferential claims (Voelkl et al., 2020). “

With regard to the scientific method, we now emphasise that even tough preclinical research is of course based on a scientific method we now emphasise distinct features (measuring human conditions entirely in model systems) that sets it apart from others disciplines.

“With its goal to closely model human disease conditions, preclinical research differs from other scientific disciplines as there is a biological gap between experimentally studied models and patients as the ultimate beneficent. This affects the role of replications as they should not only confirm previous results but ideally also increase predictive power for the human case to enable successful translation.”

2) The section "What to replicate?"

The description of the three projects in the section is too long, and needs to be better integrated into the article.

Note from the Editor: since this point is largely an editorial matter, dealing with it can be deferred until later.

We make a suggestion how to shorten this paragraph. We reduced the number of words from 1027 to 807. We look forward to discussing this paragraph further if necessary.

3) The section "How to replicate?"

The stepwise procedure presented in the text and in Figure 1 is not well developed. The trade-offs between the different dimensions of validity are not presented in any detail, and the ethical and scientific implications of different decisions at different stages in a research programme are not discussed. In particular:

– "At the start, reliability and validity are potentially low due to biased assessment of outcomes, unblinded conduct of experiments, undisclosed researcher degrees of freedom, low sample sizes, etc.". This reads like a recommendation: start your research line with quick and dirty studies and only refine methods once you have discovered something interesting. I am sure that this is not what the authors wanted to say but it is how it may come across. The fact that these problems are prevalent throughout past (and current) preclinical research does not justify them. Even at early, exploratory stages of a research programme, scientific rigour is essential for both scientific and ethical reasons. Therefore, I do not agree with the authors that internal validity may be low initially and should only be improved in the course of a research programme. Studies may be smaller (using smaller sample sizes) initially, but there is no excuse for ignoring any other measure against risks of bias (randomization, blinding, complete reporting of all measured outcome variables, etc.) as these are simply part of good research practice!

We fully agree with the reviewers here and admit that we introduced some ambiguity here that we now clarify. In particular, we stress that internal validity should be already high in early stages. We also adjusted Figure 1 to indicate that internal validity should be already considered at early stages.

“At the start, researchers usually start off with an explorative study. As not all details and confounders in such a study can be known upfront reliability and validity are potentially not fully optimized. However, even at these early stages researchers should implement e.g. strategies to mitigate risks of bias (Figure 1).”

– This brings me to another point: I am not convinced that exploratory research should be conducted under highly restricted conditions. In fact, including variation of conditions (e.g. genotype, environment, measurement) right from the beginning may be much more productive a strategy for generating valuable hypotheses (see e.g. Voelkl et al., 2020). Furthermore, designing larger and more complex confirmatory or replication studies (covering a larger inference space) may reduce the need for additional studies to assess convergent and discriminant validity and external validity. Thus, there is a trade-off between more but simpler vs. fewer but more complex studies that is not at all represented in the framework presented here (and in Figure 1).

We agree with the reviewers that the standardisation fallacy is an important topic, particularly to be considered in preclinical research. As our focus is on the role of replications, we now extend our description on within and between laboratory replications to consider heterogenization. We emphasise this at various locations now. We further point readers to the mentioned paper by Voelkl et al., 2020 as a reference for a more general discussion. At this stage, we feel that introducing also a discussion on how to start a preclinical research process and negotiate simpler vs more complex studies would stray too far from our core message. To accommodate this important thought nonetheless we now state:

“Systematic heterogeneity (additional strains, similar animal models, different sex) will further strengthen external validity. If feasible heterogenization should be introduced at early stages already.”

– "At this stage, criteria to decide whether to conduct additional experiments and replications should be lenient". Here it seems that the authors do make a difference between "experiments" and "replications", but as discussed above, it is unclear on what grounds.

We adjusted the Introduction (see above) to make the distinction between a replication and additional experiments in preclinical research clear.

– The definition of "translational validity" needs attention. They state that it covers "construct validity" and "predictive validity", but not "concurrent validity" and "content validity", which are the four common dimensions of test validity. Thus, it is unclear whether they actually mean "test validity" (but consider concurrent validity and content validity less important) or whether they consider "translational validity" to be different from "test validity". Importantly, however, "construct validity" is made up by "convergent validity" and "discriminant validity". This means that assessing "construct validity" requires tests of convergent and discriminant validity that cannot – according to common terminology – be considered as replications as they explicitly test and compare measures of different constructs that should or should not be related.

We rewrote the paragraph on translational validity to address these points. Importantly, we removed all references to other types of validity than translational validity. The types of validity cited are derived mainly from psychological test theory. They have specific meanings in this context and it is (after lengthy discussion amongst the authors) not clear whether they relate to preclinical research in the same way they relate to psychological test theory. Even in psychology, construct validity is still a hotly debated topic where Meehl/Cronbach accounts clash with Messick for example. As this paper is addressing preclinical researchers who will most likely not be familiar with these terms and their theoretical framing, we removed the terms. We retained the original implications though. That is, translational validity is about the appropriateness of the disease model on a mechanistic level and how well it predicts the human condition. We also introduced as suggested by the reviewers converging and discriminant evidence without however resorting to calling this a type of validity. We clarify also that experiments generating converging and discriminant evidence are separate from replications.

“The term translational validity is used here as an umbrella term for factors that putatively contribute to translational success. It pertains to how well measurements and animal models represent a certain disease and its underlying pathomechanisms in humans. To assess this, complementary experiments evaluate the bounds of a model to discriminate it from other very similar diseases and collect converging evidence from different approaches. Often only parts of a disease are present in animal models. In models of neurodegenerative diseases like Alzheimer’s disease (AD) (Sasaguri et al., 2017), the focus on familial early onset genes in mouse models has potentially led to translational failures as the majority of human AD diagnoses are classified as sporadic late onset form (Mullane and Williams, 2019). Translational validity thus reflects whether measured parameters in animal models are diagnostic for human conditions and consequently, to what extent the observed outcomes will predict outcomes in humans (Denayer et al., 2014; Mullane and Williams, 2019).”

– "In studies with low internal validity that stand at the beginning of a new research line, e.g. in an exploratory phase, low numbers of animals may be acceptable." Again, I don't think low internal validity is ever acceptable, certainly not if animals are involved. The number of animals always needs to be justified by the specific question to be answered – that in exploratory research often low numbers of animals are used does not necessarily mean that this is acceptable.

– The recommendations in this section are rather vague. For example, the authors write: "For replications, researchers need to specify how validity is improved by the replication compared to the initial study." Yet, after this general demand, no further details or suggestions are given how this can be achieved. Similarly, they write "Researchers need guidance on how to adjust sample size calculations..." but no guidance is offered here nor are any hints given where to find such guidance (references might be useful) on what principles this guidance should be based etc.

Here, we want to point out that the number of animals is in our view not so much related to validity as to reliability. We now address this point in more detail (the reviewers note themselves above: “Studies may be smaller (using smaller sample sizes) initially.”).

“However, even in high-validity experiments, reliability can be low when the number of experimental units is not sufficient to detect existing effects. For replications at this stage, reliability should be increased by increasing sample sizes or refining measurement procedures. As true preclinical effect sizes are frequently small and associated with considerable variance between experimental units, increased numbers of experimental units are needed to obtain reliable results (Bonapersona et al., 2020; Carneiro et al., 2018). “

We further provide concrete examples how to increase internal validity, external validity and translational validity.

Guidance on sample size calculations in replications is scarce and a theme that we are currently researching ourselves. We nonetheless now point to recent approaches that were developed mainly for psychology and thus may not transfer easily.

“Under consideration of a reduction of animals according to the 3R principles (Russell and Burch, 1959), the number of animals tested needs to reflect the current stage in the preclinical trajectory (Sneddon et al., 2017; Strech and Dirnagl, 2019). For replications, this estimation involves consideration of effect sizes from previous studies. A power calculation based on the point estimate of the initial study will often yield too small animal numbers (Albers and Lakens, 2018; Piper et al., 2019). This potentially inflates false negatives, running the risk of missing important effects and wrongfully failing a replication. Alternatives like safeguard power analysis (Perugini et al., 2014), sceptical p-value (Held, 2020), or adjusting for uncertainty (Anderson and Maxwell, 2017) have been proposed mainly for psychological experiments with human subjects. Sample sizes in animal experiments are, however, much lower than estimated by the above methods due to ethical concerns and resource constraints. This may be one reason why such approaches have not yet been implemented widely in preclinical research.“

– Please comment on how much evidence (how many replication studies/ different approaches exploring the same question) is needed? And who should collate this body of evidence – researchers, clinicians, funders, a committee?

This is indeed an interesting and pressing question. Researchers from the emerging field of meta-analysis in preclinical research will be well suited to collate such evidence for example in systematic reviews. Such analyses could in turn form the basis for investigator brochures that are required for early clinical trials. We outline this as one scenario and admit that there may be different routes how to arrive at the conclusion that there is enough data to start a clinical trial.

“Systematic replication efforts that are decision-enabling are rare in academic preclinical research and have only recently begun to be conducted (Kimmelman et al., 2014). For the decision to finally engage in a clinical trial a systematic review of all experiments is needed. In such a review, evidence should be judged on the validity and reliability criteria discussed here. Ideally, this will form the basis for informative investigator brochures that are currently lacking such decision enabling information (Wieschowski et al., 2018).”

Author details

1. Natascha Ingrid Drude

   Natascha Ingrid Drude is in the Department of Experimental Neurology, Charité–Universitätsmedizin Berlin and the BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany

   Contribution

   Conceptualization, Writing - original draft, Writing - review and editing

   Contributed equally with

   Lorena Martinez Gamboa

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-7153-2894

2. Lorena Martinez Gamboa

   Lorena Martinez Gamboa is in the Department of Experimental Neurology, Charité–Universitätsmedizin Berlin and the BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany

   Contribution

   Conceptualization, Writing - original draft, Project administration, Writing - review and editing

   Contributed equally with

   Natascha Ingrid Drude

   Competing interests

   No competing interests declared

3. Meggie Danziger

   Meggie Danziger is in the Department of Experimental Neurology, Charité–Universitätsmedizin Berlin and the BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany

   Contribution

   Conceptualization, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-9224-1722

4. Ulrich Dirnagl

   Ulrich Dirnagl is in the Department of Experimental Neurology, Charité–Universitätsmedizin Berlin and the BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany

   Contribution

   Conceptualization, Funding acquisition, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0003-0755-6119

5. Ulf Toelch

   Ulf Toelch is in the BIH QUEST Center for Transforming Biomedical Research, Berlin Institute of Health, Berlin, Germany

   Contribution

   Conceptualization, Visualization, Writing - original draft, Writing - review and editing

   For correspondence

   ulf.toelch@bihealth.de

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-8731-3530

• 3,213

  views

• 311

  downloads

• 34

  citations

Views, downloads and citations are aggregated across all versions of this paper published by eLife.