Research Article

A systematic assessment of preclinical multilaboratory studies and a comparison to single laboratory studies

Victoria T Hunniford
Agnes Grudniewicz
Dean A Fergusson
Joshua Montroy
Emma Grigor
Casey Lansdell
Manoj M Lalu
On behalf of The Canadian Critical Care Translational Biology Group

Clinical Epidemiology Program, Blueprint Translational Research Group, Ottawa Hospital Research Institute, Canada
Telfer School of Management, University of Ottawa, Canada
Faculty of Medicine, University of Ottawa, Canada
Department of Surgery, University of Ottawa, Canada
School of Epidemiology and Public Health, University of Ottawa, Canada
Department of Anesthesiology and Pain Medicine, The Ottawa Hospital, University of Ottawa, Canada
Regenerative Medicine Program, The Ottawa Hospital Research Institute, Canada
Department of Cellular and Molecular Medicine, University of Ottawa, Canada

Mar 9, 2023

https://doi.org/10.7554/eLife.76300

Open access
Copyright information

Figures
Tables
Additional files

4 figures, 9 tables and 6 additional files

Figures

Figure 1

Download asset Open asset

Preferred reporting items for systematic reviews and meta-analysis (PRISMA) flow diagram for study selection.

Figure 2

Download asset Open asset

Difference in standardized mean difference between single lab and multilaboratory preclinical studies.

Figure 2—source data 1 Source data for comparision of single lab and multilaboratory studies.: https://cdn.elifesciences.org/articles/76300/elife-76300-fig2-data1-v2.xlsx
Download elife-76300-fig2-data1-v2.xlsx
Figure 2—source data 2 Standardized mean differences for all 14 singlevs multilaboratory comparisons and sub-groups by total quality score.: https://cdn.elifesciences.org/articles/76300/elife-76300-fig2-data2-v2.pdf
Download elife-76300-fig2-data2-v2.pdf

Figure 3

Download asset Open asset

Standardized mean differences of single lab and multilaboratory preclinical studies.

Author response image 1

Download asset Open asset

Tables

Table 1

Basic study characteristics of preclinical multilaboratory studies.

Author, Year	Center location	Journal	Funding	Centers PerformingIn Vivo Work	Non-Experimental Centers*	Animal, Sex	Sample size
Reimer et al., 1985	US	Circulation Research	Government (NHLBI)	3	1	Dog, both	51
Crabbe et al., 1999	Canada, US	Science	Government	3	0	Mouse, both	384
Alam et al., 2009	US	Journal of Trauma: Injury, Infection and Critical Care	Government (US army)	3	0	Swine, F	60
Spoerke et al., 2009	US	Archives of Surgery	Government (US army)	2	0	Swine, NA	32
Jones et al., 2015	US	Circulation Research	Government (NHLBI)	3	3	Mouse, M Rabbit, M Swine, F	47 23 26
Llovera et al., 2015	France, Germany, Italy, Spain	Science Translational Medicine	Government, academic and charitable	5	1	Mouse, M	315
Maysami et al., 2016	Finland, France, Germany, Hungary, UK, Spain	Journal of Cerebral Blood Flow & Metabolism	Government (FP7/ 2007–2013, INSERM), academic	5	1^†	Mouse, M	241
Bramlett et al., 2016	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	140
Browning et al., 2016	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	130
Dixon et al., 2016	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	135
Gill et al., 2016	US	Diabetes	Government, charitable	4	0	Mouse, F	NR
Mountney et al., 2016	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	128
Shear et al., 2016	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	142
Arroyo-Araujo et al., 2022	Netherlands, Switzerland, US	Scientific Reports	Government (FP7/2007–2013), industry (EFPIA), charitable	3	0	Rat, M	72
Jha et al., 2020	US	Journal of Neurotrauma	Government (US army)	3	1	Rat, M	111
Kliewer et al., 2020	Australia, Germany, UK	British Journal of Pharmacology	Government (NHMRC, NIH), NGO	3	0	Mouse, M	108

Legend: EFPIA - European Federation of Pharmaceutical Industries and Associations; FP7/2007-2013 – European Union Commission seventh Funding Program; INSERM - Institut national de la santé et de la recherche médicale; NGO – Non-government organization; NHLBI – National Heart, Lung, and Blood Institute; NHMRC – National Health and Medical Research Council; NIH – National Institutes of Health; NR - not reported; UK – United Kingdom; US – United States.
*

Non-experimental center: A site/lab not involved with the in vivo experiment (data processing, coordinating, biomarker, or pathology centers).
†

Center that was both an experimental center and a coordinating center.

Table 2

Study design characteristics of preclinical multilaboratory studies.

Author, Year	Disease model	Intervention	Study Outcomes	Secondary Outcomes	Reported Results	Recommendations for Future Research
Reimer et al., 1985	Myocardial infarction	Verapamil and ibuprofen	Infarct size	Mortality, hemodynamic measures, pathological/histological features, regional blood flow	Null	Not reported
Crabbe et al., 1999	Stimulant exposure	Cocaine	Locomotor activity		Mixed	Further preclinical testing
Alam et al., 2009	Polytrauma	Blood transfusion	Hemodynamic parameters	Mortality	Mixed across resuscitation products	Further preclinical testing
Spoerke et al., 2009	Polytrauma	Lyophilized plasma	Residual clotting activity	Mortality, hemodynamic measures, total blood loss, coagulation profiles, inflammatory measures	Positive	Further preclinical testing
Jones et al., 2015	Myocardial infarction	Ischemic preconditioning	Infarct size	Hemodynamic measures, regional blood flow, heart weight, troponin I, mean arterial pressure	Positive	Further preclinical testing
Llovera et al., 2015	Stroke	Anti-CD49d antibody	Infarct size	Functional outcome, invasion of leukocytes to brain	Mixed across models (positive, null)	First-in-human clinical trial
Maysami et al., 2016	Stroke	Interleukin-I receptor antagonist	Infarct size	Odema, functional outcome, mortality	Positive	Extensive clinical trial
Bramlett et al., 2016	Traumatic brain injury	Erythropoietin	Cognitive outcomes, biomarkers, motor outcomes, neuropathology		Null	No further preclinical study
Browning et al., 2016	Traumatic brain injury	Levetiracetam	Cognitive outcomes, biomarkers, motor outcomes, neuropathology		Positive	Further preclinical testing and first-in-human clinical trial
Dixon et al., 2016	Traumatic brain injury	Cyclosporine	Cognitive outcomes, biomarkers, motor outcomes, neuropathology		Null	No further preclinical testing
Gill et al., 2016	Diabetes	Combined anti-CD3 +IL-1 blockade	Blood glucose		Null	Pause clinical trial
Mountney et al., 2016	Traumatic brain injury	Simvastatin	Cognitive outcomes, biomarkers, motor outcomes, neuropathology		Null	No further preclinical study
Shear et al., 2016	Traumatic brain injury	Nicotinamide	Cognitive outcomes, biomarkers, motor outcomes, neuropathology		Null	No further preclinical study
Arroyo-Araujo et al., 2019	Autism spectrum disorder	mGluR1 antagonist (JNJ16259685)	Behavioural activity		Positive	Not reported
Jha et al., 2020	Traumatic brain injury	Glibenclamide	Cognitive outcomes, biomarkers, motor outcomes, neuropathology	Glucose level, drug levels	Mixed across models and outcomes (positive, null, and negative)	Further preclinical testing
Kliewer et al., 2020	Opioid-induced respiratory depression	Morphine	Respiratory rate	Constipation	Null	Not reported

Table 3

Risk of bias assessment of preclinical multilaboratory studies.

Study	Sequence generation ^§	Baseline characteristics	Allocation concealment^§	Random housing	Blinding of personnel	Random outcome assessment	Blinding of outcome assessment	Incomplete outcome data*, ^§	Selective outcome reporting	Other sources of bias ^¶ ^§
Reimer et al., 1985	U*	U	L	U	H^†	U	L	U	L	U
Crabbe et al., 1999	U*	L	U	L	U	U	U	L	L	H
Alam et al., 2009	U*	L	U	U	U	L	U	L	L	U
Spoerke et al., 2009	U*	L	U	U	U	L	U	L	L	U
Jones et al., 2015	L	L	U	U	L	L	L	L	L	L
Llovera et al., 2015	L	L	U	U	L	L	L	L	L	L
Maysami et al., 2016	H ^‡	H	L	U	L	L	L	L	H	H
Bramlett et al., 2016	U*	U	U	U	L	L	L	U	L	H
Browning et al., 2016	U*	U	U	U	L	L	L	U	L	H
Dixon et al., 2016	U*	U	U	U	L	L	L	U	L	H
Gill et al., 2016	H	L	U	U	U	L	L	U	U	L
Mountney et al., 2016	U*	U	U	U	L	L	L	U	L	H
Shear et al., 2016	U*	U	U	U	L	L	L	U	L	H
Arroyo-Araujo et al., 2019	L	L	U	U	L	U	L	L	L	H
Jha et al., 2020	U*	U	U	U	L	L	L	U	L	H
Kliewer et al., 2020	U*	U	U	U	L	U	L	U	L	L

Legend: H=High risk of bias (red), L=Low risk of bias (green), U=Unclear risk of bias (yellow).
Baseline Characteristics: Low risk = Relevant baseline characteristics equal between experimental groups or controlled for. Unclear = Relevant baseline characteristics are unreported. High risk = Relevant baseline characteristics unbalanced between experimental groups and not controlled.
Random Housing: Low risk = Animal cages were randomly placed within an animal room/facility, Unclear = Housing placement unreported, High risk = Animals placed in a non-random arrangement in animal room/facility.
Blinding of Outcome Assessment: Low risk = Outcome assessors were blinded to the study groups when assessing endpoints/animals Unclear = Insufficient information to determine if outcome assessors were blinded during an assessment. High Risk = Outcome assessors not blinded to the study groups.
Incomplete Outcome Data: Low risk = N values were consistent between methods and results for the outcomes. Unclear = N value was either not presented in the methods or in the results, and therefore there is insufficient information to permit judgment. High risk = N values were not consistent between methods and results for the outcomes.
Selective Reporting: Low risk = The methods section indicated pre-specified outcome measures. Unclear: Was not clear about the pre-specified primary endpoints and outcome results. High risk = The outcome was presented in the results but not pre-specified in the methods section.
*

Method of randomization not specified.
†

Assessed as high because one arm of the study was inadvertently unblinded.
‡

Some labs used appropriate randomization where others used pseudo-randomization.
§

Items in agreement with the Cochrane Risk of Bias tool.
¶

other sources include funding influences, conflicts of interest, contamination, a unit of analysis errors.

Table 4

Comparison of characteristics between single lab and multilaboratory studies.

	Multilaboratory studies (n=16)	Single lab studies (n=100)
Median sample size (range)	111 (23–384)	19 (10–72)
Total animals used	2,145	2,166
Publication date range	1985–2020	1980–2019
Disease model	n (%)	n (%)
TBI	6 (38)	46 (46)
Myocardial infarction	2 (13)	20 (20)
Stroke	2 (13)	16 (16)
Traumatic injury	2 (13)	10 (10)
Stimulant exposure	2 (13)	NA
Diabetes	1 (6)	2 (2)
Autism spectrum disorder	1 (6)	6 (6)
Animal species	n (%)	n (%)
Rat	7 (44)	42 (42)
Mouse	6 (38)	31 (31)
Swine	3 (19)	12 (12)
Rabbit	1 (6)	3 (3)
Dog	1 (6)	9 (9)
Monkey	0	2 (2)
Cat	0	1 (1)
Animal sex	n (%)	n (%)
Male	10 (63)	69 (69)
Female	2 (13)	13 (13)
Both	3 (19)	12 (12)
Not reported	1 (6)	6 (6)
Quality domain	Percent of studies that performed each measure (%)
Randomization	94	57
Randomization methods	19	7
Blinding of personnel	69	24
Blinding of outcome assessment	75	53
Complete outcome data	38	38

Appendix 1—table 1

Statements of future recommendations.

Author, Year	Recommendation statements
Reimer et al., 1985	Nothing reported
Crabbe et al., 1999	Relatively small genetic effects should first be replicated locally before drawing conclusions... genotypes should be tested in multiple labs and evaluated with multiple tests of a single behavioral domain
Alam et al., 2009	Based upon the findings of the current study that demonstrated the impressive hemostatic properties of plasma, we have proceeded to successfully develop and test (in the same model) lyophilized [freeze dried plasma].
Spoerke et al., 2009	The species-specific differences in factor activities will require ongoing investigation to ensure full safety and efficacy. Our future investigations will include a comprehensive evaluation of the effects of the lyophilization process on coagulation properties of the LP.
Jones et al., 2015	other investigators can adopt the protocols [for measuring infarct size in mice, rabbits, and pigs in a manner that is rigorous, accurate, and reproducible] in their own laboratories.
Llovera et al., 2015	future clinical trials testing immunotherapeutic drugs for stroke will need to ensure that the included study population feature a substantial neuroinflammatory reaction to the brain injury
Maysami et al., 2016	interleukin 1 receptor antagonist should be evaluated in more extensive clinical stroke trials
Bramlett et al., 2016	Although we cannot rule out the possibility that other doses or more prolonged treatment could show different effects, the lack of efficacy of EPO reduced enthusiasm for its further investigation in OBTT.
Browning et al., 2016	…need for OBTT to study LEV further. This includes studies of dose response, therapeutic window, mechanism, and testing in our large animal FPI model in micropigs… consider a randomized controlled trial examining early administration in patients
Dixon et al., 2016	Our findings reduce enthusiasm for further investigation of this therapy in OBTT and suggest that if this strategy is to be pursued further, alternative CsA analogs with reduced toxicity should be used.
Gill et al., 2016	…pause in proceeding with clinical trials without further preclinical testing.
Mountney et al., 2016	the current findings do not support the beneficial effects of simvastatin… it will not be further pursued by OBTT.
Shear et al., 2016	The marginal benefits achieved with nicotinamide, however, which appeared sporadically across the TBI models, has reduced enthusiasm for further investigation by the OBTT Consortium.
Arroyo-Araujo et al., 2019	Nothing reported
Jha et al., 2020	Optimizing [GLY] treatment regimens (dose, duration, timing), surrogate markers for edema subtypes on MRI, pathway-specific biomarkers, and genetic risk stratification may facilitate precision medicine and patient selection for future clinical trials.
Kliewer et al., 2020	Nothing reported

Legend: FDP – Freeze-dried plasma; LP – Lyophilized plasma; EPO – Erythropietin; OBTT – Operation Brain Trauma Therapy; LEV – Levetiracetam; FPI – Fluid percussion brain injury; CsA – cyclosporin-A; cyclosporine; TBI – Traumatic Brain Injury; GLY - Glibenclamide.

Appendix 1—table 2

Risk of bias for other sources of bias.

Study	Funding influences	Conflicts of interest	Contamination	Unit of analysis errors
Reimer et al., 1985	L	U	L	U
Crabbe et al., 1999	L	U	L	H
Alam et al., 2009	L	U*	L	U
Spoerke et al., 2009	L	U*	L	U
Jones et al., 2015	L	L	L	L
Llovera et al., 2015	L	L	L	U
Maysami et al., 2016	L	H	L	U
Bramlett et al., 2016	L	H	L	U
Browning et al., 2016	L	H	L	U
Dixon et al., 2016	L	H	L	U
Gill et al., 2016	L	L	L	U
Mountney et al., 2016	L	H	L	U
Shear et al., 2016	L	H	L	U
Arroyo-Araujo et al., 2019	H	H	L	L
Jha et al., 2020	L	H	L	U
Kliewer et al., 2020	L	L	L	U

*

financial disclosure, no statement of other conflicts provided.

Appendix 1—table 3

Frequency of reported preclinical multilaboratory checklist items.

Domain	#	Item Description	% of studies that reported
Intro/ abstract	1	Identification as a multicenter/multilaboratory study in title	38
Intro/ abstract	2	Abstract states number of participating centers	50
Standards	3	Community based reporting guidelines listed	13
	4	Names of each participating center listed	100
	5	List roles of participating centers (central coordinating center, experimental site)	88
	6	No changes, or if applicable major changes to study protocol after commencement are documented	94
Replicates (biological vs. technical)	7	Results substantiated by repetition under a range of conditions at each site	100
	8	Number of subjects per outcome	100
	9	Number of measurements per subject for one experimental outcome stated	75
	10	Number of subjects per lab	81
Statistics	11	List of the total number of subjects used in each experimental group	81
	12	List of all statistical tests used	100
	13	Definition of the measure of central tendency	100
	14	Definition of the measure of dispersion	100
Randomization	15	Random group assignment reported	100
Randomization	16	Description of the method of random group assignment	31
Blinding	17	Experimenters blinded to group allocation during conduct of the experiment	75
Blinding	18	Experimenters blinded to group allocation during result assessment	75
Sample Size Estimation	19	Description of an a priori primary outcome	94
	20	Sample size for each site computed during study design	31
	21	Description of the method of sample size determination	31
Inclusion and Exclusion Criteria	22	Total number of animals for the experiment reported	88
	23	Description of the criteria used for the exclusion of any data or subjects	50
	24	List losses and exclusions of animals at the end of experiment	50
	25	All outcomes described, or description of any outcomes that were measured and not reported in the results section	100
	26	Previous or pilot/preliminary studies performed and listed	88
	27	Results were significant, or if not, null or negative outcomes included in the results	100
Discussion	28	Limitations of the study are documented	75
Discussion	29	Discrepancies in results across labs expected or absent, or if not, they discussed	100

Legend: Coloured cells indicate the frequency (%) of item reported over all included studies. Frequency (%) ranges: 0-37 = red; 38-76 = yellow; 77-100 = green.

Appendix 1—table 4

Preclinical single lab studies selection process for the comparison.

	Reimer et al., 1985	Spoerke et al., 2009	Alam et al., 2009	Llovera et al., 2015	Maysami et al., 2016	Gill et al., 2016	Bramlett et al., 2016	Browning et al., 2016	Dixon, 2016	Mountney et al., 2016	Shear et al., 2016	Arroyo-Araujo et al., 2019	Jha et al., 2020
Records identified/ abstracts screened	71	189	61	52	33	39	177	31	197	45	28	26	29
Full-texts assessed for eligibility	29	22	18	14	21	11	57	15	48	30	18	12	10
Records considered eligible	13	5	5	6	10	2	1 SR	4	8	9	12	6	5
Records used for comparison	10	5	5	6	10	2	1 SR	4	8	9	10	6	5

Appendix 1—table 5

Multilaboratory reporting checklist with item domain and source(s).

Domain	#	Item Description	Source(s)
Intro/abstract	1	Identification as a multicenter/multilaboratory study in title	CONSORT
Intro/abstract	2	Abstract states number of participating centers	CONSORT
Standards	3	Community based reporting guidelines listed	NIH
	4	Names of each participating center listed	GCP E6(R2)
	5	List roles of participating centers (central coordinating center, experimental site)	GCP E6(R2)
	6	No changes, or if applicable major changes to study protocol after commencement are documented	CONSORT
Replicates (biological vs. technical)	7	Results substantiated by repetition under a range of conditions at each site	NIH, CONSORT
	8	Number of subjects per outcome	NIH, CONSORT
	9	Number of measurements per subject for one experimental outcome stated	NIH, CONSORT
	10	Number of subjects per lab	GCP E6(R2)
Statistics	11	List of the total number of subjects used in each experimental group	NIH, CONSORT
	12	List of all statistical tests used	NIH, CONSORT
	13	Definition of the measure of central tendency	NIH
	14	Definition of the measure of dispersion	NIH
Randomization	15	Random group assignment reported	NIH, CONSORT
Randomization	16	Description of the method of random group assignment	NIH, CONSORT
Blinding	17	Experimenters blinded to group allocation during conduct of the experiment	NIH, CONSORT
Blinding	18	Experimenters blinded to group allocation during result assessment	NIH, CONSORT
Sample Size Estimation	19	Description of an a priori primary outcome	CONSORT
	20	Sample size computed during study design	NIH, CONSORT
	21	Description of the method of sample size determination	NIH, CONSORT
Inclusion and Exclusion Criteria	22	Total number of animals for the experiment reported	GCP E6(R2)
	23	Description of the criteria used for the exclusion of any data or subjects	NIH, CONSORT
	24	List losses and exclusions of animals at the end of experiment	CONSORT
	25	All outcomes described, or description of any outcomes measured but not reported in results	NIH, CONSORT
	26	Previous or pilot/preliminary studies performed and listed	NIH
	27	Results were significant, or if not, null or negative outcomes included in the results	NIH
Discussion	28	Limitations of the study are documented	CONSORT
Discussion	29	Discrepancies in results across labs expected or absent, or if not, they discussed	CONSORT