Improving reproducibility in animal research by splitting the study population into several 'mini-experiments'.
Journal
Scientific reports
ISSN: 2045-2322
Titre abrégé: Sci Rep
Pays: England
ID NLM: 101563288
Informations de publication
Date de publication:
06 10 2020
06 10 2020
Historique:
received:
07
04
2020
accepted:
11
09
2020
entrez:
7
10
2020
pubmed:
8
10
2020
medline:
9
3
2021
Statut:
epublish
Résumé
In light of the hotly discussed 'reproducibility crisis', a rethinking of current methodologies appears essential. Implementing multi-laboratory designs has been shown to enhance the external validity and hence the reproducibility of findings from animal research. We here aimed at proposing a new experimental strategy that transfers this logic into a single-laboratory setting. We systematically introduced heterogeneity into our study population by splitting an experiment into several 'mini-experiments' spread over different time points a few weeks apart. We hypothesised to observe improved reproducibility in such a 'mini-experiment' design in comparison to a conventionally standardised design, according to which all animals are tested at one specific point in time. By comparing both designs across independent replicates, we could indeed show that the use of such a 'mini-experiment' design improved the reproducibility and accurate detection of exemplary treatment effects (behavioural and physiological differences between four mouse strains) in about half of all investigated strain comparisons. Thus, we successfully implemented and empirically validated an easy-to-handle strategy to tackle poor reproducibility in single-laboratory studies. Since other experiments within different life science disciplines share the main characteristics with the investigation reported here, these studies are likely to also benefit from this approach.
Identifiants
pubmed: 33024165
doi: 10.1038/s41598-020-73503-4
pii: 10.1038/s41598-020-73503-4
pmc: PMC7538440
doi:
Types de publication
Journal Article
Research Support, Non-U.S. Gov't
Langues
eng
Sous-ensembles de citation
IM
Pagination
16579Références
McNutt, M. Reproducibility. Science 343, 229. https://doi.org/10.1126/science.1250475 (2014).
doi: 10.1126/science.1250475
pubmed: 24436391
Drucker, D. J. Crosstalk never waste a good crisis: Confronting reproducibility in translational research crosstalk. Cell Metab. 24, 348–360 (2016).
pubmed: 27626191
doi: 10.1016/j.cmet.2016.08.006
Reed, W. R. For the student a primer on the ‘ reproducibility crisis ’ and ways to fix it. Aust. Econ. Rev. 51, 286–300 (2018).
doi: 10.1111/1467-8462.12262
Samsa, G. & Samsa, L. A guide to reproducibility in preclinical research. Acad. Med. 94, 47–52 (2019).
pubmed: 29995667
doi: 10.1097/ACM.0000000000002351
Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 (2016).
pubmed: 27225100
doi: 10.1038/533452a
Begley, C. G. & Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
pubmed: 22460880
doi: 10.1038/483531a
Nosek, B. A. & Errington, T. M. Reproducibility in cancer biology: Making sense of replications. Elife 6, e23383 (2017).
pubmed: 28100398
pmcid: 5245957
doi: 10.7554/eLife.23383
Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
doi: 10.1126/science.aac4716
Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: How much can we rely on published data on potential drug targets ?. Nat. Publ. Gr. https://doi.org/10.1038/nrd3439-c1 (2011).
doi: 10.1038/nrd3439-c1
Begley, C. G. & Ioannidis, J. P. A. Reproducibility in science: Improving the standard for basic and preclinical research. Circ. Res. 116, 116–126 (2015).
pubmed: 25552691
doi: 10.1161/CIRCRESAHA.114.303819
Branch, M. N. The, “ Reproducibility Crisis: ” Might the methods used frequently in behavior-analysis research help?. Perspect. Behav. Sci. 42, 77–89 (2019).
pubmed: 31976422
doi: 10.1007/s40614-018-0158-5
Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13(6), 1–9. https://doi.org/10.1371/journal.pbio.1002165 (2015).
doi: 10.1371/journal.pbio.1002165
Head, M. L., Holman, L., Lanfear, R., Kahn, A. T. & Jennions, M. D. The extent and consequences of P-hacking in science. PLoS Biol. 13(3), 1–15. https://doi.org/10.1371/journal.pbio.1002106 (2015).
doi: 10.1371/journal.pbio.1002106
Kerr, N. L. HARKing: Hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2, 196–217 (1998).
doi: 10.1207/s15327957pspr0203_4
Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015).
pubmed: 26113702
pmcid: 4550299
doi: 10.1126/science.aab2374
Kilkenny, C., Browne, W., Cuthill, I. C., Emerson, M. & Altman, D. G. Animal research: Reporting in vivo experiments: The ARRIVE guidelines. Br. J. Pharmacol. 160, 1577–1579. https://doi.org/10.1111/j.1476-5381.2010.00872.x (2010).
doi: 10.1111/j.1476-5381.2010.00872.x
pubmed: 20649561
pmcid: 2936830
Percie du Sert N, Hurst V, Ahluwalia A, Alam S, Avey MT, Baker M, et al. The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research. PLoS Biol. 18 (7), e3000410. https://doi.org/10.1371/journal.pbio.3000410 (2020)
doi: 10.1371/journal.pbio.3000410
pubmed: 32663219
pmcid: 7360023
Smith, A. J., Clutton, R. E., Lilley, E., Hansen, K. E. A. & Brattelid, T. PREPARE: Guidelines for planning animal research and testing. Lab. Anim. 52, 135–141 (2018).
pubmed: 28771074
doi: 10.1177/0023677217724823
Nosek, B. A. & Lakens, D. Editorial registered reports. Soc. Psychol. 45, 137–141 (2014).
doi: 10.1027/1864-9335/a000192
Center for Open Science https://osf.io/ (2020).
Wharton University of Pennsylvania, Credibility Lab, AsPredicted https://aspredicted.org/ (2020).
German Federal Institute for Risk Assessment, Animal Study Registry https://www.animalstudyregistry.org/ (2020).
NPQIP Collaborative Group. Did a change in Nature journals’ editorial policy for life sciences research improve reporting?. BMJ Open Sci. https://doi.org/10.17605/OSF.IO/HC7FK (2019).
doi: 10.17605/OSF.IO/HC7FK
Crabbe, J. C., Wahlsten, D. & Dudek, B. C. Genetics of mouse behavior: Interactions with laboratory environment. Science 284, 1670–1672 (1999).
doi: 10.1126/science.284.5420.1670
Castelhano-Carlos, M. J. & Baumans, V. The impact of light, noise, cage cleaning and in-house transport on welfare and stress of laboratory rats. Lab. Anim. 43, 311–327 (2009).
pubmed: 19505937
doi: 10.1258/la.2009.0080098
Leystra, A. A. & Clapper, M. L. Gut microbiota influences experimental outcomes in mouse models of colorectal cancer. Genes 10, 900 (2019).
pmcid: 6895921
doi: 10.3390/genes10110900
pubmed: 6895921
Sorge, R. E. et al. Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nat. Methods 11, 629–632 (2014).
pubmed: 24776635
doi: 10.1038/nmeth.2935
Voelkl, B., Vogt, L., Sena, E. S. & Würbel, H. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol. 16, 1–13. https://doi.org/10.1371/journal.pbio.2003693 (2018).
doi: 10.1371/journal.pbio.2003693
Richter, S. H. Systematic heterogenization for better reproducibility in animal experimentation. Lab. Anim. (NY) 46, 343 (2017).
doi: 10.1038/laban.1330
Voelkl, B. et al. Reproducibility of animal research in light of biological variation. Nat. Rev. Neurosci. 21, 384–393. https://doi.org/10.1038/s41583-020-0313-3 (2020).
doi: 10.1038/s41583-020-0313-3
pubmed: 32488205
Richter, S. H., Garner, J. P. & Würbel, H. Environmental standardization: Cure or cause of poor reproducibility in animal experiments?. Nat. Methods 6, 257–261 (2009).
pubmed: 19333241
doi: 10.1038/nmeth.1312
Richter, S. H., Garner, J. P., Auer, C., Kunert, J. & Würbel, H. Systematic variation improves reproducibility of animal experiments. Nat. Methods 7, 167–168 (2010).
pubmed: 20195246
doi: 10.1038/nmeth0310-167
Richter, S. H. et al. Effect of population heterogenization on the reproducibility of mouse behavior: A multi-laboratory study. PLoS ONE 6, e16461 (2011).
pubmed: 21305027
pmcid: 3031565
doi: 10.1371/journal.pone.0016461
Bodden, C. et al. Heterogenising study samples across testing time improves reproducibility of behavioural data. Sci. Rep. 9, 1–9. https://doi.org/10.1038/s41598-019-44705-2 (2019).
doi: 10.1038/s41598-019-44705-2
Richter, S.H., von Kortzfleisch, V. It is time for an empirically informed paradigm shift in animal research. Nat. Rev. Neurosci. 1, 1. https://doi.org/10.1038/s41583-020-0369-0 (2020).
doi: 10.1038/s41583-020-0369-0
Bailoo, J. D., Reichlin, T. S. & Würbel, H. Refinement of experimental design and conduct in laboratory animal research. ILAR J. 55, 383–391 (2014).
pubmed: 25541540
doi: 10.1093/ilar/ilu037
Paylor, R. Questioning standardization in science Footprints by deep sequencing. Nat. Methods 6, 253–254 (2009).
pubmed: 19333239
doi: 10.1038/nmeth0409-253
Chesler, E. J., Wilson, S. G., Lariviere, W. R., Rodriguez-Zas, S. L. & Mogil, J. S. Identification and ranking of genetic and laboratory environment factors influencing a behavioral trait, thermal nociception, via computational analysis of a large data archive. Neurosci. Biobehav. Rev. 26, 907–923 (2002).
pubmed: 12667496
doi: 10.1016/S0149-7634(02)00103-3
Karp, N. A. et al. Impact of temporal variation on design and analysis of mouse knockout phenotyping studies. PLoS ONE 9, e111239 (2014).
pubmed: 25343444
pmcid: 4208881
doi: 10.1371/journal.pone.0111239
Lad, H. V. et al. Physiology and behavior behavioural battery testing: Evaluation and behavioural outcomes in 8 inbred mouse strains. Physiol. Behav. 99, 301–316 (2010).
pubmed: 19931548
doi: 10.1016/j.physbeh.2009.11.007
Mandillo, S. et al. Reliability, robustness, and reproducibility in mouse behavioral phenotyping: A cross-laboratory study. Physiol. Genom. 34, 243–255. https://doi.org/10.1152/physiolgenomics.90207.2008 (2008).
doi: 10.1152/physiolgenomics.90207.2008
Brooks, S. P., Pask, T., Jones, L. & Dunnett, S. B. Behavioural profiles of inbred mouse strains used as transgenic backgrounds II: Cognitive tests. Genes Brain Behav. 4, 307–317. https://doi.org/10.1111/j.1601-183X.2004.00109.x (2005).
doi: 10.1111/j.1601-183X.2004.00109.x
pubmed: 16011577
Podhorna, J. & Brown, R. E. Strain differences in activity and emotionality do not account for differences in learning and memory performance between C57BL/6 and DBA/2 mice. Genes Brain Behav. 1, 96–110. https://doi.org/10.1034/j.1601-183X.2002.10205.x (2002).
doi: 10.1034/j.1601-183X.2002.10205.x
pubmed: 12884980
Kafkafi, N., Lahav, T. & Benjamini, Y. What’s always wrong with my mouse. Proceedings of Measuring Behavior 2014: 9
Pigliucci, M. Phenotypic plasticity: Beyond nature and nurture (JHU Press, Baltimore, 2001).
Voelkl, B. & Würbel, H. Reproducibility crisis: Are we ignoring reaction norms?. Trends Pharmacol. Sci. 37, 509–510 (2016).
pubmed: 27211784
doi: 10.1016/j.tips.2016.05.003
Åhlgren, J. & Voikar, V. Experiments done in Black-6 mice: What does it mean?. Lab. Anim. 48, 171. https://doi.org/10.1038/s41684-019-0288-8 (2019).
doi: 10.1038/s41684-019-0288-8
Bohlen, M. et al. Experimenter effects on behavioral test scores of eight inbred mouse strains under the influence of ethanol. Behav. Brain Res. 272, 46–54. https://doi.org/10.1016/j.bbr.2014.06.017 (2014).
doi: 10.1016/j.bbr.2014.06.017
pubmed: 24933191
pmcid: 4968576
Milcu, A. et al. Genotypic variability enhances the reproducibility of an ecological study. Nat. Ecol. Evol. 2, 279–287 (2018).
pubmed: 29335575
doi: 10.1038/s41559-017-0434-x
Karp, N. A., Melvin, D., Mouse, S., Project, G. & Mott, R. F. Robust and sensitive analysis of mouse knockout phenotypes. PLoS ONE 7, e52410 (2012).
doi: 10.1371/journal.pone.0052410
Krakenberg, V. et al. Technology or ecology ? New tools to assess cognitive judgement bias in mice. Behav. Brain Res. 362, 279–287 (2019).
pubmed: 30654122
doi: 10.1016/j.bbr.2019.01.021
Beynen, A. C., Gärtner, K. & Van Zutphen, L. F. M. Standardization of animal experimentation. Princ. Lab. Anim. Sci. A Contrib. to Hum. Use Care Anim. to Qual. Exp. Results. 2nd edn. Amsterdam Elsevier 103–110 (2001).
Festing, M. F. W. Refinement and reduction through the control of variation. Altern. Lab. Anim. 32, 259–263 (2004).
pubmed: 23577470
doi: 10.1177/026119290403201s43
Festing, M. F. W. Randomized block experimental designs can increase the power and reproducibility of laboratory animal experiments. ILAR J. 55, 472–476 (2014).
pubmed: 25541548
doi: 10.1093/ilar/ilu045
Karp, N. A. et al. A multi-batch design to deliver robust estimates of efficacy and reduce animal use—a syngeneic tumour case study. Sci. Rep. 10, 1–10. https://doi.org/10.1038/s41598-020-62509-7 (2020).
doi: 10.1038/s41598-020-62509-7
Russell, W. M. S., Burch, R. L. & Hume, C. W. The principles of humane experimental technique. Methuen London 238, 64 (1959).
Würbel, H. Focus on reproducibility more than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research Focus on Reproducibility. Nat. Publ. Gr. 46, 164–166 (2017).
Kappel, S., Hawkins, P. & Mendl, M. T. To group or not to group? Good practice for housing male laboratory mice. Animals 7, 88 (2017).
doi: 10.3390/ani7120088
Melotti, L. et al. Can live with ‘em, can live without ‘em: Pair housed male C57BL/6J mice show low aggression and increasing sociopositive interactions with age, but can adapt to single housing if separated. Appl. Anim. Behav. Sci. 214, 79–88 (2019).
doi: 10.1016/j.applanim.2019.03.010
Lister, R. G. The use of a plus-maze to measure anxiety in the mouse. Psychopharmacology 92, 180–185 (1987).
pubmed: 3110839
Crawley, J. N. Exploratory behavior models of anxiety in mice. Neurosci. Biobehav. Rev. 9, 37–44 (1985).
pubmed: 2858080
doi: 10.1016/0149-7634(85)90030-2
Fuss, J. et al. Are you real ? Visual simulation of social housing by mirror image stimulation in single housed mice. Behav. Brain Res. 243, 191–198 (2013).
pubmed: 23333841
doi: 10.1016/j.bbr.2013.01.015
Chourbaji, S. et al. Nature vs nurture: Can enrichment rescue the behavioural phenotype of BDNF heterozygous mice?. Behav. Brain Res. 192, 254–258 (2008).
pubmed: 18538870
doi: 10.1016/j.bbr.2008.04.015
O’Connor, A. M., Burton, T. J., Leamey, C. A. & Sawatari, A. The use of the puzzle box as a means of assessing the efficacy of environmental enrichment. JoVE J. Vis. Exp. 94, e52225 (2014).
Touma, C., Sachser, N., Erich, M. & Palme, R. Effects of sex and time of day on metabolism and excretion of corticosterone in urine and feces of mice. Gen. Comp. Endocrinol. 130, 267–278 (2003).
pubmed: 12606269
doi: 10.1016/S0016-6480(02)00620-2
Touma, C., Palme, R. & Sachser, N. Analyzing corticosterone metabolites in fecal samples of mice: A noninvasive technique to monitor stress hormones. Horm. Behav. 45, 10–22 (2004).
pubmed: 14733887
doi: 10.1016/j.yhbeh.2003.07.002
Auer, K. E. et al. Measurement of fecal testosterone metabolites in mice: Replacement of invasive techniques. Animals 10, 1–17 (2020).
doi: 10.3390/ani10010165
Strekalova, T., Spanagel, R., Bartsch, D., Henn, F. A. & Gass, P. Stress-induced anhedonia in mice is associated with deficits in forced swimming and exploration. Neuropsychopharmacology 29, 2007–2017. https://doi.org/10.1038/sj.npp.1300532 (2017).
doi: 10.1038/sj.npp.1300532
Deacon, R. M. J. Assessing nest building in mice. Nat. Protoc. 1, 1117–1119 (2006).
pubmed: 17406392
doi: 10.1038/nprot.2006.170
Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010).
doi: 10.18637/jss.v036.i03
Lenth, R. & Lenth, M. R. Package ‘lsmeans’. Am. Stat. 34, 216–221 (2018).
R Core Team. R: A Language and Environment for Statistical Computing.