The Standard Error/Standard Deviation Mix-Up: Potential Impacts on Meta-Analyses in Sports Medicine.

Journal

Sports medicine (Auckland, N.Z.)

ISSN: 1179-2035

Titre abrégé: Sports Med

Pays: New Zealand

ID NLM: 8412297

Informations de publication

Date de publication:
25 Jan 2024

Historique:

accepted: 18 12 2023

medline: 25 1 2024

pubmed: 25 1 2024

entrez: 25 1 2024

Statut: aheadofprint

Résumé

A recent review found that 45% of meta-analyses included statistical errors, of which, the most common was the calculation of effect sizes based on standard error (SE) rather than standard deviation (SD) [the SE/SD mix-up]. The first aim of this study was to assess the impact of the SE/SD mix-up on the results of one highly cited meta-analysis. Our second aim was to identify one potential source of the SE/SD mix-up, by assessing how often SE is reported as a measure of sample variability in randomised controlled trials in sports medicine. We checked for potential SE/SD mix-ups in a 2015 meta-analysis of randomised controlled trials reporting the effects of recreational football interventions on aerobic fitness in adults. We corrected effect sizes affected by SE/SD mix-ups and re-analysed the data according to the original methodology. We compared pooled estimates of effect sizes from our re-analysis of corrected values with those of the original study. To assess how often SE was reported instead of SD as a measure of sample variance, we text mined results of randomised controlled trials from seven sports medicine journals and reported the proportion reporting of SE versus SD. We identified potential SE/SD mix-ups in 9/16 effect sizes included in the meta-analysis describing the effects of football-based interventions versus non-exercise control. The published effect size was standardised mean difference (SMD) = 1.46 (95% confidence interval [CI] 0.91, 2.01). After correcting for SE/SD mix-ups, our re-analysis produced a smaller pooled estimate (SMD = 0.54 [95% CI 0.37, 0.71]). The original pooled estimate for trials comparing football versus running interventions was SMD = 0.68 (95% CI 0.06, 1.4). After correcting for SE/SD mix-ups and re-analysis, the effect was no longer statistically significant (SMD = 0.20 [95% CI - 0.10, 0.49)]). We found that 19.3% of randomised controlled trials reported SE rather than SD to describe sample variability. The relative frequency of the practice ranged from 0 to 25% across the seven journals sampled. We found the SE/SD mix-up had inflated estimates for the effects of football on aerobic fitness. Meta-analysts should be vigilant to avoid miscalculating effect sizes. Authors, reviewers and editors should avoid and discourage (respectively) the practice of reporting SE as a measure of sample variability in sports medicine research.

Sections du résumé

BACKGROUND BACKGROUND

OBJECTIVES OBJECTIVE

The first aim of this study was to assess the impact of the SE/SD mix-up on the results of one highly cited meta-analysis. Our second aim was to identify one potential source of the SE/SD mix-up, by assessing how often SE is reported as a measure of sample variability in randomised controlled trials in sports medicine.

METHODS METHODS

We checked for potential SE/SD mix-ups in a 2015 meta-analysis of randomised controlled trials reporting the effects of recreational football interventions on aerobic fitness in adults. We corrected effect sizes affected by SE/SD mix-ups and re-analysed the data according to the original methodology. We compared pooled estimates of effect sizes from our re-analysis of corrected values with those of the original study. To assess how often SE was reported instead of SD as a measure of sample variance, we text mined results of randomised controlled trials from seven sports medicine journals and reported the proportion reporting of SE versus SD.

RESULTS RESULTS

We identified potential SE/SD mix-ups in 9/16 effect sizes included in the meta-analysis describing the effects of football-based interventions versus non-exercise control. The published effect size was standardised mean difference (SMD) = 1.46 (95% confidence interval [CI] 0.91, 2.01). After correcting for SE/SD mix-ups, our re-analysis produced a smaller pooled estimate (SMD = 0.54 [95% CI 0.37, 0.71]). The original pooled estimate for trials comparing football versus running interventions was SMD = 0.68 (95% CI 0.06, 1.4). After correcting for SE/SD mix-ups and re-analysis, the effect was no longer statistically significant (SMD = 0.20 [95% CI - 0.10, 0.49)]). We found that 19.3% of randomised controlled trials reported SE rather than SD to describe sample variability. The relative frequency of the practice ranged from 0 to 25% across the seven journals sampled.

CONCLUSIONS CONCLUSIONS

We found the SE/SD mix-up had inflated estimates for the effects of football on aerobic fitness. Meta-analysts should be vigilant to avoid miscalculating effect sizes. Authors, reviewers and editors should avoid and discourage (respectively) the practice of reporting SE as a measure of sample variability in sports medicine research.

Identifiants

DOI: 10.1007/s40279-023-01989-9 PMID: 38270793

pubmed: 38270793

doi: 10.1007/s40279-023-01989-9

pii: 10.1007/s40279-023-01989-9

doi:

Types de publication

Journal Article

Langues

eng

Sous-ensembles de citation

Informations de copyright

Références

Kadlec D, Sainani KL, Nimphius S. With great power comes great responsibility: common errors in meta-analyses and meta-regressions in strength & conditioning research. Sports Med. 2023;53(2):313–25.

pubmed: 36208412 doi: 10.1007/s40279-022-01766-0

Milanovic Z, Pantelic S, Covic N, Sporis G, Krustrup P. Is recreational soccer effective for improving VO

pubmed: 26210229 pmcid: 4536283 doi: 10.1007/s40279-015-0361-4

Sloth M, Sloth D, Overgaard K, Dalgas U. Effects of sprint interval training on VO

pubmed: 23889316 doi: 10.1111/sms.12092

Salom Huffman L, Wadsworth DD, McDonald JR, Foote SJ, Hyatt H, Pascoe DD. Effects of a sprint interval and resistance concurrent exercise training program on aerobic capacity of inactive adult women. J Strength Cond Res. 2019;33(6):1640–7.

pubmed: 28777244 doi: 10.1519/JSC.0000000000002013

Gist NH, Fedewa MV, Dishman RK, Cureton KJ. Sprint interval training effects on aerobic capacity: a systematic review and meta-analysis. Sports Med. 2014;44(2):269–79.

pubmed: 24129784 doi: 10.1007/s40279-013-0115-0

Hespanhol Junior LC, Pillay JD, van Mechelen W, Verhagen E. Meta-analyses of the effects of habitual running on indices of health in physically inactive adults. Sports Med. 2015;45(10):1455–68.

pubmed: 26178328 pmcid: 4579257 doi: 10.1007/s40279-015-0359-y

Huang G, Gibson CA, Tran ZV, Osness WH. Controlled endurance exercise training and VO

pubmed: 16230876 doi: 10.1111/j.0197-3118.2005.04324.x

Seitz LB, Reyes A, Tran TT, Saez de Villarreal E, Haff GG. Increases in lower-body strength transfer positively to sprint performance: a systematic review with meta-analysis. Sports Med. 2014;44(12):1693–702.

pubmed: 25059334 doi: 10.1007/s40279-014-0227-1

Herxheimer A. Misuse of the standard error of the mean. Br J Clin Pharmacol. 1988;26(2):197.

pubmed: 3207558 pmcid: 1386530 doi: 10.1111/j.1365-2125.1988.tb03389.x

Ko WR, Hung WT, Chang HC, Lin LY. Inappropriate use of standard error of the mean when reporting variability of study samples: a critical evaluation of four selected journals of obstetrics and gynecology. Taiwan J Obstet Gynecol. 2014;53(1):26–9.

pubmed: 24767642 doi: 10.1016/j.tjog.2013.04.035

Nagele P. Misuse of standard error of the mean (SEM) when reporting variability of a sample: a critical evaluation of four anaesthesia journals. Br J Anaesth. 2003;90(4):514–6.

pubmed: 12644429 doi: 10.1093/bja/aeg087

Feinstein AR. Clinical biostatistics. XXXVII. Demeaned errors, confidence games, nonplussed minuses, inefficient coefficients, and other statistical disruptions of scientific communication. Clin Pharmacol Ther. 1976;20(5):617–31.

pubmed: 985637 doi: 10.1002/cpt1976205617

Strasak AM, Zaman Q, Pfeiffer KP, Gobel G, Ulmer H. Statistical errors in medical research: a review of common pitfalls. Swiss Med Wkly. 2007;137(3–4):44–9.

pubmed: 17299669

Wullschleger M, Aghlmandi S, Egger M, Zwahlen M. High incorrect use of the standard error of the mean (SEM) in original articles in three cardiovascular journals evaluated for 2012. PLoS ONE. 2014;9(10): e110364.

pubmed: 25354300 pmcid: 4212967 doi: 10.1371/journal.pone.0110364

Altman DG, Bland JM. Standard deviations and standard errors. BMJ. 2005;331(7521):903.

pubmed: 16223828 pmcid: 1255808 doi: 10.1136/bmj.331.7521.903

Andrade C. Understanding the difference between standard deviation and standard error of the mean, and knowing when to use which. Indian J Psychol Med. 2020;42(4):409–10.

pubmed: 33402813 pmcid: 7746895 doi: 10.1177/0253717620933419

Barde MP, Barde PJ. What to use to express the variability of data: standard deviation or standard error of mean? Perspect Clin Res. 2012;3(3):113–6.

pubmed: 23125963 pmcid: 3487226 doi: 10.4103/2229-3485.100662

Carter RE. A standard error: distinguishing standard deviation from standard error. Diabetes. 2013;62(8): e15.

pubmed: 23881207 pmcid: 3717876 doi: 10.2337/db13-0692

Streiner DL. Maintaining standards: differences between the standard deviation and standard error, and when to use each. Can J Psychiatry. 1996;41(8):498–502.

pubmed: 8899234 doi: 10.1177/070674379604100805

Lang T. Twenty statistical errors even YOU can find in biomedical research articles. Croat Med J. 2004;45(4):361–70.

pubmed: 15311405

Lee S, Lee KH, Park KM, Park SJ, Kim WJ, Lee J, et al. Impact of data extraction errors in meta-analyses on the association between depression and peripheral inflammatory biomarkers: an umbrella review. Psychol Med. 2023;53(5):2017–30.

pubmed: 34749836 doi: 10.1017/S0033291721003767

Krustrup P, Christensen JF, Randers MB, Pedersen H, Sundstrup E, Jakobsen MD, et al. Muscle adaptations and performance enhancements of soccer training for untrained men. Eur J Appl Physiol. 2010;108(6):1247–58.

pubmed: 20039057 doi: 10.1007/s00421-009-1319-8

Krustrup P, Hansen PR, Randers MB, Nybo L, Martone D, Andersen LJ, et al. Beneficial effects of recreational football on the cardiovascular risk profile in untrained premenopausal women. Scand J Med Sci Sports. 2010;20(Suppl. 1):40–9.

pubmed: 20210906 doi: 10.1111/j.1600-0838.2010.01110.x

Krustrup P, Nielsen JJ, Krustrup BR, Christensen JF, Pedersen H, Randers MB, et al. Recreational soccer is an effective health-promoting activity for untrained men. Br J Sports Med. 2009;43(11):825–31.

pubmed: 19098116 doi: 10.1136/bjsm.2008.053124

Andersen LJ, Randers MB, Hansen PR, Hornstrup T, Schmidt JF, Dvorak J, et al. Structural and functional cardiac adaptations to 6 months of football training in untrained hypertensive men. Scand J Med Sci Sports. 2014;24(Suppl. 1):27–35.

pubmed: 24903081 doi: 10.1111/sms.12237

Barene S, Krustrup P, Brekke OL, Holtermann A. Soccer and Zumba as health-promoting activities among female hospital employees: a 40-weeks cluster randomised intervention study. J Sport Sci. 2014;32(16):1539–49.

doi: 10.1080/02640414.2014.906043

Barene S, Krustrup P, Jackman SR, Brekke OL, Holtermann A. Do soccer and Zumba exercise improve fitness and indicators of health among female hospital employees? A 12-week RCT. Scand J Med Sci Spor. 2014;24(6):990–9.

doi: 10.1111/sms.12138

Randers MB, Petersen J, Andersen LJ, Krustrup BR, Hornstrup T, Nielsen JJ, et al. Short-term street soccer improves fitness and cardiovascular health status of homeless men. Eur J Appl Physiol. 2012;112(6):2097–106.

pubmed: 21956486 doi: 10.1007/s00421-011-2171-1

Schmidt JF, Hansen PR, Andersen TR, Andersen LJ, Hornstrup T, Krustrup P, et al. Cardiovascular adaptations to 4 and 12 months of football or strength training in 65- to 75-year-old untrained men. Scand J Med Sci Sports. 2014;24(Suppl. 1):86–97.

pubmed: 24902992 doi: 10.1111/sms.12217

Knoepfli-Lenzin C, Sennhauser C, Toigo M, Boutellier U, Bangsbo J, Krustrup P, et al. Effects of a 12-week intervention period with football and running for habitually active men with mild hypertension. Scand J Med Sci Sports. 2010;20(Suppl. 1):72–9.

pubmed: 20136764 doi: 10.1111/j.1600-0838.2009.01089.x

Uth J, Fristrup B, Sorensen V, Helge EW, Christensen MK, Kjaergaard JB, et al. One year of football fitness improves L1–L4 BMD, postural balance, and muscle strength in women treated for breast cancer. Scand J Med Sci Sports. 2021;31(7):1545–57.

pubmed: 33794005 doi: 10.1111/sms.13963

Andersen TR, Schmidt JF, Thomassen M, Hornstrup T, Frandsen U, Randers MB, et al. A preliminary study: effects of football training on glucose control, body composition, and performance in men with type 2 diabetes. Scand J Med Sci Sports. 2014;24(Suppl. 1):43–56.

pubmed: 24903461 doi: 10.1111/sms.12259

Krustrup P, Hansen PR, Andersen LJ, Jakobsen MD, Sundstrup E, Randers MB, et al. Long-term musculoskeletal and cardiac health effects of recreational football and running for premenopausal women. Scand J Med Sci Sports. 2010;20(Suppl. 1):58–71.

pubmed: 20546545 doi: 10.1111/j.1600-0838.2010.01111.x

Andersen LJ, Hansen PR, Sogaard P, Madsen JK, Bech J, Krustrup P. Improvement of systolic and diastolic heart function after physical training in sedentary women. Scand J Med Sci Sports. 2010;20(Suppl. 1):50–7.

pubmed: 20136765 doi: 10.1111/j.1600-0838.2009.01088.x

Andersen TR, Schmidt JF, Nielsen JJ, Randers MB, Sundstrup E, Jakobsen MD, et al. Effect of football or strength training on functional ability and physical performance in untrained old men. Scand J Med Sci Sports. 2014;24(Suppl. 1):76–85.

pubmed: 24903323 doi: 10.1111/sms.12245

de Sousa MV, Fukui R, Krustrup P, Pereira RM, Silva PR, Rodrigues AC, et al. Positive effects of football on fitness, lipid profile, and insulin resistance in Brazilian patients with type 2 diabetes. Scand J Med Sci Sports. 2014;24(Suppl. 1):57–65.

pubmed: 24944132 doi: 10.1111/sms.12258

Randers MB, Nielsen JJ, Krustrup BR, Sundstrup E, Jakobsen MD, Nybo L, et al. Positive performance and health effects of a football training program over 12 weeks can be maintained over a 1-year period with reduced training frequency. Scand J Med Sci Sports. 2010;20:80–9.

pubmed: 20210904 doi: 10.1111/j.1600-0838.2010.01091.x

Andersen LJ, Randers MB, Westh K, Martone D, Hansen PR, Junge A, et al. Football as a treatment for hypertension in untrained 30–55-year-old men: a prospective randomized study. Scand J Med Sci Sports. 2010;20:98–102.

pubmed: 20210907 doi: 10.1111/j.1600-0838.2010.01109.x

Oja P, Titze S, Kokko S, Kujala UM, Heinonen A, Kelly P, et al. Health benefits of different sport disciplines for adults: systematic review of observational and intervention studies with meta-analysis. Br J Sports Med. 2015;49(7):434–40.

pubmed: 25568330 doi: 10.1136/bjsports-2014-093885

Sultana RN, Sabag A, Keating SE, Johnson NA. The effect of low-volume high-intensity interval training on body composition and cardiorespiratory fitness: a systematic review and meta-analysis. Sports Med. 2019;49(11):1687–721.

pubmed: 31401727 doi: 10.1007/s40279-019-01167-w

Beato M, Coratella G, Schena F, Impellizzeri FM. Effects of recreational football performed once a week (1 h per 12 weeks) on cardiovascular risk factors in middle-aged sedentary men. Sci Med Football. 2017;1(2):171–7.

doi: 10.1080/24733938.2017.1325966

Uth J, Fristrup B, Sorensen V, Helge EW, Christensen MK, Kjaergaard JB, et al. Exercise intensity and cardiovascular health outcomes after 12 months of football fitness training in women treated for stage I-III breast cancer: results from the football fitness After Breast Cancer (ABC) randomized controlled trial. Prog Cardiovasc Dis. 2020;63(6):792–9.

pubmed: 32800792 doi: 10.1016/j.pcad.2020.08.002

Skoradal MB, Weihe P, Patursson P, Mortensen J, Connolly L, Krustrup P, et al. Football training improves metabolic and cardiovascular health status in 55-to 70-year-old women and men with prediabetes. Scand J Med Sci Spor. 2018;28:42–51.

doi: 10.1111/sms.13081

Møller TK, Nielsen TT, Andersen R, Lundager I, Hansen HF, Ottesen L, et al. Health effects of 12 weeks of team-sport training and fitness training in a community health centre for sedentary men with lifestyle diseases. BioMed Res Int. 2018;2018:1571807.

pubmed: 29888250 pmcid: 5977017 doi: 10.1155/2018/1571807

Milanovic Z, Pantelic S, Sporis G, Mohr M, Krustrup P. Health-related physical fitness in healthy untrained men: effects on VO

pubmed: 26305880 pmcid: 4549149 doi: 10.1371/journal.pone.0135319