Recommendations for Determining the Validity of Consumer Wearables and Smartphones for the Estimation of Energy Expenditure: Expert Statement and Checklist of the INTERLIVE Network.
Journal
Sports medicine (Auckland, N.Z.)
ISSN: 1179-2035
Titre abrégé: Sports Med
Pays: New Zealand
ID NLM: 8412297
Informations de publication
Date de publication:
08 2022
08 2022
Historique:
accepted:
15
02
2022
pubmed:
10
3
2022
medline:
29
7
2022
entrez:
9
3
2022
Statut:
ppublish
Résumé
Consumer wearables and smartphone devices commonly offer an estimate of energy expenditure (EE) to assist in the objective monitoring of physical activity to the general population. Alongside consumers, healthcare professionals and researchers are seeking to utilise these devices for the monitoring of training and improving human health. However, the methods of validation and reporting of EE estimation in these devices lacks rigour, negatively impacting on the ability to make comparisons between devices and provide transparent accuracy. The Towards Intelligent Health and Well-Being Network of Physical Activity Assessment (INTERLIVE) is a joint European initiative of six universities and one industrial partner. The network was founded in 2019 and strives towards developing best-practice recommendations for evaluating the validity of consumer wearables and smartphones. This expert statement presents a best-practice validation protocol for consumer wearables and smartphones in the estimation of EE. The recommendations were developed through (1) a systematic literature review; (2) an unstructured review of the wider literature discussing the potential factors that may introduce bias during validation studies; and (3) evidence-informed expert opinions from members of the INTERLIVE network. The systematic literature review process identified 1645 potential articles, of which 62 were deemed eligible for the final dataset. Based on these studies and the wider literature search, a validation framework is proposed encompassing six key domains for validation: the target population, criterion measure, index measure, testing conditions, data processing and the statistical analysis. The INTERLIVE network recommends that the proposed protocol, and checklists provided, are used to standardise the testing and reporting of the validation of any consumer wearable or smartphone device to estimate EE. This in turn will maximise the potential utility of these technologies for clinicians, researchers, consumers, and manufacturers/developers, while ensuring transparency, comparability, and replicability in validation. PROSPERO ID: CRD42021223508.
Sections du résumé
BACKGROUND
Consumer wearables and smartphone devices commonly offer an estimate of energy expenditure (EE) to assist in the objective monitoring of physical activity to the general population. Alongside consumers, healthcare professionals and researchers are seeking to utilise these devices for the monitoring of training and improving human health. However, the methods of validation and reporting of EE estimation in these devices lacks rigour, negatively impacting on the ability to make comparisons between devices and provide transparent accuracy.
OBJECTIVES
The Towards Intelligent Health and Well-Being Network of Physical Activity Assessment (INTERLIVE) is a joint European initiative of six universities and one industrial partner. The network was founded in 2019 and strives towards developing best-practice recommendations for evaluating the validity of consumer wearables and smartphones. This expert statement presents a best-practice validation protocol for consumer wearables and smartphones in the estimation of EE.
METHODS
The recommendations were developed through (1) a systematic literature review; (2) an unstructured review of the wider literature discussing the potential factors that may introduce bias during validation studies; and (3) evidence-informed expert opinions from members of the INTERLIVE network.
RESULTS
The systematic literature review process identified 1645 potential articles, of which 62 were deemed eligible for the final dataset. Based on these studies and the wider literature search, a validation framework is proposed encompassing six key domains for validation: the target population, criterion measure, index measure, testing conditions, data processing and the statistical analysis.
CONCLUSIONS
The INTERLIVE network recommends that the proposed protocol, and checklists provided, are used to standardise the testing and reporting of the validation of any consumer wearable or smartphone device to estimate EE. This in turn will maximise the potential utility of these technologies for clinicians, researchers, consumers, and manufacturers/developers, while ensuring transparency, comparability, and replicability in validation.
TRIAL REGISTRATION
PROSPERO ID: CRD42021223508.
Identifiants
pubmed: 35260991
doi: 10.1007/s40279-022-01665-4
pii: 10.1007/s40279-022-01665-4
pmc: PMC9325806
doi:
Types de publication
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
1817-1832Informations de copyright
© 2022. The Author(s).
Références
Tankovska H. Fitness Trackers—Statistics and Facts. Statista; 2020. https://www.statista.com/topics/4393/fitness-and-activity-tracker/ . Accessed 24 Feb 2021.
O’Dea S. Smartphone users worldwide 2016–2021. Statista; 2020. https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/ . Accessed 24 Feb 2021.
Bunn JA, Navalta JW, Fountaine CJ, et al. Current state of commercial wearable technology in physical activity monitoring 2015–2017. Int J Exerc Sci. 2018;11:503–15.
pubmed: 29541338
pmcid: 5841672
de Weir VJB. New methods for calculating metabolic rate with special reference to protein metabolism. J Physiol. 1949;109:1–9.
pubmed: 15394301
pmcid: 1392602
doi: 10.1113/jphysiol.1949.sp004363
Levine JA. Measurement of energy expenditure. Public Health Nutr. 2005;8:1123–32.
pubmed: 16277824
doi: 10.1079/PHN2005800
Hills AP, Mokhtar N, Byrne NM. Assessment of physical activity and energy expenditure: an overview of objective measures. Front Nutr. 2014;1:5.
pubmed: 25988109
pmcid: 4428382
doi: 10.3389/fnut.2014.00005
Schoeller DA, Van Santen E. Measurement of energy expenditure in humans by doubly labeled water method. J Appl Physiol. 1982;53:955–9.
pubmed: 6759491
doi: 10.1152/jappl.1982.53.4.955
Düking P, Giessing L, Frenkel MO, et al. Wrist-worn wearables for monitoring heart rate and energy expenditure while sitting or performing light-to-vigorous physical activity: validation study. JMIR Mhealth Uhealth. 2020;8:e16716.
pubmed: 32374274
pmcid: 7240439
doi: 10.2196/16716
Kendall B, Bellovary B, Gothe NP. Validity of wearable activity monitors for tracking steps and estimating energy expenditure during a graded maximal treadmill test. J Sports Sci. 2019;37:42–9.
pubmed: 29863968
doi: 10.1080/02640414.2018.1481723
Murakami H, Kawakami R, Nakae S, et al. Accuracy of 12 wearable devices for estimating physical activity energy expenditure using a metabolic chamber and the doubly labeled water method: validation study. JMIR Mhealth Uhealth. 2019;7:e13938.
pubmed: 31376273
pmcid: 6696858
doi: 10.2196/13938
O’Driscoll R, Turicchi J, Beaulieu K, et al. How well do activity monitors estimate energy expenditure? A systematic review and meta-analysis of the validity of current technologies. Br J Sports Med. 2020;54:332–40.
pubmed: 30194221
Fuller D, Colwell E, Low J, et al. Reliability and validity of commercially available wearable devices for measuring steps, energy expenditure, and heart rate: systematic review. JMIR Mhealth Uhealth. 2020;8:e18694.
pubmed: 32897239
pmcid: 7509623
doi: 10.2196/18694
Welk GJ, McClain J, Ainsworth BE. Protocols for evaluating equivalency of accelerometry-based activity monitors. Med Sci Sport Exerc. 2012;44(1 Suppl 1):S39-49.
doi: 10.1249/MSS.0b013e3182399d8f
Sperlich B, Holmberg H-C. Wearable, yes, but able…? It is time for evidence-based marketing claims! Br J Sports Med. 2017;51:1240.
pubmed: 27986762
doi: 10.1136/bjsports-2016-097295
Welk GJ, Bai Y, Lee J-M, et al. Standardizing analytic methods and reporting in activity monitor validation studies. Med Sci Sports Exerc. 2019;51:1767.
pubmed: 30913159
pmcid: 6693923
doi: 10.1249/MSS.0000000000001966
Keadle SK, Lyden KA, Strath SJ, et al. A framework to evaluate devices that assess physical behavior. Exerc Sport Sci Rev. 2019;47(4):206–14.
pubmed: 31524786
doi: 10.1249/JES.0000000000000206
Mühlen JM, Stang J, Skovgaard EL, et al. Recommendations for determining the validity of consumer wearable heart rate devices: expert statement and checklist of the INTERLIVE Network. Br J Sports Med. 2012;55(14):767–79.
doi: 10.1136/bjsports-2020-103148
Johnston W, Judice PB, García PM, et al. Recommendations for determining the validity of consumer wearable and smartphone step count: expert statement and checklist of the INTERLIVE network. Br J Sports Med. 2021;55(14):780–93.
pubmed: 33361276
doi: 10.1136/bjsports-2020-103147
Bassett DR Jr, Rowlands AV, Trost SG. Calibration and validation of wearable monitors. Med Sci Sports Exerc. 2012;44:S32.
pubmed: 22157772
pmcid: 3273335
doi: 10.1249/MSS.0b013e3182399cf7
Freedson P, Bowles HR, Troiano R, et al. Assessment of physical activity using wearable monitors: recommendations for monitor calibration and use in the field. Med Sci Sport Exerc. 2012;44(1 Suppl 1):S1-4.
doi: 10.1249/MSS.0b013e3182399b7e
Veritas. Covidence Systematic Review Software. Veritas Health Innovation; 2021. www.covidence.org .
Fukagawa NK, Bandini LG, Young JB. Effect of age on body composition and resting metabolic rate. Am J Physiol Metab. 1990;259:E233–8.
Poehlman ET, Goran MI, Gardner AW, et al. Determinants of decline in resting metabolic rate in aging females. Am J Physiol Metab. 1993;264:E450–5.
Van Pelt RE, Jones PP, Davy KP, et al. Regular exercise and the age-related decline in resting metabolic rate in women. J Clin Endocrinol Metab. 1997;82:3208–12.
pubmed: 9329340
Arciero PJ, Goran MI, Poehlman ET. Resting metabolic rate is lower in women than in men. J Appl Physiol. 1993;75:2514–20.
pubmed: 8125870
doi: 10.1152/jappl.1993.75.6.2514
Fitzpatrick TB. The validity and practicality of sun-reactive skin types I through VI. Arch Dermatol. 1988;124:869–71.
pubmed: 3377516
doi: 10.1001/archderm.1988.01670060015008
Lu M-J, Zhong W-H, Liu Y-X, et al. Sample size for assessing agreement between two methods of measurement by Bland−Altman method. Int J Biostat. 2016. https://doi.org/10.1515/ijb-2015-0039 .
doi: 10.1515/ijb-2015-0039
pubmed: 27838682
Carstensen B. Comparing clinical measurement methods: a practical guide. New York: Wiley; 2011.
Butler PJ, Green JA, Boyd IL, et al. Measuring metabolic rate in the field: the pros and cons of the doubly labelled water and heart rate methods. Funct Ecol. 2004;18:168–83.
doi: 10.1111/j.0269-8463.2004.00821.x
Westerterp KR. Doubly labelled water assessment of energy expenditure: principle, practice, and promise. Eur J Appl Physiol. 2017;117:1277–85.
pubmed: 28508113
pmcid: 5486561
doi: 10.1007/s00421-017-3641-x
Davidsson L. Assessment of body composition and total energy expenditure in humans using stable isotope techniques. Vienna: International Atomic Energy Agency; 2009.
DeLany JP, Lovejoy JC. Energy expenditure. Endocrinol Metab Clin N Am. 1996;25:831–46.
doi: 10.1016/S0889-8529(05)70357-1
Delsoglio M, Achamrah N, Berger MM, et al. Indirect calorimetry in clinical practice. J Clin Med. 2019;8:1387.
pmcid: 6780066
doi: 10.3390/jcm8091387
Mtaweh H, Tuira L, Floh AA, et al. Indirect calorimetry: history, technology, and application. Front Pediatr. 2018;6:257.
pubmed: 30283765
pmcid: 6157446
doi: 10.3389/fped.2018.00257
Leonard WR. Laboratory and field methods for measuring human energy expenditure. Am J Hum Biol. 2012;24:372–84.
pubmed: 22419374
doi: 10.1002/ajhb.22260
Kinnunen H, Häkkinen K, Schumann M, et al. Training-induced changes in daily energy expenditure: Methodological evaluation using wrist-worn accelerometer, heart rate monitor, and doubly labeled water technique. PLoS ONE. 2019;14:e0219563.
pubmed: 31291373
pmcid: 6619827
doi: 10.1371/journal.pone.0219563
Murakami H, Kawakami R, Nakae S, et al. Accuracy of wearable devices for estimating total energy expenditure: comparison with metabolic chamber and doubly labeled water method. JAMA Intern Med. 2016;176:702–3.
pubmed: 26999758
doi: 10.1001/jamainternmed.2016.0152
Ramirez-Marrero FA, Smith BA, Sherman WM, et al. Comparison of methods to estimate physical activity and energy expenditure in African American children. Int J Sport Med. 2005;26:363–71.
doi: 10.1055/s-2004-821011
Siddall A, Powell S, Needham-Beck S, et al. Validity of energy expenditure estimation methods during 10 days of military training. Scand J Med Sci Sport. 2019;29:1313–21.
doi: 10.1111/sms.13488
Dannecker K, Sazonova N, Melanson E, et al. A comparison of energy expenditure estimation of several physical activity monitors. Med Sci Sport Exerc. 2013;45:2105–12.
doi: 10.1249/MSS.0b013e318299d2eb
Evenson KR, Goto MM, Furberg RD. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Act. 2015;12:1–22.
doi: 10.1186/s12966-015-0314-1
Montoye AHK, Mitrzyk JR, Molesky MJ. Comparative accuracy of a wrist-worn activity tracker and a smart shirt for physical activity assessment. Meas Phys Educ Exerc Sci. 2017;21:201–11.
doi: 10.1080/1091367X.2017.1331166
Kane NA, Simmons MC, John D, et al. Validity of the Nike plus device during walking and running. Int J Sport Med. 2010;31:101–5.
doi: 10.1055/s-0029-1242810
Nolan M, Mitchell J, Doyle-Baker P. Validity of the Apple iPhone®/iPod Touch® as an accelerometer-based physical activity monitor: a proof-of-concept study. J Phys Act Heal. 2014;11:759–69.
doi: 10.1123/jpah.2011-0336
Maddison R, Gemming L, Monedero J, et al. Quantifying Human Movement Using the Movn Smartphone App: Validation and Field Study. JMIR Mhealth Uhealth. 2017;5(8):e122.
pubmed: 28818819
pmcid: 5579324
doi: 10.2196/mhealth.7167
Xie J, Wen D, Liang L, et al. Evaluating the validity of current mainstream wearable devices in fitness tracking under various physical activities: comparative study. JMIR Mhealth Uhealth. 2018;6:e94.
pubmed: 29650506
pmcid: 5920198
doi: 10.2196/mhealth.9754
Johnson M, Turek J, Dornfeld C, et al. Validity of the Samsung Phone S Health application for assessing steps and energy expenditure during walking and running: Does phone placement matter? Digit Heal. 2016;2:2055207616652747.
Cui Y, Chipchase J, Ichikawa F. A cross culture study on phone carrying and physical personalization. In: International conference on usability and internationalization. Springer, Berlin; 2007. pp. 483–92.
Chowdhury E, Western M, Nightingale T, et al. Assessment of laboratory and daily energy expenditure estimates from consumer multi-sensor physical activity monitors. PLoS ONE. 2017;12:e0171720.
pubmed: 28234979
pmcid: 5325221
doi: 10.1371/journal.pone.0171720
Thiebaud R, Funk M, Patton J, et al. Validity of wrist-worn consumer products to measure heart rate and energy expenditure. Digit Health. 2018;4:2055207618770322.
pubmed: 29942628
pmcid: 6001222
Adam Noah J, Spierer D, Gu J, et al. Comparison of steps and energy expenditure assessment in adults of Fitbit Tracker and Ultra to the Actical and indirect calorimetry. J Med Eng Technol. 2013;37:456–62.
pubmed: 24007317
doi: 10.3109/03091902.2013.831135
Alsubheen S, George A, Baker A, et al. Accuracy of the vivofit activity tracker. J Med Eng Technol. 2016;40:298–306.
pubmed: 27266422
doi: 10.1080/03091902.2016.1193238
Machač S, Procházka M, Radvanský J, et al. Validation of physical activity monitors in individuals with diabetes: energy expenditure estimation by the multisensor SenseWear Armband Pro3 and the step counter Omron HJ-720 against indirect calorimetry during walking. Diabetes Technol Ther. 2013;15:413–8.
pubmed: 23557200
pmcid: 3643221
doi: 10.1089/dia.2012.0235
McArdle WD, Katch FI, Katch VL. Exercise physiology: nutrition, energy, and human performance. New York: Lippincott Williams & Wilkins; 2010.
McClave SA, Spain DA, Skolnick JL, et al. Achievement of steady state optimizes results when performing indirect calorimetry. J Parenter Enter Nutr. 2003;27:16–20.
doi: 10.1177/014860710302700116
Plowman SA, Smith DL. Exercise physiology for health fitness and performance. New York: Lippincott Williams & Wilkins; 2013.
Popp CJ, Tisch JJ, Sakarcan KE, et al. Approximate time to steady-state resting energy expenditure using indirect calorimetry in young, healthy adults. Front Nutr. 2016;3:49.
pubmed: 27857943
pmcid: 5093115
doi: 10.3389/fnut.2016.00049
Borges JH, Langer RD, Cirolini VX, et al. Minimum time to achieve the steady state and optimum abbreviated period to estimate the resting energy expenditure by indirect calorimetry in healthy young adults. Nutr Clin Pract. 2016;31:349–54.
pubmed: 26888859
doi: 10.1177/0884533615627268
Reeves MM, Davies PSW, Bauer J, et al. Reducing the time period of steady state does not affect the accuracy of energy expenditure measurements by indirect calorimetry. J Appl Physiol. 2004;97:130–4.
pubmed: 15020579
doi: 10.1152/japplphysiol.01212.2003
Jones AM, Doust JH. A 1% treadmill grade most accurately reflects the energetic cost of outdoor running. J Sports Sci. 1996;14:321–7.
pubmed: 8887211
doi: 10.1080/02640419608727717
Scott CB, Leighton BH, Ahearn KJ, et al. Aerobic, anaerobic, and excess postexercise oxygen consumption energy expenditure of muscular endurance and strength: 1-set of bench press to muscular fatigue. J Strength Cond Res. 2011;25:903–8.
pubmed: 20703175
doi: 10.1519/JSC.0b013e3181c6a128
Scott CB. Contribution of blood lactate to the energy expenditure of weight training. J Strength Cond Res. 2006;20:404.
pubmed: 16686572
Medbo JI, Mohn A-C, Tabata I, et al. Anaerobic capacity determined by maximal accumulated O2 deficit. J Appl Physiol. 1988;64:50–60.
pubmed: 3356666
doi: 10.1152/jappl.1988.64.1.50
Speakman J. Doubly labelled water: theory and practice. New York: Springer Science & Business Media; 1997.
Zaki R, Bulgiba A, Ismail R, et al. Statistical methods used to test for agreement of medical instruments measuring continuous variables in method comparison studies: a systematic review. PLoS ONE. 2012;7:e37908.
pubmed: 22662248
pmcid: 3360667
doi: 10.1371/journal.pone.0037908
Ludbrook J. Statistical techniques for comparing measurers and methods of measurement: a critical review. Clin Exp Pharmacol Physiol. 2002;29:527–36.
pubmed: 12060093
doi: 10.1046/j.1440-1681.2002.03686.x
Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;327:307–10.
doi: 10.1016/S0140-6736(86)90837-8
De Myttenaere A, Golden B, Le Grand B, et al. Mean absolute percentage error for regression models. Neurocomputing. 2016;192:38–48.
doi: 10.1016/j.neucom.2015.12.114
Bennie JA, Shakespear-Druery J, De Cocker K. Muscle-strengthening Exercise Epidemiology: a New Frontier in Chronic Disease Prevention. Sport Med. 2020;6:1–8.
Ferro Magosso R, Campanholi Neto J, de Carli JPC, et al. A Review of Ergogenesis and Effect of Training Variables on Energy Expenditure in Resistance Training Exercises. J Exerc Physiol Online. 2017;20(2).
Scott C. Misconceptions about aerobic and anaerobic energy expenditure. J Int Soc Sports Nutr. 2005;2:1–6.
doi: 10.1186/1550-2783-2-2-32