Keeping Pace with Wearables: A Living Umbrella Review of Systematic Reviews Evaluating the Accuracy of Consumer Wearable Technologies in Health Measurement.
Journal
Sports medicine (Auckland, N.Z.)
ISSN: 1179-2035
Titre abrégé: Sports Med
Pays: New Zealand
ID NLM: 8412297
Informations de publication
Date de publication:
30 Jul 2024
30 Jul 2024
Historique:
accepted:
10
07
2024
medline:
31
7
2024
pubmed:
31
7
2024
entrez:
30
7
2024
Statut:
aheadofprint
Résumé
Consumer wearable technologies have become ubiquitous, with clinical and non-clinical populations leveraging a variety of devices to quantify various aspects of health and wellness. However, the accuracy with which these devices measure biometric outcomes such as heart rate, sleep and physical activity remains unclear. To conduct a 'living' (i.e. ongoing) evaluation of the accuracy of consumer wearable technologies in measuring various physiological outcomes. A systematic search of the literature was conducted in the following scientific databases: MEDLINE via PubMed, Embase, Cinahl and SPORTDiscus via EBSCO. The inclusion criteria required systematic reviews or meta-analyses that evaluated the validation of consumer wearable devices against accepted reference standards. In addition to publication details, review protocol, device specifics and a summary of the authors' results, we extracted data on mean absolute percentage error (MAPE), pooled absolute bias, intraclass correlation coefficients (ICCs) and mean absolute differences. Of 904 identified studies through the initial search, 24 systematic reviews met our inclusion criteria; these systematic reviews included 249 non-duplicate validation studies of consumer wearable devices involving 430,465 participants (43% female). Of the commercially available wearable devices released to date, approximately 11% have been validated for at least one biometric outcome. However, because a typical device can measure a multitude of biometric outcomes, the number of validation studies conducted represents just 3.5% of the total needed for a comprehensive evaluation of these devices. For heart rate, wearables showed a mean bias of ± 3%. In arrhythmia detection, wearables exhibited a pooled sensitivity and specificity of 100% and 95%, respectively. For aerobic capacity, wearables significantly overestimated VO While consumer wearables show promise in health monitoring, a conclusive assessment of their accuracy is impeded by pervasive heterogeneity in research outcomes and methodologies. There is a need for standardised validation protocols and collaborative industry partnerships to enhance the reliability and practical applicability of wearable technology assessments. CRD42023402703.
Sections du résumé
BACKGROUND
BACKGROUND
Consumer wearable technologies have become ubiquitous, with clinical and non-clinical populations leveraging a variety of devices to quantify various aspects of health and wellness. However, the accuracy with which these devices measure biometric outcomes such as heart rate, sleep and physical activity remains unclear.
OBJECTIVE
OBJECTIVE
To conduct a 'living' (i.e. ongoing) evaluation of the accuracy of consumer wearable technologies in measuring various physiological outcomes.
METHODS
METHODS
A systematic search of the literature was conducted in the following scientific databases: MEDLINE via PubMed, Embase, Cinahl and SPORTDiscus via EBSCO. The inclusion criteria required systematic reviews or meta-analyses that evaluated the validation of consumer wearable devices against accepted reference standards. In addition to publication details, review protocol, device specifics and a summary of the authors' results, we extracted data on mean absolute percentage error (MAPE), pooled absolute bias, intraclass correlation coefficients (ICCs) and mean absolute differences.
RESULTS
RESULTS
Of 904 identified studies through the initial search, 24 systematic reviews met our inclusion criteria; these systematic reviews included 249 non-duplicate validation studies of consumer wearable devices involving 430,465 participants (43% female). Of the commercially available wearable devices released to date, approximately 11% have been validated for at least one biometric outcome. However, because a typical device can measure a multitude of biometric outcomes, the number of validation studies conducted represents just 3.5% of the total needed for a comprehensive evaluation of these devices. For heart rate, wearables showed a mean bias of ± 3%. In arrhythmia detection, wearables exhibited a pooled sensitivity and specificity of 100% and 95%, respectively. For aerobic capacity, wearables significantly overestimated VO
CONCLUSIONS
CONCLUSIONS
While consumer wearables show promise in health monitoring, a conclusive assessment of their accuracy is impeded by pervasive heterogeneity in research outcomes and methodologies. There is a need for standardised validation protocols and collaborative industry partnerships to enhance the reliability and practical applicability of wearable technology assessments.
PROSPERO ID
UNASSIGNED
CRD42023402703.
Identifiants
pubmed: 39080098
doi: 10.1007/s40279-024-02077-2
pii: 10.1007/s40279-024-02077-2
doi:
Types de publication
Journal Article
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Subventions
Organisme : Science Foundation Ireland
ID : 22/NCF/FD/10949
Pays : Ireland
Informations de copyright
© 2024. The Author(s).
Références
Lupton D. The quantified self: a sociology of self-tracking. Sociol Health Illn. 2016;39:1557–71.
doi: 10.1111/1467-9566.12617
Piwek L, Ellis DA, Andrews S, Joinson A. The rise of consumer health wearables: promises and barriers. PLoS Med. 2016;13(2): e1001953.
pubmed: 26836780
pmcid: 4737495
doi: 10.1371/journal.pmed.1001953
GVR. Wearable technology market size, share & trends analysis report by product (head & eyewear, wristwear), by application (consumer electronics, healthcare), by region (Asia Pacific, Europe), And Segment Forecasts, 2023–2030: Grand View Research; 2023. Report No.: 978-1-68038-165-8.
ACSM. Wearable technology named top fitness trend for 2024. 2024 [cited; https://www.acsm.org/education-resources/trending-topics-resources/acsm-fitness-trends .
Carpenter A, Frontera A. Smart-watches: a potential challenger to the implantable loop recorder? Europace. 2016;18(6):791–3.
pubmed: 26847074
doi: 10.1093/europace/euv427
Jia Y, Wang W, Wen D, Liang L, Gao L, Lei J. Perceived user preferences and usability evaluation of mainstream wearable devices for health monitoring. PeerJ. 2018;6: e5350.
pubmed: 30065893
pmcid: 6064199
doi: 10.7717/peerj.5350
Hickey AM, Freedson PS. Utility of consumer physical activity trackers as an intervention tool in cardiovascular disease prevention and treatment. Prog Cardiovasc Dis. 2016;58(6):613–9.
pubmed: 26943981
doi: 10.1016/j.pcad.2016.02.006
Radin JM, Wineinger NE, Topol EJ, Steinhubl SR. Harnessing wearable device data to improve state-level real-time surveillance of influenza-like illness in the USA: a population-based study. Lancet Digit Health. 2020;2(2):e85–93.
pubmed: 33334565
pmcid: 8048388
doi: 10.1016/S2589-7500(19)30222-5
Perez MV, Mahaffey KW, Hedlin H, Rumsfeld JS, Garcia A, Ferris T, et al. Large-scale assessment of a smartwatch to identify atrial fibrillation. N Engl J Med. 2019;381(20):1909–17.
pubmed: 31722151
pmcid: 8112605
doi: 10.1056/NEJMoa1901183
Kimura N, Aso Y, Yabuuchi K, Ishibashi M, Hori D, Sasaki Y, et al. Association of modifiable lifestyle factors with cortical amyloid burden and cerebral glucose metabolism in older adults with mild cognitive impairment. JAMA Netw Open. 2020;3(6): e205719.
pubmed: 32515796
pmcid: 7284299
doi: 10.1001/jamanetworkopen.2020.5719
Shilaih M, Goodale BM, Falco L, Kübler F, De Clerck V, Leeners B. Modern fertility awareness methods: wrist wearables capture the changes in temperature associated with the menstrual cycle. Biosci Rep. 2018;38(6): BSR20171279.
pubmed: 29175999
pmcid: 6265623
doi: 10.1042/BSR20171279
clinicaltrials.gov. Clinicaltrials.gov search results. 2023 [cited; https://clinicaltrials.gov/search?term=fitbit&aggFilters=status:rec .
Shiffman S, Stone AA, Hufford MR. Ecological momentary assessment. Annu Rev Clin Psychol. 2008;4:1–32.
pubmed: 18509902
doi: 10.1146/annurev.clinpsy.3.022806.091415
Zapata-Lamana R, Lalanza JF, Losilla JM, Parrado E, Capdevila L. mHealth technology for ecological momentary assessment in physical activity research: a systematic review. PeerJ. 2020;8: e8848.
pubmed: 32257648
pmcid: 7103204
doi: 10.7717/peerj.8848
Fuller D, Colwell E, Low J, Orychock K, Tobin MA, Simango B, et al. Reliability and validity of commercially available wearable devices for measuring steps, energy expenditure, and heart rate: systematic review. JMIR Mhealth Uhealth. 2020;8(9): e18694.
pubmed: 32897239
pmcid: 7509623
doi: 10.2196/18694
Keogh A, Argent R, Doherty C, Duignan C, Fennelly O, Purcell C, et al. Breaking down the digital fortress: the unseen challenges in healthcare technology—lessons learned from 10 years of research. Sensors. 2024;24(12):3780.
pubmed: 38931564
pmcid: 11207951
doi: 10.3390/s24123780
Huhn S, Axt M, Gunga H-C, Maggioni MA, Munga S, Obor D, et al. The impact of wearable technologies in health research: scoping review. JMIR Mhealth Uhealth. 2022;10(1): e34384.
pubmed: 35076409
pmcid: 8826148
doi: 10.2196/34384
Dunn J, Runge R, Snyder M. Wearables and the medical revolution. Per Med. 2018;15(5):429–48.
pubmed: 30259801
doi: 10.2217/pme-2018-0044
Gualtieri L, Rosenbluth S, Phillips J. Can a free wearable activity tracker change behavior? The impact of trackers on adults in a physician-led wellness group. JMIR Res Protoc. 2016;5(4): e237.
pubmed: 27903490
pmcid: 5156822
doi: 10.2196/resprot.6534
Montoye AHK, Mitrzyk JR, Molesky MJ. Comparative accuracy of a wrist-worn activity tracker and a smart shirt for physical activity assessment. Meas Phys Educ Exerc Sci. 2017;21(4):201–11.
doi: 10.1080/1091367X.2017.1331166
Evenson KR, Goto MM, Furberg RD. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Activity. 2015;12(1):159.
doi: 10.1186/s12966-015-0314-1
Toth LP, Park S, Springer CM, Feyerabend MD, Steeves JA, Bassett DR. Video-recorded validation of wearable step counters under free-living conditions. Med Sci Sports Exerc. 2018;50(6):1315–22.
pubmed: 29381649
doi: 10.1249/MSS.0000000000001569
Kim J, Campbell AS, de Ávila BE-F, Wang J. Wearable biosensors for healthcare monitoring. Nat Biotechnol. 2019;37(4):389–406.
pubmed: 30804534
pmcid: 8183422
doi: 10.1038/s41587-019-0045-y
Ash GI, Stults-Kolehmainen M, Busa MA, Gaffey AE, Angeloudis K, Muniz-Pardos B, et al. Establishing a global standard for wearable devices in sport and exercise medicine: perspectives from academic and industry stakeholders. Sports Med. 2021;51(11):2237–50.
pubmed: 34468950
pmcid: 8666971
doi: 10.1007/s40279-021-01543-5
Argent R, Hetherington-Rauth M, Stang J, Tarp J, Ortega FB, Molina-Garcia P, et al. Recommendations for determining the validity of consumer wearables and smartphones for the estimation of energy expenditure: expert statement and checklist of the INTERLIVE network. Sports Med. 2022;52(8):1817–32.
pubmed: 35260991
pmcid: 9325806
doi: 10.1007/s40279-022-01665-4
Johnston W, Judice PB, Molina García P, Mühlen JM, Lykke Skovgaard E, Stang J, et al. Recommendations for determining the validity of consumer wearable and smartphone step count: expert statement and checklist of the INTERLIVE network. Br J Sports Med. 2021;55(14):780–93.
pubmed: 33361276
doi: 10.1136/bjsports-2020-103147
Mühlen JM, Stang J, Lykke Skovgaard E, Judice PB, Molina-Garcia P, Johnston W, et al. Recommendations for determining the validity of consumer wearable heart rate devices: expert statement and checklist of the INTERLIVE Network. Br J Sports Med. 2021;55(14):767–79.
pubmed: 33397674
doi: 10.1136/bjsports-2020-103148
Düking P, Stammel C, Sperlich B, Sutehall S, Muniz-Pardos B, Lima G, et al. Necessary steps to accelerate the integration of wearable sensors into recreation and competitive sports. Curr Sports Med Rep. 2018;17(6):178–82.
pubmed: 29889145
doi: 10.1249/JSR.0000000000000495
Simmonds M, Elliott JH, Synnot A, Turner T. Living systematic reviews. Methods Mol Biol. 2022;2345:121–34.
pubmed: 34550587
doi: 10.1007/978-1-0716-1566-9_7
Cochrane. Living systematic reviews. 2023 [cited; https://community.cochrane.org/review-development/resources/living-systematic-reviews#:~:text=Living%20Evidence%20Network-,What%20is%20a%20living%20systematic%20review%3F,evidence%20as%20it%20becomes%20available .
Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J (Online). 2009;339(7716):332–6.
Drucker AM, Fleming P, Chan AW. Research techniques made simple: assessing risk of bias in systematic reviews. J Invest Dermatol. 2016;136(11):e109–14.
pubmed: 27772550
doi: 10.1016/j.jid.2016.08.021
Belani S, Wahood W, Hardigan P, Placzek AN, Ely S. Accuracy of detecting atrial fibrillation: a systematic review and meta-analysis of wrist-worn wearable technology. Cureus. 2021;13(12): e20362.
pubmed: 35036196
pmcid: 8752409
Board EM, Ispoglou T, Ingle L. Validity of telemetric-derived measures of heart rate variability: a systematic review. J Exerc Physiol Online. 2016;19(6):64–84.
Byrne J, Lynch S, Shipp A, Tran B, Mohan S, Reindel K. Investigating the accuracy of wheelchair push counts measured by fitness watches: a systematic review. Cureus. 2023;15(9): e45322.
pubmed: 37849605
pmcid: 10577091
Chevance G, Golaszewski NM, Tipton E, Hekler EB, Buman M, Welk GJ, et al. Accuracy and precision of energy expenditure, heart rate, and steps measured by combined-sensing fitbits against reference measures: systematic review and meta-analysis. JMIR Mhealth Uhealth. 2022;10(4): e35626.
pubmed: 35416777
pmcid: 9047731
doi: 10.2196/35626
Evenson KR, Goto MM, Furberg RD. Systematic review of the validity and reliability of consumer-wearable activity trackers. Int J Behav Nutr Phys Act. 2015;12:1–22.
doi: 10.1186/s12966-015-0314-1
Feehan LM, Geldman J, Sayre EC, Park C, Ezzat AM, Yoo JY, et al. Accuracy of fitbit devices: systematic review and narrative syntheses of quantitative data. JMIR Mhealth Uhealth. 2018;6(8): e10527.
pubmed: 30093371
pmcid: 6107736
doi: 10.2196/10527
Georgiou K, Larentzakis AV, Khamis NN, Alsuhaibani GI, Alaska YA, Giallafos EJ. Can wearable devices accurately measure heart rate variability? A systematic review. Folia Med. 2018;60(1):7–20.
Germini F, Noronha N, Debono VB, Philip BA, Pete D, Navarro T, et al. Accuracy and acceptability of wrist-wearable activity-tracking devices: systematic review of the literature. J Med Internet Res. 2022;24(1): e30791.
pubmed: 35060915
pmcid: 8817215
doi: 10.2196/30791
Giebel GD, Gissel C. Accuracy of mHealth devices for atrial fibrillation screening: systematic review. JMIR Mhealth Uhealth. 2019;7(6): e13641.
pubmed: 31199337
pmcid: 6598422
doi: 10.2196/13641
Haghayegh S, Khoshnevis S, Smolensky MH, Diller KR, Castriotta RJ. Accuracy of wristband fitbit models in assessing sleep: systematic review and meta-analysis. J Med Internet Res. 2019;21(11): e16273.
pubmed: 31778122
pmcid: 6908975
doi: 10.2196/16273
Henriksen A, Johansson J, Hartvigsen G, Grimsgaard S, Hopstock L. Measuring physical activity using triaxial wrist worn polar activity trackers: a systematic review. Int J Exerc Sci. 2020;13(4):438–54.
pubmed: 32509122
pmcid: 7241625
Hermans ANL, Gawalko M, Dohmen L, van der Velden RMJ, Betz K, Duncker D, et al. Mobile health solutions for atrial fibrillation detection and management: a systematic review. Clin Res Cardiol. 2022;111(5):479–91.
pubmed: 34549333
doi: 10.1007/s00392-021-01941-9
Irwin C, Gary R. Systematic review of Fitbit Charge 2 validation studies for exercise tracking. Transl J Am Coll Sports Med. 2022;7(4):1–7.
pubmed: 36711436
pmcid: 9881599
Kenyon A, McEvoy M, Sprod J, Maher C. Validity of pedometers in people with physical disabilities: a systematic review. Arch Phys Med Rehabil. 2013;94(6):1161–70.
pubmed: 23201318
doi: 10.1016/j.apmr.2012.11.030
Koerber D, Khan S, Shamsheri T, Kirubarajan A, Mehta S. Accuracy of heart rate measurement with wrist-worn wearable devices in various skin tones: a systematic review. J Racial Ethnic Health Disparities. 2022;10:2676–84.
doi: 10.1007/s40615-022-01446-9
Leung W, Case L, Sung MC, Jung J. A meta-analysis of Fitbit devices: same company, different models, different validity evidence. J Med Eng Technol. 2022;46(2):102–15.
pubmed: 34881682
doi: 10.1080/03091902.2021.2006350
Lopez Perales CR, Van Spall HGC, Maeda S, Jimenez A, Laţcu DG, Milman A, et al. Mobile health applications for the detection of atrial fibrillation: a systematic review. Europace. 2021;23(1):11–28.
pubmed: 33043358
doi: 10.1093/europace/euaa139
Molina-Garcia P, Notbohm HL, Schumann M, Argent R, Hetherington-Rauth M, Stang J, et al. Validity of estimating the maximal oxygen consumption by consumer wearables: a systematic review with meta-analysis and expert statement of the INTERLIVE network. Sports Med. 2022;52(7):1577–97.
pubmed: 35072942
pmcid: 9213394
doi: 10.1007/s40279-021-01639-y
Nazarian S, Lam K, Darzi A, Ashrafian H. Diagnostic accuracy of smartwatches for the detection of cardiac arrhythmia: systematic review and meta-analysis. J Med Internet Res. 2021;23(8): e28974.
pubmed: 34448706
pmcid: 8433941
doi: 10.2196/28974
O’Driscoll R, Turicchi J, Beaulieu K, Scott S, Matu J, Deighton K, et al. How well do activity monitors estimate energy expenditure? A systematic review and meta-analysis of the validity of current technologies. Br J Sports Med. 2020;54(6):332–40.
pubmed: 30194221
doi: 10.1136/bjsports-2018-099643
Schyvens AM, Van Oost NC, Aerts JM, Masci F, Peters B, Neven A, et al. Accuracy of Fitbit Charge 4, Garmin Vivosmart 4, and WHOOP versus polysomnography: systematic review. JMIR Mhealth Uhealth. 2024;27(12): e52192.
doi: 10.2196/52192
Windisch P, Schröder C, Förster R, Cihoric N, Zwahlen DR. Accuracy of the Apple Watch oxygen saturation measurement in adults: a systematic review. Cureus. 2023;15(2): e35355.
pubmed: 36974257
pmcid: 10039641
Zhang Y, Weaver RG, Armstrong B, Burkart S, Zhang S, Beets MW. Validity of wrist-worn photoplethysmography devices to measure heart rate: a systematic review and meta-analysis. J Sports Sci. 2020;38(17):2021–34.
pubmed: 32552580
doi: 10.1080/02640414.2020.1767348
Kokkinos P, Faselis C, Samuel IBH, Pittaras A, Doumas M, Murphy R, et al. Cardiorespiratory fitness and mortality risk across the spectra of age, race, and sex. J Am Coll Cardiol. 2022;80(6):598–609.
pubmed: 35926933
doi: 10.1016/j.jacc.2022.05.031
Mandsager K, Harb S, Cremer P, Phelan D, Nissen SE, Jaber W. Association of cardiorespiratory fitness with long-term mortality among adults undergoing exercise treadmill testing. JAMA Netw Open. 2018;1(6): e183605.
pubmed: 30646252
pmcid: 6324439
doi: 10.1001/jamanetworkopen.2018.3605
Spaccarotella C, Polimeni A, Mancuso C, Pelaia G, Esposito G, Indolfi C. Assessment of non-invasive measurements of oxygen saturation and heart rate with an Apple Smartwatch: comparison with a standard pulse oximeter. J Clin Med. 2022;11(6):1467.
pubmed: 35329793
pmcid: 8951323
doi: 10.3390/jcm11061467
Brownlee M. Apple Watch Series 9 & Ultra 2: What Are We Waiting For?! 2023 [cited 13/10/23]; https://www.youtube.com/watch?v=oNCs4C2SMjo .
Denny JC, Rutter JL, Goldstein DB, Philippakis A, Smoller JW, Jenkins G, et al. The “all of us” research program. N Engl J Med. 2019;381(7):668–76.
pubmed: 31412182
doi: 10.1056/NEJMsr1809937
Stamatakis E, Ahmadi MN, Gill JMR, Thøgersen-Ntoumani C, Gibala MJ, Doherty A, et al. Association of wearable device-measured vigorous intermittent lifestyle physical activity with mortality. Nat Med. 2022;28(12):2521–9.
pubmed: 36482104
pmcid: 9800274
doi: 10.1038/s41591-022-02100-x
Thryve. 2023 [cited; Available from: https://thryve.health .
Fitrockr. fitrockr health solutions. 2023.
Fitabase. 2023.
Labfront. 2023 [cited; Available from: https://www.labfront.com .
Doherty C. YouTube Channel: @CailbheDoherty. 2024. https://www.youtube.com/channel/UCxet-B3eDU6naEo68SbnMMA .
Statista. Statistics report on wearables; 2023.
Belbasis L, Bellou V, Ioannidis JPA. Conducting umbrella reviews. BMJ Med. 2022;1(1): e000071.
pubmed: 36936579
pmcid: 9951359
doi: 10.1136/bmjmed-2021-000071