Tools and frameworks for evaluating the implementation of learning health systems: a scoping review.
Evaluation
Healthcare improvement
Implementation science
Learning health systems
Journal
Health research policy and systems
ISSN: 1478-4505
Titre abrégé: Health Res Policy Syst
Pays: England
ID NLM: 101170481
Informations de publication
Date de publication:
06 Aug 2024
06 Aug 2024
Historique:
received:
12
03
2024
accepted:
03
07
2024
medline:
7
8
2024
pubmed:
7
8
2024
entrez:
6
8
2024
Statut:
epublish
Résumé
Despite increased interest in learning health systems (LHS), a paucity of guidance and tools for evaluating LHS implementation exists. To address this, we aim to undertake a scoping review on existing tools and evaluation of exemplars of LHS implementation. We conducted a scoping review of peer-reviewed studies within Scopus, EMBASE, MEDLINE, and MEDLINE in-process that described (1) the evaluation of the implementation of an operating LHS or (2) the development of a framework or tool to facilitate this evaluation. Anima, basic research, abstracts, non-English language articles, and publications before 2018 were excluded. All study designs were considered. From 1300 studies initially identified, 4 were eligible, revealing three tools with nine implementation evaluation examples. The identified tools shared constructs which were evaluated, including: Stakeholders, Data, Research Evidence, Implementation, and Sociotechnical Infrastructure. However, there was divergence in evaluation methodology. Tools ranged from a five-point numerical rating system for process maturity with a radar chart called the Network Maturity Grid (NMG); the Kaiser Permanente Washington (KPWA) LHS Logic Model, which provides a broad list of constructs and sample measures relevant to LHS operations; and finally LADDERS, a simple tool or form-based template designed for consistent evaluation over time. The NMG tool was the most mature in terms of adaptation and adoption. Notably, two (NMG and the KPWA LHS Logic Model) out of three tools conceptualized the LHS as a suite of processes and devised tools were processes that linked these constructs. The evaluation of LHS implementation remains an under explored area of investigation, as this scoping review found only three tools for LHS implementation evaluation. Our findings indicate a need for further empirical research in this area and suggest early consensus in constructs that need to be considered during evaluation.
Identifiants
pubmed: 39107779
doi: 10.1186/s12961-024-01179-7
pii: 10.1186/s12961-024-01179-7
doi:
Types de publication
Journal Article
Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
95Informations de copyright
© 2024. The Author(s).
Références
Institute of Medicine (IOM). The learning healthcare system: workshop summary. Washington, DC; 2007.
Enticott J, Johnson A, Teede H. Learning health systems using data to drive healthcare improvement and impact: a systematic review. BMC Health Serv Res. 2021;21(1):200.
doi: 10.1186/s12913-021-06215-8
pubmed: 33663508
pmcid: 7932903
Menear M, Blanchette MA, Demers-Payette O, Roy D. A framework for value-creating learning health systems. Health Res Policy Syst. 2019;17(1):79.
doi: 10.1186/s12961-019-0477-3
pubmed: 31399114
pmcid: 6688264
Bull C, Teede H, Watson D, Callander EJ. Selecting and implementing patient-reported outcome and experience measures to assess health system performance. JAMA Health Forum. 2022;3(4): e220326.
doi: 10.1001/jamahealthforum.2022.0326
pubmed: 36218960
Wells BJ, Downs SM, Ostasiewski B. Using electronic health records for the learning health system: creation of a diabetes research registry. JMIR Med Inform. 2022;10(9): e39746.
doi: 10.2196/39746
pubmed: 36149742
pmcid: 9547333
Chen A, Chen DO. Simulation of a machine learning enabled learning health system for risk prediction using synthetic patient data. Sci Rep. 2022;12(1):17917.
doi: 10.1038/s41598-022-23011-4
pubmed: 36289292
pmcid: 9606301
Somerville M, Cassidy C, Curran JA, Johnson C, Sinclair D, Elliott RA. Implementation strategies and outcome measures for advancing learning health systems: a mixed methods systematic review. Health Research Policy and Systems. 2023;21(1):120.
doi: 10.1186/s12961-023-01071-w
pubmed: 38012681
pmcid: 10680228
Holdsworth LM, Safaeinili N, Winget M, Lorenz KA, Lough M, Asch S, et al. Adapting rapid assessment procedures for implementation research using a team-based approach to analysis: a case example of patient quality and safety interventions in the ICU. Implement Sci. 2020;15(1):12.
doi: 10.1186/s13012-020-0972-5
pubmed: 32087724
pmcid: 7036173
Fung-Kee-Fung M, Maziak DE, Pantarotto JR, Smylie J, Taylor L, Timlin T, et al. Regional process redesign of lung cancer care: a learning health system pilot project. Curr Oncol. 2018;25(1):59–66.
doi: 10.3747/co.25.3719
pubmed: 29507485
pmcid: 5832278
Smith SK, Rowe K, Abernethy AP. Use of an electronic patient-reported outcome measurement system to improve distress management in oncology. Palliat Support Care. 2014;12(1):69–73.
doi: 10.1017/S1478951513000345
pubmed: 24128592
Vo MT, Uratsu CS, Estacio KR, Altschuler A, Kim E, Alexeeff SE, et al. Prompting patients with poorly controlled diabetes to identify visit priorities before primary care visits: a pragmatic cluster randomized trial. J Gen Intern Med. 2019;34(6):831–8.
doi: 10.1007/s11606-018-4756-4
pubmed: 30746642
pmcid: 6544732
Pomare C, Mahmoud Z, Vedovi A, Ellis LA, Knaggs G, Smith CL, et al. Learning health systems: a review of key topic areas and bibliometric trends. Learn Health Syst. 2022;6(1): e10265.
doi: 10.1002/lrh2.10265
pubmed: 35036549
Ellis LA, Sarkies M, Churruca K, Dammery G, Meulenbroeks I, Smith CL, et al. The science of learning health systems: scoping review of empirical research. JMIR Med Inform. 2022;10(2): e34907.
doi: 10.2196/34907
pubmed: 35195529
pmcid: 8908194
Enticott JC, Melder A, Johnson A, Jones A, Shaw T, Keech W, et al. A learning health system framework to operationalize health data to improve quality care: an australian perspective. Front Med (Lausanne). 2021;8:730021–730021.
doi: 10.3389/fmed.2021.730021
pubmed: 34778291
Safaeinili N, Brown-Johnson C, Shaw JG, Mahoney M, Winget M. CFIR simplified: Pragmatic application of and adaptations to the Consolidated Framework for Implementation Research (CFIR) for evaluation of a patient-centered care transformation within a learning health system. Learn Health Syst. 2020;4(1): e10201.
doi: 10.1002/lrh2.10201
pubmed: 31989028
Cadilhac DA, Bravata DM, Bettger JP, Mikulik R, Norrving B, Uvere EO, et al. Stroke learning health systems: a topical narrative review with case examples. Stroke. 2023;54(4):1148–59.
doi: 10.1161/STROKEAHA.122.036216
pubmed: 36715006
pmcid: 10050099
Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
doi: 10.1080/1364557032000119616
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implementation Sci. 2010;5(1):69.
doi: 10.1186/1748-5908-5-69
Alison O’Mara-Eves, James D. Thomas, John McNaught, Makoto Miwa, Sophia Ananiadou. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. {’id’: ‘ https://openalex.org/S82452678 ’, ‘issn_l’: ‘2046–4053’, ‘issn’: [’2046–4053’], ‘display_name’: ‘Systematic Reviews’, ‘publisher’: ‘Springer Science+Business Media’, ‘type’: ‘journal’, ‘url’: ‘ https://systematicreviewsjournal.biomedcentral.com/track/pdf/101186/s13643-020-01542-z ’, ‘is_oa’: True, ‘version’: ‘publishedVersion’, ‘license’: ’cc-by’}. 2021 Jan 26;
Lannon C, Schuler CL, Seid M, Provost LP, Fuller S, Purcell D, et al. A maturity grid assessment tool for learning networks. Learn Health Syst. 2021;5(2): e10232.
doi: 10.1002/lrh2.10232
pubmed: 33889737
Allen C, Coleman K, Mettert K, Lewis C, Westbrook E, Lozano P. A roadmap to operationalize and evaluate impact in a learning health system. Learn Health Syst. 2021;5(4): e10258.
doi: 10.1002/lrh2.10258
pubmed: 34667878
pmcid: 8512726
Meissner P. LADDERS: A dynamic paradigm for planning, implementing, and evaluating sustainable change in learning health systems. Learn Health Syst. 2018;2(3): e10058.
doi: 10.1002/lrh2.10058
pubmed: 31245586
pmcid: 6508793
Bailes AF, Kean J, Gross PH, Narayanan U, Noritz G, Hurvitz E, et al. The cerebral palsy research network: Building a learning health network for cerebral palsy. JPRM. 2021;14(2):161–71.
Bernal-Delgado E, Estupiñán-Romero F. A data infrastructure for the assessment of health care performance: lessons from the BRIDGE-health project. Archives of Public Health. 2018;76(1):6.
doi: 10.1186/s13690-017-0245-1
pubmed: 29416858
pmcid: 5784587
Embedded researchers’ purpose and practice: Current perspectives from Australia. 2024. https://doi.org/10.1002/hpm.3318
Fisher CM, Sangadji R, Mwangi DN. Using bottom-up evaluation to build community practice-based evidence and strengthen community-engaged research. J Community Pract. 2023;31(3–4):410–28.
doi: 10.1080/10705422.2023.2276350
Sabatier PA. Top-down and bottom-up approaches to implementation research: a critical analysis and suggested synthesis. J Publ Policy. 1986;6(1):21–48.
doi: 10.1017/S0143814X00003846
Stewart GL, Manges KA, Ward MM. Empowering sustained patient safety: the benefits of combining top-down and bottom-up approaches. J Nurs Care Qual. 2015;30(3):240.
doi: 10.1097/NCQ.0000000000000103
pubmed: 25479238
Colldén C, Hellström A. From, “Invented here” to “Use it everywhere!”: A Learning health system from bottom and/or top? Learning Health Systems. 2022;6(3): e10307.
doi: 10.1002/lrh2.10307
pubmed: 35860319
pmcid: 9284931
Grimshaw JM, Patey AM, Kirkham KR, Hall A, Dowling SK, Rodondi N, et al. De-implementing wisely: developing the evidence base to reduce low-value care. BMJ Qual Saf. 2020;29(5):409–17.
doi: 10.1136/bmjqs-2019-010060
pubmed: 32029572
pmcid: 7229903
Carroll C, Patterson M, Wood S, Booth A, Rick J, Balain S. A conceptual framework for implementation fidelity. Implement Sci. 2007;2(1):40.
doi: 10.1186/1748-5908-2-40
pubmed: 18053122
pmcid: 2213686
Guerbaai RA, DeGeest S, Popejoy LL, Simon M, Wellens NIH, Denhaerynck K, et al. Evaluating the implementation fidelity to a successful nurse-led model (INTERCARE) which reduced nursing home unplanned hospitalisations. BMC Health Serv Res. 2023;23(1):138.
doi: 10.1186/s12913-023-09146-8
pubmed: 36759902
pmcid: 9910256
Huckfeldt PJ, Kane RL, Yang Z, Engstrom G, Tappen R, Rojido C, et al. Degree of implementation of the interventions to reduce acute care transfers (INTERACT) quality improvement program associated with number of hospitalizations. J Am Geriatr Soc. 2018;66(9):1830–7.
doi: 10.1111/jgs.15476
pubmed: 30094818
pmcid: 6156928