Development and psychometric evaluation of the Implementation Support Competencies Assessment.
Journal
Implementation science : IS
ISSN: 1748-5908
Titre abrégé: Implement Sci
Pays: England
ID NLM: 101258411
Informations de publication
Date de publication:
06 Aug 2024
06 Aug 2024
Historique:
received:
11
04
2024
accepted:
01
08
2024
medline:
7
8
2024
pubmed:
7
8
2024
entrez:
6
8
2024
Statut:
epublish
Résumé
Implementation support practitioners (ISPs) are professionals that support others to implement evidence-informed practices, programs, and policies in various service delivery settings to achieve population outcomes. Measuring the use of competencies by ISPs provides a unique opportunity to assess an understudied facet of implementation science-how knowledge, attitudes, and skills used by ISPs affects sustainable change in complicated and complex service systems. This study describes the development and validation of a measure-the Implementation Support Competencies Assessment (ISCA)-that assesses implementation support competencies, with versatile applications across service contexts. Recently developed practice guide materials included operationalizations of core competencies for ISPs across three domains: co-creation and engagement, ongoing improvement, and sustaining change. These operationalizations, in combination with recent empirical and conceptual work, provided an initial item pool and foundation on which to advance measurement development, largely from a confirmatory perspective (as opposed to exploratory). The measure was further refined through modified cognitive interviewing with three highly experienced ISPs and pilot-testing with 39 individuals enrolled in a university-based certificate program in implementation practice. To recruit a sample for validation analyses, we leveraged a listserv of nearly 4,000 individuals who have registered for or expressed interest in various events and trainings focused on implementation practice offered by an implementation science collaborative housed within a research-intensive university in the Southeast region of the United States. Our final analytic sample included 357 participants who self-identified as ISPs. Assessments of internal consistency reliability for each competency-specific item set yielded evidence of strong reliability. Results from confirmatory factor analyses provided evidence for the factorial and construct validity of all three domains and associated competencies in the ISCA. The findings suggest that one's possession of high levels of competence across each of the three competency domains is strongly associated with theorized outcomes that can promote successful and sustainable implementation efforts among those who receive implementation support from an ISP. The ISCA serves as a foundational tool for workforce development to formally measure and assess improvement in the skills that are required to tailor a package of implementation strategies situated in context.
Sections du résumé
BACKGROUND
BACKGROUND
Implementation support practitioners (ISPs) are professionals that support others to implement evidence-informed practices, programs, and policies in various service delivery settings to achieve population outcomes. Measuring the use of competencies by ISPs provides a unique opportunity to assess an understudied facet of implementation science-how knowledge, attitudes, and skills used by ISPs affects sustainable change in complicated and complex service systems. This study describes the development and validation of a measure-the Implementation Support Competencies Assessment (ISCA)-that assesses implementation support competencies, with versatile applications across service contexts.
METHODS
METHODS
Recently developed practice guide materials included operationalizations of core competencies for ISPs across three domains: co-creation and engagement, ongoing improvement, and sustaining change. These operationalizations, in combination with recent empirical and conceptual work, provided an initial item pool and foundation on which to advance measurement development, largely from a confirmatory perspective (as opposed to exploratory). The measure was further refined through modified cognitive interviewing with three highly experienced ISPs and pilot-testing with 39 individuals enrolled in a university-based certificate program in implementation practice. To recruit a sample for validation analyses, we leveraged a listserv of nearly 4,000 individuals who have registered for or expressed interest in various events and trainings focused on implementation practice offered by an implementation science collaborative housed within a research-intensive university in the Southeast region of the United States. Our final analytic sample included 357 participants who self-identified as ISPs.
RESULTS
RESULTS
Assessments of internal consistency reliability for each competency-specific item set yielded evidence of strong reliability. Results from confirmatory factor analyses provided evidence for the factorial and construct validity of all three domains and associated competencies in the ISCA.
CONCLUSIONS
CONCLUSIONS
The findings suggest that one's possession of high levels of competence across each of the three competency domains is strongly associated with theorized outcomes that can promote successful and sustainable implementation efforts among those who receive implementation support from an ISP. The ISCA serves as a foundational tool for workforce development to formally measure and assess improvement in the skills that are required to tailor a package of implementation strategies situated in context.
Identifiants
pubmed: 39107794
doi: 10.1186/s13012-024-01390-8
pii: 10.1186/s13012-024-01390-8
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
58Informations de copyright
© 2024. The Author(s).
Références
Albers B, Metz A, Burke K. Implementation support practitioners- A proposal for consolidating a diverse evidence base. BMC Health Serv Res. 2020;20(1):368.
doi: 10.1186/s12913-020-05145-1
pubmed: 32357877
pmcid: 7193379
Albers B, Metz A, Burke K, Bührmann L, Bartley L, Driessen P, et al. The Mechanisms of Implementation Support - Findings from a Systematic Integrative Review. Res Soc Work Pract. 2022;32(3):259–80.
doi: 10.1177/10497315211042375
Albers B, Metz A, Burke K, Bührmann L, Bartley L, Driessen P, et al. Implementation Support Skills: Findings From a Systematic Integrative Review. Res Soc Work Pract. 2021;31(2):147–70.
doi: 10.1177/1049731520967419
Metz A, Albers B, Burke K, Bartley L, Louison L, Ward C, et al. Implementation Practice in Human Service Systems: Understanding the Principles and Competencies of Professionals Who Support Implementation. Hum Serv Organ Manag Leadersh Gov. 2021;45(3):238–59.
Bührmann L, Driessen P, Metz A, Burke K, Bartley L, Varsi C, et al. Knowledge and attitudes of Implementation Support Practitioners—Findings from a systematic integrative review. Ochodo E, editor. PLoS One. 2022;17(5):e0267533.
Moore JE, Rashid S, Park JS, Khan S, Straus SE. Longitudinal evaluation of a course to build core competencies in implementation practice. Implement Sci. 2018;13(1):1–13.
doi: 10.1186/s13012-018-0800-3
Mosson R, Augustsson H, Bäck A, Åhström M, Von Thiele Schwarz U, Richter A, et al. Building implementation capacity (BIC): A longitudinal mixed methods evaluation of a team intervention. BMC Health Serv Res. 2019;19(1):1–12.
doi: 10.1186/s12913-019-4086-1
Park JS, Moore JE, Sayal R, Holmes BJ, Scarrow G, Graham ID, et al. Evaluation of the “Foundations in Knowledge Translation” training initiative: preparing end users to practice KT. Implement Sci. 2018;13(1):63.
doi: 10.1186/s13012-018-0755-4
pubmed: 29695267
pmcid: 5918493
William A, Aldridge I, Roppolo RH, Brown J, Bumbarger BK, Boothroyd RI. Mechanisms of change in external implementation support: A conceptual model and case examples to guide research and practice. 2023 Jun 21. https://doi.org/10.1177/26334895231179761 .
Westerlund A, Sundberg L, Nilsen P. Implementation of Implementation Science Knowledge: The Research-Practice Gap Paradox. Worldviews Evid Based Nurs. 2019;16(5):332–4.
doi: 10.1111/wvn.12403
pubmed: 31603625
pmcid: 6899530
Juckett LA, Bunger AC, McNett MM, Robinson ML, Tucker SJ. Leveraging academic initiatives to advance implementation practice: a scoping review of capacity building interventions. Implement Sci. 2022;17(1):1–14.
doi: 10.1186/s13012-022-01216-5
Jensen TM, Metz AJ, Disbennett ME, Farley AB. Developing a practice-driven research agenda in implementation science: Perspectives from experienced implementation support practitioners. Implement Res Pract. 2023;3:4.
Boaz A, Baeza J, Fraser A, Persson E. ‘It depends’: what 86 systematic reviews tell us about what strategies to use to support the use of research in clinical practice. Implement Sci. 2024;19(1):1–30.
doi: 10.1186/s13012-024-01337-z
Mazurek Melnyk B, Gallagher-Ford L, English Long L, Fineout-Overholt E. The establishment of evidence-based practice competencies for practicing registered nurses and advanced practice nurses in real-world clinical settings: Proficiencies to improve healthcare quality, reliability, patient outcomes, and costs. Worldviews Evid Based Nurs. 2014;11(1):5–15.
doi: 10.1111/wvn.12021
Metz A, Jensen T, Farley A, Boaz A. Is implementation research out of step with implementation practice? Pathways to effective implementation support over the last decade. Implement Res Pract. 2022;3:263348952211055.
doi: 10.1177/26334895221105585
Metz A, Albers B, Jensen T. Implementation support competencies assessment (ISCA). https://doi.org/10.17605/OSF.IO/ZH82F . 2024.
DeVellis R. Scale development: theory and applications. 3rd ed. Thousand Oaks, CA: Sage; 2012.
Metz A, Louison L, Burke K, Albers B, Ward C. Implementation support practitioner profile: Guiding principles and core competencies for implementation practice. University of North Carolina at Chapel Hill; 2020.
Wilson J, Ward C, Fetvadjiev VH, Bethel A. Measuring Cultural Competencies: The Development and Validation of a Revised Measure of Sociocultural Adaptation. J Cross Cult Psychol. 2017;48(10):1475–506.
doi: 10.1177/0022022117732721
McNeish D. Thanks coefficient alpha, we’ll take it from here. Psychol Methods. 2018;23(3):412–33.
doi: 10.1037/met0000144
pubmed: 28557467
Padilla MA, Divers J. A Comparison of Composite Reliability Estimators: Coefficient Omega Confidence Intervals in the Current Literature. Educ Psychol Meas. 2016;76(3):436–53.
doi: 10.1177/0013164415593776
pubmed: 29795872
Muthén L, Muthén B. Mplus User’s Guide. 8th ed. Los Angeles, CA: Muthén & Muthén; 2017.
Flora DB, Curran PJ. An Empirical Evaluation of Alternative Methods of Estimation for Confirmatory Factor Analysis With Ordinal Data. Psychol Methods. 2004;9(4):466–91.
doi: 10.1037/1082-989X.9.4.466
pubmed: 15598100
pmcid: 3153362
Shi D, Lee T, Fairchild AJ, Maydeu-Olivares A. Fitting Ordinal Factor Analysis Models With Missing Data: A Comparison Between Pairwise Deletion and Multiple Imputation. Educ Psychol Meas. 2020;80(1):41–66.
doi: 10.1177/0013164419845039
pubmed: 31933492
Li C. Little’s test of missing completely at random. Stata J. 2013;13(4):795–809.
doi: 10.1177/1536867X1301300407
Browne MW, Cudeck R. Alternative Ways of Assessing Model Fit. Sociol Methods Res. 1992;21(2):230–58.
doi: 10.1177/0049124192021002005
Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Struct Equ Modeling. 1999;6(1):1–55.
doi: 10.1080/10705519909540118
MacCallum RC, Browne MW, Sugawara HM. Power analysis and determination of sample size for covariance structure modeling. Psychol Methods. 1996;1(2):130–49.
doi: 10.1037/1082-989X.1.2.130
Cheung GW, Rensvold RB. Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance. Struct Equ Modeling. 2002;9(2):233–55.
doi: 10.1207/S15328007SEM0902_5
Wandersman A, Duffy J, Flaspohler P, Noonan R, Lubell K, Stillman L, et al. Bridging the Gap Between Prevention Research and Practice: The Interactive Systems Framework for Dissemination and Implementation. Am J Comm Psychol. 2008;41(3–4):171–81.
doi: 10.1007/s10464-008-9174-z
McGuier EA, Kolko DJ, Stadnick NA, Brookman-Frazee L, Wolk CB, Yuan CT, et al. Advancing research on teams and team effectiveness in implementation science: An application of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Res Pract. 2023;1:4.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76.
doi: 10.1007/s10488-010-0319-7
pubmed: 20957426
Proctor EK, Bunger AC, Lengnick-Hall R, Gerke DR, Martin JK, Phillips RJ, et al. Ten years of implementation outcomes research: a scoping review. Implement Sci. 2023;18(1):1–19.
doi: 10.1186/s13012-023-01313-z