Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C): development and psychometric evaluation of a measure of sustainability determinants in the early childhood education and care setting.
Development
Early childhood education and care
Measurement
Reliability
Sustainability
Sustainment
Validity
Journal
Implementation science : IS
ISSN: 1748-5908
Titre abrégé: Implement Sci
Pays: England
ID NLM: 101258411
Informations de publication
Date de publication:
20 Jun 2024
20 Jun 2024
Historique:
received:
02
08
2023
accepted:
13
06
2024
medline:
21
6
2024
pubmed:
21
6
2024
entrez:
20
6
2024
Statut:
epublish
Résumé
There is a need for valid and reliable measures of determinants of sustainability of public health interventions in early childhood education and care (ECEC) settings. This study aimed to develop and evaluate the psychometric and pragmatic properties of such a measure - the Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C). We undertook a two-phase process guided by the COnsensus-based Standards for the selection of health status Measurement INstruments checklist (COSMIN) and Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Phase 1 involved measure development; i.e., determining items and scales through an iterative process and assessment of face and content validity. Phase 2 involved the evaluation of psychometric and pragmatic properties. The 29-item measure completed by service executives (directors and nominated supervisors) was embedded in a larger survey from a national sample of Australian ECEC services assessing their implementation of nutrition and physical activity programs. Structural validity, concurrent validity, known groups validity, internal consistency, floor and ceiling effects, norms, and pragmatic qualities of the measure were assessed according to the PAPERS criteria. The final measure contained 26 items, with respondents reporting how strongly they agreed or disagreed on a five-point Likert scale. Phase 1 assessments confirmed the relevance, and face and content validity of the scale. In Phase 2, we obtained 482 completed surveys, of which 84% (n = 405) completed the entire measure across 405 ECEC settings (one executive per service). Three of the four fit indices for the confirmatory factor analysis met the pre-specified criteria (SRMR = 0.056, CFI = 0.993, RMSEA = 0.067) indicating 'good' structural validity. The IMPRESS-C illustrated: 'good' internal consistency, with Cronbach's alpha values from 0.53 to 0.92; 'emerging' concurrent validity; 'poor' known groups validity; 'good' norms; and 'good' overall pragmatic qualities (cost, readability, length, and assessor burden). The IMPRESS-C possesses strong psychometric and pragmatic qualities for assessing service executive-level perceptions of determinants influencing sustainment of public health interventions within ECEC settings. To achieve a full range of perspectives in this setting, future work should be directed to also develop and test measures of sustainability determinants at the implementer level (e.g., among individual educators and staff).
Sections du résumé
BACKGROUND
BACKGROUND
There is a need for valid and reliable measures of determinants of sustainability of public health interventions in early childhood education and care (ECEC) settings. This study aimed to develop and evaluate the psychometric and pragmatic properties of such a measure - the Integrated Measure of PRogram Element SuStainability in Childcare Settings (IMPRESS-C).
METHODS
METHODS
We undertook a two-phase process guided by the COnsensus-based Standards for the selection of health status Measurement INstruments checklist (COSMIN) and Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Phase 1 involved measure development; i.e., determining items and scales through an iterative process and assessment of face and content validity. Phase 2 involved the evaluation of psychometric and pragmatic properties. The 29-item measure completed by service executives (directors and nominated supervisors) was embedded in a larger survey from a national sample of Australian ECEC services assessing their implementation of nutrition and physical activity programs. Structural validity, concurrent validity, known groups validity, internal consistency, floor and ceiling effects, norms, and pragmatic qualities of the measure were assessed according to the PAPERS criteria.
RESULTS
RESULTS
The final measure contained 26 items, with respondents reporting how strongly they agreed or disagreed on a five-point Likert scale. Phase 1 assessments confirmed the relevance, and face and content validity of the scale. In Phase 2, we obtained 482 completed surveys, of which 84% (n = 405) completed the entire measure across 405 ECEC settings (one executive per service). Three of the four fit indices for the confirmatory factor analysis met the pre-specified criteria (SRMR = 0.056, CFI = 0.993, RMSEA = 0.067) indicating 'good' structural validity. The IMPRESS-C illustrated: 'good' internal consistency, with Cronbach's alpha values from 0.53 to 0.92; 'emerging' concurrent validity; 'poor' known groups validity; 'good' norms; and 'good' overall pragmatic qualities (cost, readability, length, and assessor burden).
CONCLUSIONS
CONCLUSIONS
The IMPRESS-C possesses strong psychometric and pragmatic qualities for assessing service executive-level perceptions of determinants influencing sustainment of public health interventions within ECEC settings. To achieve a full range of perspectives in this setting, future work should be directed to also develop and test measures of sustainability determinants at the implementer level (e.g., among individual educators and staff).
Identifiants
pubmed: 38902763
doi: 10.1186/s13012-024-01372-w
pii: 10.1186/s13012-024-01372-w
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
41Subventions
Organisme : National Health and Medical Research Council
ID : APP1197022
Organisme : National Health and Medical Research Council
ID : APP1194785
Organisme : Australia Research Council Discovery Early Career Research Awards
ID : DE170100382
Organisme : American Cancer Society
ID : RSG-17-156-01-CPPB
Informations de copyright
© 2024. The Author(s).
Références
Wolfenden L, Nathan NK, Sutherland R, Yoong SL, Hodder RK, Wyse RJ, et al. Strategies for enhancing the implementation of school-based policies or practices targeting risk factors for chronic disease. Cochrane Database Syst Rev. 2017;11(11):CD011677. https://doi.org/10.1002/14651858.CD011677.pub2 .
doi: 10.1002/14651858.CD011677.pub2
pubmed: 29185627
Matwiejczyk L, Mehta K, Scott J, Tonkin E, Coveney J. Characteristics of effective interventions promoting healthy eating for pre-schoolers in childcare settings: an umbrella review. Nutrients. 2018;10(3):293. https://doi.org/10.3390/nu10030293 .
doi: 10.3390/nu10030293
pubmed: 29494537
pmcid: 5872711
Wolfenden L, Barnes C, Jones J, Finch M, Wyse RJ, Kingsland M, et al. Strategies to improve the implementation of healthy eating, physical activity and obesity prevention policies, practices or programmes within childcare services. Cochrane Database of Syst Rev. 2020;2(2):CD011779. https://doi.org/10.1002/14651858.CD011779.pub3 .
doi: 10.1002/14651858.CD011779.pub3
Lum M, Wolfenden L, Jones J, Grady A, Christian H, Reilly K, Yoong SL. Interventions to improve child physical activity in the early childhood education and care setting: an umbrella review. Int J Environ Res Public Health. 2022;19(4):1963. https://doi.org/10.3390/ijerph19041963 .
doi: 10.3390/ijerph19041963
pubmed: 35206152
pmcid: 8872396
Yoong SL, Lum M, Wolfenden L, Jackson J, Barnes C, Hall AE, et al. Healthy eating interventions delivered in early childhood education and care settings for improving the diet of children aged six months to six years. Cochrane Database of Syst Rev. 2023;6(6):CD013862. https://doi.org/10.1002/14651858 .
doi: 10.1002/14651858
WiltseyStirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research. Implement Sci. 2012;7(1):1–19. https://doi.org/10.1186/1748-5908-7-17 .
doi: 10.1186/1748-5908-7-17
Herlitz L, MacIntyre H, Osborn T, Bonell C. The sustainability of public health interventions in schools: a systematic review. Implement Sci. 2020;15(1):1–28. https://doi.org/10.1186/s13012-019-0961-8 .
doi: 10.1186/s13012-019-0961-8
Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health programs. Am J Public Health. 2011;101(11):2059–67. https://doi.org/10.2105/AJPH.2011.300193 .
doi: 10.2105/AJPH.2011.300193
pubmed: 21940916
pmcid: 3222409
Shelton RC, Cooper BR, Stirman SW. The sustainability of evidence-based interventions and practices in public health and health care. Annu Rev Public Health. 2018;39:55–76. https://doi.org/10.1146/annurev-publhealth-040617-014731 .
doi: 10.1146/annurev-publhealth-040617-014731
pubmed: 29328872
Bodkin A, Hakimi S. Sustainable by design: a systematic review of factors for health promotion program sustainability. BMC Public Health. 2020;20(1):964. https://doi.org/10.1186/s12889-020-09091-9 .
doi: 10.1186/s12889-020-09091-9
pubmed: 32560718
pmcid: 7304137
Moore JE, Mascarenhas A, Bain J, Straus SE. Developing a comprehensive definition of sustainability. Implement Sci. 2017;12(1):1–8. https://doi.org/10.1186/s13012-017-0637-1 .
doi: 10.1186/s13012-017-0637-1
Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The program sustainability assessment tool: a new instrument for public health programs. Prev Chronic Dis. 2014;11: 130184. http://dx.doi.org/10.5888/pcd11.130184 .
doi: 10.5888/pcd11.130184
pubmed: 24456645
Birken SA, Haines ER, Hwang S, Chambers DA, Bunger AC, Nilsen P. Advancing understanding and identifying strategies for sustaining evidence-based practices: a review of reviews. Implement Sci. 2020;15:88. https://doi.org/10.1186/s13012-020-01040-9 .
doi: 10.1186/s13012-020-01040-9
pubmed: 33036653
pmcid: 7545853
Cassar S, Salmon J, Timperio A, Naylor PJ, van Nassau F, Contardo Ayala AM, et al. Adoption, implementation and sustainability of school-based physical activity and sedentary behaviour interventions in real-world settings: a systematic review. Int J Behav Nutr Phys Act. 2019;16(1):120. https://doi.org/10.1186/s12966-019-0876-4 .
doi: 10.1186/s12966-019-0876-4
pubmed: 31791341
pmcid: 6889569
Shoesmith A, Hall A, Wolfenden L, Shelton RC, Powell BJ, Brown H, et al. Barriers and facilitators influencing the sustainment of health behaviour interventions in schools and childcare services: a systematic review. Implement Sci. 2021;16:62. https://doi.org/10.1186/s13012-021-01134-y .
doi: 10.1186/s13012-021-01134-y
pubmed: 34118955
pmcid: 8199827
Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(1):1–11. https://doi.org/10.1186/1748-5908-8-117 .
doi: 10.1186/1748-5908-8-117
Schell SF, Luke DA, Schooley MW, Elliott MB, Herbers SH, Mueller NB, et al. Public health programs capacity for sustainability: a new framework. Implement Sci. 2013;8:15. https://doi.org/10.1186/1748-5908-8-15 .
doi: 10.1186/1748-5908-8-15
pubmed: 23375082
pmcid: 3599102
Hodge LM, Turner KMT. Sustained implementation of evidence-based programs in disadvantaged communities: a conceptual framework of supporting factors. Am J Community Psychol. 2016;58(1–2):192–210. https://doi.org/10.1002/ajcp.12082 .
doi: 10.1002/ajcp.12082
pubmed: 27624514
Seward K, Wolfenden L, Wiggers J, Finch M, Wyse R, Oldmeadow C, et al. Measuring implementation behaviour of menu guidelines in the childcare setting: confirmatory factor analysis of a theoretical domains framework questionnaire (TDFQ). Int J Behav Nutr Phys Act. 2017;14:45. https://doi.org/10.1186/s12966-017-0499-6 .
doi: 10.1186/s12966-017-0499-6
pubmed: 28376892
pmcid: 5381057
Hall A, Shoesmith A, Doherty E, McEvoy B, Mettert K, Lewis CC, et al. Evaluation of measures of sustainability and sustainability determinants for use in community, public health, and clinical settings: a systematic review. Implement Sci. 2022;17:81. https://doi.org/10.1186/s13012-022-01252-1 .
doi: 10.1186/s13012-022-01252-1
pubmed: 36514059
pmcid: 9746194
Boateng GO, Neilands TB, Frongillo EA, Melgar-Quiñonez HR, Young SL. Best practices for developing and validating scales for health, social, and behavioral research: a primer. Front Public Health. 2018;6:149. https://doi.org/10.3389/fpubh.2018.00149 .
doi: 10.3389/fpubh.2018.00149
pubmed: 29942800
pmcid: 6004510
Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):115–1170. https://doi.org/10.1007/s11136-018-1829-0 .
doi: 10.1007/s11136-018-1829-0
Hull L, Boulton R, Jones F, Boaz A, Sevdalis N. Defining, conceptualizing and evaluating pragmatic qualities of quantitative instruments measuring implementation determinants and outcomes: a scoping and critical review of the literature and recommendations for future research. Transl Behav Med. 2022;12(11):1049–64. https://doi.org/10.1093/tbm/ibac064 .
doi: 10.1093/tbm/ibac064
pubmed: 36318228
pmcid: 9677469
Lewis CC, Mettert KD, Stanick CF, Halko HM, Nolen EA, Powell BJ, Weiner BJ. The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implement Res Pract. 2021;10(2):26334895211037390. https://doi.org/10.1177/26334895211037391 .
doi: 10.1177/26334895211037391
Moullin JC, Sklar M, Green A, Dickson KS, Stadnick NA, Reeder K, et al. Advancing the pragmatic measurement of sustainment: a narrative review of measures. Implement Sci Commun. 2020;1:76. https://doi.org/10.1186/s43058-020-00068-8 .
doi: 10.1186/s43058-020-00068-8
pubmed: 32964208
pmcid: 7499830
Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155. https://doi.org/10.1186/s13012-015-0342-x .
doi: 10.1186/s13012-015-0342-x
pubmed: 26537706
pmcid: 4634818
Mettert K, Lewis C, Dorsey C, Halko H, Weiner B. Measuring implementation outcomes: an updated systematic review of measures’ psychometric properties. Implement Res Pract. 2020;1:1–29. https://doi.org/10.1177/263348952093664 .
doi: 10.1177/263348952093664
Moullin JC, Sklar M, Ehrhart MG, Green A, Aarons G. Provider REport of Sustainment Scale (PRESS): development and validation of a brief measure of inner context sustainment. Implement Sci. 2021;16:86. https://doi.org/10.1186/s13012-021-01152-w .
doi: 10.1186/s13012-021-01152-w
pubmed: 34461948
pmcid: 8404332
Chamberlain P, Hendricks Brown C, Saldana L. Observational measure of implementation progress in community based settings: the Stages of Implementation Completion (SIC). Implement Sci. 2011;6:116.
doi: 10.1186/1748-5908-6-116
pubmed: 21974914
pmcid: 3197550
Palinkas LA, Chou CP, Spear SE, Mendon SJ, Villamar J, Brown CH. Measurement of sustainment of prevention programs and initiatives: the sustainment measurement system scale. Implement Sci. 2020;15(1):71.
doi: 10.1186/s13012-020-01030-x
pubmed: 32883352
pmcid: 7470441
Stanick CF, Halko HM, Nolen EA, Powell BJ, Dorsey CN, Mettert KD, et al. Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Transl Behav Med. 2021;11(1):11–20. https://doi.org/10.1093/tbm/ibz164 .
doi: 10.1093/tbm/ibz164
pubmed: 31747021
McIntosh K, MacKay LD, Hume AE, Doolittle J, Vincent CG, Horner RH, et al. Development and initial validation of a measure to assess factors related to sustainability of school-wide positive behavior support. J Posit Behav Interv. 2010;13(4):208–18.
doi: 10.1177/1098300710385348
Malone S, Prewitt K, Hackett R, Lin JC, McKay V, Walsh-Bailey C, et al. The clinical sustainability assessment tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun. 2021;2(1):77.
doi: 10.1186/s43058-021-00181-2
pubmed: 34274004
pmcid: 8285819
Kittelman A, Mercer SH, McIntosh K, Nese RNT. Development and validation of a measure assessing sustainability of tier 2 and 3 behavior support systems. J Sch Psychol. 2021;85:140–54.
doi: 10.1016/j.jsp.2021.02.001
pubmed: 33715778
Hodge LM, Turner KMT, Sanders MR, Filus A. Sustained implementation support scale: validation of a measure of program characteristics and workplace functioning for sustained program implementation. J Behav Health Serv Res. 2017;44(3):442–64.
doi: 10.1007/s11414-016-9505-z
pubmed: 27048371
Ehrhart MG, Torres EM, Green AE, Trott E, Willging CE, Moullin JC, et al. Leading for the long haul: a mixed-method evaluation of the Sustainment Leadership Scale (SLS). Implement Sci. 2018;13:17. https://doi.org/10.1186/s13012-018-0710-4 .
doi: 10.1186/s13012-018-0710-4
pubmed: 29351767
pmcid: 5775537
Mancini JA, Marek LI. Sustaining community-based programs for families: conceptualisation and measurement. Fam Relat. 2004;53(4):339–47.
doi: 10.1111/j.0197-6664.2004.00040.x
Williams RM, Zhang J, Woodard N, Slade JL, Santos LZ, Knott CL. Development and validation of an instrument to assess institutionalization of health promotion in faith-based organizations. Eval Program Plann. 2020;79:101781.
doi: 10.1016/j.evalprogplan.2020.101781
pubmed: 31991309
Solberg LI, Asche SE, Margolis KL, Whitebird RR. Measuring an organization’s ability to manage change: the change process capability questionnaire and its use for improving depression care. Am J Med Qual. 2008;23(3):193–200.
doi: 10.1177/1062860608314942
pubmed: 18539980
The Board of Regents of the University System of Georgia by and on behalf of Georgia State University and the Georgia Health Policy Center. Positioning for sustainability: a formative assessment tool – quick course. 2011.
Office of Adolescent Health. Building sustainable programs: the resource guide. 2014.
Maher L, Gustafson DH, Evans A. Sustainability model and guide. 2010.
OECD. Enrolment in childcare and pre-schools. Available at: https://www.oecd.org/els/soc/PF3_2_Enrolment_childcare_preschool.pdf . Accessed 12 June 2023.
Scheirer MA. Linking sustainability research to intervention types. Am J Public Health. 2013;103(4):e73–80. https://doi.org/10.2105/AJPH.2012.300976 .
doi: 10.2105/AJPH.2012.300976
pubmed: 23409904
pmcid: 3673273
Asada Y, Lin S, Siegel L, Kong A. Facilitators and barriers to implementation and sustainability of nutrition and physical activity interventions in early childcare settings: a systematic review. Prev Sci. 2023;24(1):64–83. https://doi.org/10.1007/s11121-022-01436-7 .
doi: 10.1007/s11121-022-01436-7
pubmed: 36198924
Mokkink LB, Terwee CB, Knol DL, Stratford PW, Alonso J, Patrick DL, et al. Protocol of the COSMIN study: COnsensus-based Standards for the selection of health Measurement INstruments. BMC Med Res Methodol. 2006;6:2. https://doi.org/10.1186/1471-2288-6-2 .
doi: 10.1186/1471-2288-6-2
pubmed: 16433905
pmcid: 1368990
Finch TL, Girling M, May CR, Mair FS, Murray E, Treweek S, et al. Improving the normalization of complex interventions: part 2 - validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. 2018;18(1):135. https://doi.org/10.1186/s12874-018-0591-x .
doi: 10.1186/s12874-018-0591-x
pubmed: 30442094
pmcid: 6238372
Rapley T, Girling M, Mair FS, Murray E, Treweek S, McColl E, et al. Improving the normalization of complex interventions: part 1 - development of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. 2018;18(1):133. https://doi.org/10.1186/s12874-018-0590-y .
doi: 10.1186/s12874-018-0590-y
pubmed: 30442093
pmcid: 6238361
Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42. https://doi.org/10.1016/j.jclinepi.2006.03.012 .
doi: 10.1016/j.jclinepi.2006.03.012
pubmed: 17161752
Yoong SL, Pearson N, Reilly K, Wolfenden L, Jones J, Nathan N, et al. A randomised controlled trial of an implementation strategy delivered at scale to increase outdoor free play opportunities in early childhood education and care (ECEC) services: a study protocol for the get outside get active (GOGA) trial. BMC Public Health. 2022;22:610. https://doi.org/10.1186/s12889-022-12883-w .
doi: 10.1186/s12889-022-12883-w
pubmed: 35351035
pmcid: 8961494
Rosseel Y. lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48(2):1–36. https://doi.org/10.18637/jss.v048.i02 .
doi: 10.18637/jss.v048.i02
R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2022. https://www.R-project.org/ .
O’Rourke N, Hatcher L. A step-by-step approach to using SAS® for factor analysis and structural equation modeling. 2nd ed. Cary: SAS Institute Inc.; 2013.
Fabrigar LR, Wegener DT, MacCallum RC, Strahan E. Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods. 1999;4(3):272–99.
doi: 10.1037/1082-989X.4.3.272
Mîndrilă D. Maximum Likelihood (ML) and Diagonally Weighted Least Squares (DWLS) estimation procedures: a comparison of estimation bias with ordinal and multivariate non-normal data. IJDS. 2010;1(1):60–6.
doi: 10.20533/ijds.2040.2570.2010.0010
Li CH. Confirmatory factor analysis with ordinal data: comparing robust maximum likelihood and diagonally weighted least squares. Behav Res. 2016;48:936–49. https://doi.org/10.3758/s13428-015-0619-7 .
doi: 10.3758/s13428-015-0619-7
Harrell Jr F & Dupont CH. Hmisc: Harrell Miscellaneous. R Package Version 4.2–0. 2019. https://CRAN.R-project.org/package=Hmisc .
Kline RB. Principles and practice of structural equation modeling. New York: Guilford Press; 2005.
Weiner BJ, Lewis CC, Stanick C, Powel BJ, Dorsey C, Clary A, et al. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017;12(1):108. https://doi.org/10.1186/s13012-017-0635-3 .
doi: 10.1186/s13012-017-0635-3
pubmed: 28851459
pmcid: 5576104
DeVellis RF. Scale development: theory and application. Los Angeles: Sage; Publications; 2012.
Hooper D, Coughlan J, Mullen M. Structural equation modelling: guidelines for determining model fit. Electron J Bus Res Methods. 2007;6(1):53–60.
Lin WL, Yao G. Concurrent Validity. In: Michalos AC, editor. Encyclopedia of quality of life and well-being research. Dordrecht: Springer; 2014. https://doi.org/10.1007/978-94-007-0753-5_516 .
doi: 10.1007/978-94-007-0753-5_516
Akoglu H. User’s guide to correlation coefficients. Turk J Emerg Med. 2018;18(3):91–3. https://doi.org/10.1016/j.tjem.2018.08.001 .
doi: 10.1016/j.tjem.2018.08.001
pubmed: 30191186
pmcid: 6107969
Hailemariam M, Bustos T, Montgomery B, Barajas R, Evans LB, Drahota A. Evidence-based intervention sustainability strategies: a systematic review. Implement Sci. 2019;14:57. https://doi.org/10.1186/s13012-019-0910-6 .
doi: 10.1186/s13012-019-0910-6
pubmed: 31171004
pmcid: 6554955
Comfrey AL, Lee HB. A First Course in Factor Analysis. Hillsdale: Lawrence Erlbaum Associates; 1992.
García AA. Cognitive interviews to test and refine questionnaires. Public Health Nurs. 2011;28(5):444–50. https://doi.org/10.1111/j.1525-1446.2010.00938.x .
doi: 10.1111/j.1525-1446.2010.00938.x
pubmed: 22092429
Souza AC, Alexandre NMC, Guirardello EB. Psychometric properties in instruments evaluation of reliability and validity. Epidemiol Serv Saude. 2017;26(3):649–59. https://doi.org/10.5123/S1679-49742017000300022 . English, Portuguese.
doi: 10.5123/S1679-49742017000300022
pubmed: 28977189
Lennox L, Maher L, Reed J. Navigating the sustainability landscape: a systematic review of sustainability approaches in healthcare. Implement Sci. 2018;13:27. https://doi.org/10.1186/s13012-017-0707-4 .
doi: 10.1186/s13012-017-0707-4
pubmed: 29426341
pmcid: 5810192
Bruce B, Fries J, Lingala B, Hussain YN, Krishnan E. Development and assessment of floor and ceiling items for the PROMIS physical function item bank. Arthritis Res Ther. 2013;15(5):R144. https://doi.org/10.1186/ar4327 .
doi: 10.1186/ar4327
pubmed: 24286166
pmcid: 3978724
Glasgow RE, Riley WT. Pragmatic measures: what they are and why we need them. Am J Prev Med. 2013;45(2):237–43. https://doi.org/10.1016/j.amepre.2013.03.010 .
doi: 10.1016/j.amepre.2013.03.010
pubmed: 23867032