What makes a good quality indicator set? A systematic review of criteria.
MeSH: health care quality indicators
content validity
criteria
indicator set
Journal
International journal for quality in health care : journal of the International Society for Quality in Health Care
ISSN: 1464-3677
Titre abrégé: Int J Qual Health Care
Pays: England
ID NLM: 9434628
Informations de publication
Date de publication:
31 Jul 2021
31 Jul 2021
Historique:
received:
21
04
2021
revised:
09
06
2021
accepted:
19
07
2021
pubmed:
21
7
2021
medline:
4
8
2021
entrez:
20
7
2021
Statut:
ppublish
Résumé
While single indicators measure a specific aspect of quality (e.g. timely support during labour), users of these indicators, such as patients, providers and policy-makers, are typically interested in some broader construct (e.g. quality of maternity care) whose measurement requires a set of indicators. However, guidance on desirable properties of indicator sets is lacking. Based on the premise that a set of valid indicators does not guarantee a valid set of indicators, the aim of this review is 2-fold: First, we introduce content validity as a desirable property of indicator sets and review the extent to which studies in the peer-reviewed health care quality literature address this criterion. Second, to obtain a complete inventory of criteria, we examine what additional criteria of quality indicator sets were used so far. We searched the databases Web of Science, Medline, Cinahl and PsycInfo from inception to May 2021 and the reference lists of included studies. English- or German-language, peer-reviewed studies concerned with desirable characteristics of quality indicator sets were included. Applying qualitative content analysis, two authors independently coded the articles using a structured coding scheme and discussed conflicting codes until consensus was reached. Of 366 studies screened, 62 were included in the review. Eighty-five per cent (53/62) of studies addressed at least one of the component criteria of content validity (content coverage, proportional representation and contamination) and 15% (9/62) addressed all component criteria. Studies used various content domains to structure the targeted construct (e.g. quality dimensions, elements of the care pathway and policy priorities), providing a framework to assess content validity. The review revealed four additional substantive criteria for indicator sets: cost of measurement (21% [13/62] of the included studies), prioritization of 'essential' indicators (21% [13/62]), avoidance of redundancy (13% [8/62]) and size of the set (15% [9/62]). Additionally, four procedural criteria were identified: stakeholder involvement (69% [43/62]), using a conceptual framework (44% [27/62]), defining the purpose of measurement (26% [16/62]) and transparency of the development process (8% [5/62]). The concept of content validity and its component criteria help assessing whether conclusions based on a set of indicators are valid conclusions about the targeted construct. To develop a valid indicator set, careful definition of the targeted construct including its (sub-)domains is paramount. Developers of quality indicators should specify the purpose of measurement and consider trade-offs with other criteria for indicator sets whose application may reduce content validity (e.g. costs of measurement) in light thereof.
Sections du résumé
BACKGROUND
BACKGROUND
While single indicators measure a specific aspect of quality (e.g. timely support during labour), users of these indicators, such as patients, providers and policy-makers, are typically interested in some broader construct (e.g. quality of maternity care) whose measurement requires a set of indicators. However, guidance on desirable properties of indicator sets is lacking.
OBJECTIVE
OBJECTIVE
Based on the premise that a set of valid indicators does not guarantee a valid set of indicators, the aim of this review is 2-fold: First, we introduce content validity as a desirable property of indicator sets and review the extent to which studies in the peer-reviewed health care quality literature address this criterion. Second, to obtain a complete inventory of criteria, we examine what additional criteria of quality indicator sets were used so far.
METHODS
METHODS
We searched the databases Web of Science, Medline, Cinahl and PsycInfo from inception to May 2021 and the reference lists of included studies. English- or German-language, peer-reviewed studies concerned with desirable characteristics of quality indicator sets were included. Applying qualitative content analysis, two authors independently coded the articles using a structured coding scheme and discussed conflicting codes until consensus was reached.
RESULTS
RESULTS
Of 366 studies screened, 62 were included in the review. Eighty-five per cent (53/62) of studies addressed at least one of the component criteria of content validity (content coverage, proportional representation and contamination) and 15% (9/62) addressed all component criteria. Studies used various content domains to structure the targeted construct (e.g. quality dimensions, elements of the care pathway and policy priorities), providing a framework to assess content validity. The review revealed four additional substantive criteria for indicator sets: cost of measurement (21% [13/62] of the included studies), prioritization of 'essential' indicators (21% [13/62]), avoidance of redundancy (13% [8/62]) and size of the set (15% [9/62]). Additionally, four procedural criteria were identified: stakeholder involvement (69% [43/62]), using a conceptual framework (44% [27/62]), defining the purpose of measurement (26% [16/62]) and transparency of the development process (8% [5/62]).
CONCLUSION
CONCLUSIONS
The concept of content validity and its component criteria help assessing whether conclusions based on a set of indicators are valid conclusions about the targeted construct. To develop a valid indicator set, careful definition of the targeted construct including its (sub-)domains is paramount. Developers of quality indicators should specify the purpose of measurement and consider trade-offs with other criteria for indicator sets whose application may reduce content validity (e.g. costs of measurement) in light thereof.
Identifiants
pubmed: 34282841
pii: 6324323
doi: 10.1093/intqhc/mzab107
pmc: PMC8325455
pii:
doi:
Types de publication
Journal Article
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Informations de copyright
© The Author(s) 2021. Published by Oxford University Press on behalf of International Society for Quality in Health Care.
Références
Parkinsonism Relat Disord. 2019 Mar;60:3-9
pubmed: 30772278
Int J Qual Health Care. 2001 Dec;13(6):469-74
pubmed: 11769749
Qual Health Res. 2005 Nov;15(9):1277-88
pubmed: 16204405
Gesundheitswesen. 2019 Oct;81(10):781-787
pubmed: 31574557
J Pain Symptom Manage. 2015 Apr;49(4):773-81
pubmed: 25697097
BMC Health Serv Res. 2010 Apr 06;10:88
pubmed: 20370925
Int J Qual Health Care. 2001 Dec;13(6):475-80
pubmed: 11769750
Med Care Res Rev. 2011 Jun;68(3):290-310
pubmed: 21156708
Qual Life Res. 2012 Jun;21(5):739-46
pubmed: 21866374
Can J Commun Ment Health. 2002 Spring;21(1):5-16
pubmed: 12630128
Can J Public Health. 1999 Nov-Dec;90 Suppl 1:S22-6
pubmed: 10686755
BMJ Qual Saf. 2016 Jul;25(7):525-34
pubmed: 26384710
Intern Med J. 2009 Jun;39(6):352-60
pubmed: 19323697
Hum Reprod. 2013 Jun;28(6):1584-97
pubmed: 23508250
PLoS One. 2013 May 01;8(5):e60947
pubmed: 23658684
Qual Health Res. 1993 Feb;3(1):112-21
pubmed: 8457790
BMC Fam Pract. 2010 Oct 27;11:81
pubmed: 20979612
Emerg Med Australas. 2014 Apr;26(2):113-24
pubmed: 24707999
Int J Qual Health Care. 2015 Apr;27(2):137-46
pubmed: 25758443
Int J Qual Health Care. 2010 Feb;22(1):44-69
pubmed: 19951964
Int J Qual Health Care. 2003 Dec;15(6):523-30
pubmed: 14660535
Neth J Med. 2007 Jan;65(1):15-22
pubmed: 17293635
Z Evid Fortbild Qual Gesundhwes. 2018 Jul;134:9-17
pubmed: 29274920
Health Policy. 2013 Nov;113(1-2):160-9
pubmed: 24095275
Med Care. 2003 Jan;41(1 Suppl):I30-8
pubmed: 12544814
Int J Qual Health Care. 2006 Sep;18 Suppl 1:5-13
pubmed: 16954510
Eur J Public Health. 2003 Sep;13(3 Suppl):101-6
pubmed: 14533758
Eur J Pediatr. 2018 Jun;177(6):921-933
pubmed: 29654400
Int J Integr Care. 2018 Jan 25;18(1):10
pubmed: 29588644
Springerplus. 2015 May 14;4:220
pubmed: 26069870
Health Policy. 2020 May;124(5):501-510
pubmed: 32192738
Int J Qual Health Care. 2003 Feb;15(1):79-85
pubmed: 12630804
J Clin Epidemiol. 2013 Dec;66(12):1328-37
pubmed: 24018342
Stroke. 2006 Oct;37(10):2573-8
pubmed: 16960092
J Aging Soc Policy. 2019 May-Jun;31(3):234-249
pubmed: 29883271
Int J Qual Health Care. 2003 Dec;15 Suppl 1:i5-11
pubmed: 14660518
PLoS One. 2017 Jun 2;12(6):e0178418
pubmed: 28575102
Health Policy. 2005 Dec;75(1):59-73
pubmed: 16298229
BMC Palliat Care. 2013 Feb 08;12:6
pubmed: 23394401
Intern Med J. 2009 Oct;39(10):648-54
pubmed: 19371394
Int J Qual Health Care. 2008 Jun;20(3):162-71
pubmed: 18339665
BMJ Qual Saf. 2012 Nov;21(11):964-8
pubmed: 22893696
Qual Life Res. 2018 May;27(5):1159-1170
pubmed: 29550964
Psychol Rev. 1955 May;62(3):193-217
pubmed: 14371898
Midwifery. 2013 Apr;29(4):316-24
pubmed: 23357096
Int J Qual Health Care. 2005 Dec;17(6):487-96
pubmed: 16155049
Res Synth Methods. 2014 Sep;5(3):221-34
pubmed: 26052848
Health Econ. 2018 Feb;27(2):e13-e27
pubmed: 28833902
Milbank Mem Fund Q. 1966 Jul;44(3):Suppl:166-206
pubmed: 5338568