Sequentially Determined Measures of Interobserver Agreement (Kappa) in Clinical Trials May Vary Independent of Changes in Observer Performance.
Cohen kappa
biased estimator
central reading
clinical trials
interobserver agreement
repeated measures
simulation
Journal
Therapeutic innovation & regulatory science
ISSN: 2168-4804
Titre abrégé: Ther Innov Regul Sci
Pays: Switzerland
ID NLM: 101597411
Informations de publication
Date de publication:
30 Sep 2019
30 Sep 2019
Historique:
entrez:
2
10
2019
pubmed:
2
10
2019
medline:
2
10
2019
Statut:
aheadofprint
Résumé
Cohen's kappa is a statistic that estimates interobserver agreement. It was originally introduced to help develop diagnostic tests. Interpretative readings of 2 observers, for example, of a mammogram or other imaging, were compared at a single point in time. It is known that kappa depends on the prevalence of disease and that, therefore, kappas across different settings are hard to compare. Using simulation, we examine an analogous situation, not previously described, that occurs in clinical trials where sequential measurements are obtained to evaluate disease progression or clinical improvement over time. We show that weighted kappa, used for multilevel outcomes, changes during the trial even if we keep the performance of the observer constant. Kappa and closely related measures can therefore only be used with great difficulty, if at all, in quality assurance in clinical trials.
Sections du résumé
BACKGROUND
BACKGROUND
Cohen's kappa is a statistic that estimates interobserver agreement. It was originally introduced to help develop diagnostic tests. Interpretative readings of 2 observers, for example, of a mammogram or other imaging, were compared at a single point in time. It is known that kappa depends on the prevalence of disease and that, therefore, kappas across different settings are hard to compare.
METHODS
METHODS
Using simulation, we examine an analogous situation, not previously described, that occurs in clinical trials where sequential measurements are obtained to evaluate disease progression or clinical improvement over time.
RESULTS
RESULTS
We show that weighted kappa, used for multilevel outcomes, changes during the trial even if we keep the performance of the observer constant.
CONCLUSIONS
CONCLUSIONS
Kappa and closely related measures can therefore only be used with great difficulty, if at all, in quality assurance in clinical trials.
Identifiants
pubmed: 31569962
doi: 10.1177/2168479019874059
doi:
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM