A Systematic Review and Meta-Analysis of the Diagnostic Accuracy of the Advanced Clinical Solutions Word Choice Test as a Performance Validity Test.
Diagnostic accuracy
Meta-analysis
Performance validity
Systematic review
WCT
Word choice test
Journal
Neuropsychology review
ISSN: 1573-6660
Titre abrégé: Neuropsychol Rev
Pays: United States
ID NLM: 9009029
Informations de publication
Date de publication:
06 2021
06 2021
Historique:
received:
08
06
2020
accepted:
29
11
2020
pubmed:
16
1
2021
medline:
29
10
2021
entrez:
15
1
2021
Statut:
ppublish
Résumé
Thorough assessment of performance validity has become an established standard of practice in neuropsychological assessment. While there has been a large focus on the development and cross-validation of embedded performance validity tests (PVTs) in recent years, new freestanding PVTs have also been developed, including the Word Choice Test (WCT) as part of the Advanced Clinical Solutions Effort System. And, while the WCT's general utility for identifying invalid performance has been demonstrated in the ensuing decade since its initial publication, optimal cut-scores and associated psychometric properties have varied widely across studies. This study sought to synthesize the existing diagnostic accuracy literature regarding the WCT via a systematic review and to conduct a meta-analysis to determine the performance validity cut-score that best maximizes sensitivity while maintaining acceptable specificity. A systematic search of the literature resulted in 14 studies for synthesis, with eight of those available for meta-analysis. Meta-analytic results revealed an optimal cut-score of ≤ 42 with 54% sensitivity and 93% specificity for identifying invalid neuropsychological test performance. Collectively, the WCT demonstrated adequate diagnostic accuracy as a PVT across a variety of populations. Recommendations for future studies are also provided.
Identifiants
pubmed: 33447952
doi: 10.1007/s11065-020-09468-y
pii: 10.1007/s11065-020-09468-y
doi:
Types de publication
Journal Article
Meta-Analysis
Review
Systematic Review
Langues
eng
Sous-ensembles de citation
IM
Pagination
349-359Références
American Academy of Clinical Neuropsychology. (2007). American Academy of Clinical Neuropsychology (AACN) practice guidelines for neuropsychological assessment and consultation. The Clinical Neuropsychologist, 21, 209–231.
doi: 10.1080/13825580601025932
An, K. Y., Charles, J., Ali, S., Enache, A., Dhuga, J., & Erdodi, L. A. (2019). Reexamining performance validity cutoffs within the Complex Ideational Material and the Boston Naming Test-Short Form using an experimental malingering paradigm. Journal of Clinical and Experimental Neuropsychology, 41, 15–25.
doi: 10.1080/13803395.2018.1483488
Bain, K. M., & Soble, J. R. (2019). Validation of the Advanced Clinical Solutions Word Choice Test (WCT) in a mixed clinical sample: Establishing classification accuracy, sensitivity/specificity, and cutoff scores. Assessment, 26, 1320–1328.
doi: 10.1177/1073191117725172
Bain, K. M., Soble, J. R., Webber, T. A., Messerly, J. M., Bailey, K. C., Kirton, J. W., & McCoy, K. J. M. (2019). Cross-validation of three Advanced Clinical Solutions performance validity tests: Examining combinations of measures to maximize classification of invalid performance. Applied Neuropsychology: Adult. Advance online publication. https://doi.org/10.1080/23279095.2019.1585352
Bailey, K. C., Soble, J. R., Bain, K. M., & Fullen, C. (2018). Embedded performance validity tests in the Hopkins Verbal Learning Test – Revised and the Brief Visuospatial Memory Test – Revised: A replication study. Archives of Clinical Neuropsychology, 33(7), 895–900. https://doi.org/10.1093/arclin/acx111
doi: 10.1093/arclin/acx111
pubmed: 29161350
Barhon, L. I., Batchelor, J., Meares, S., Chekaluk, E., & Shores, E. A. (2015). A comparison of the degree of effort involved in the TOMM and the ACS Word Choice Test using dual task paradigm. Applied Neuropsychology Adult, 22, 114–123.
doi: 10.1080/23279095.2013.863775
Bashem, J. R., Rapport, L. J., Miller, J. B., Hanks, R. A., Axelrod, B. N., & Millis, S. R. (2014). Comparisons of five performance validity indices in bona fide and simulated traumatic brain injury. The Clinical Neuropsychologist, 28, 851–875.
doi: 10.1080/13854046.2014.927927
Boone, K. B., Lu, P., & Herzberg, D. (2002). The Dot Counting Test manual. Western Psychological Services.
Boone, K. B., Salazar, X., Lu, P., Warner-Chacon, K., & Razani, J. (2002). The Rey 15-Item Recognition Trial: A technique to enhance sensitivity of the Rey 15-Item Memorization Test. Journal of Clinical and Experimental Neuropsychology, 24, 561–573.
doi: 10.1076/jcen.24.5.561.1004
Boone, K. B. (2012). Clinical practice of forensic neuropsychology: An evidenced-based approach. Guilford Press.
Brennan, A. M., & Gouvier, W. D. (2006). Are we honestly studying malingering? A profile and comparison of simulated and suspected malingerers. Applied Neuropsychology, 13, 1–11.
doi: 10.1207/s15324826an1301_1
Chelune, G. J. (2017). Evidence-based practices in neuropsychology. In S. C. Bowden (Ed.), Neuropsychological assessment in the age of evidence-based practice: Diagnostic and treatment evaluations (pp. 155–181). Oxford University Press.
Critchfield, E., Soble, J. R., Marceaux, J. C., Bain, K. M., Bailey, K. C., Webber, T. A., et al. (2019). Cognitiveimpairment does not cause performance validity failure: Analyzing performance patterns among unimpaired, impaired, and noncredible participants across six tests. The Clinical Neuropsychologist, 33(6), 1083–1101. https://doi.org/10.1080/13854046.2018.1508615
doi: 10.1080/13854046.2018.1508615
pubmed: 30475095
Curtis, K. L., Thompson, L. K., Greve, K. W., & Bianchini, K. J. (2008). Verbal fluency indicators of malingering in traumatic brain injury: Classification accuracy in known groups. The Clinical Neuropsychologist, 22, 930–945.
doi: 10.1080/13854040701563591
Davis, J. J. (2014). Further consideration of Advanced Clinical Solutions Word Choice: comparison to the Recognition Memory Test-words and classification accuracy in a clinical sample. The Clinical Neuropsychologist, 28, 1278–1294.
doi: 10.1080/13854046.2014.975844
Dean, A. C., Victor, T. L., Boone, K. B., Philpott, L. M., & Hess, R. A. (2009). Dementia and effort test performance. The Clinical Neuropsychologist, 23, 133–152.
doi: 10.1080/13854040701819050
Erdodi, L. A., Kirsch, N. L., Lajiness-O’Neill, R., Vingilis, E., & Medoff, B. (2014). Comparing the Recognition Memory Test and the Word Choice Test in a mixed clinical sample: Are they equivalent? Psychological Injury and Law, 7, 255–263.
doi: 10.1007/s12207-014-9197-8
Erdodi, L. A., Tyson, B. T., Abeare, C. A., Zuccato, B. G., Rai, J. K., Seke, K. R., & Roth, R. M. (2017). Utility of critical items within the Recognition Memory Test and Word Choice Test. Applied Neuropsychology: Adult, 25, 327–339.
doi: 10.1080/23279095.2017.1298600
Erdodi, L. A., Tyson, B. T., Shahein, A. G., Lichtenstein, J. D., Abeare, C. A., Pelletier, C. L., & Roth, R. M. (2017). The power of timing: Adding a time-to-completion cutoff to the Word Choice Test and Recognition Memory Test improves classification accuracy. Journal of Clinical and Experimental Neuropsychology, 39, 369–383.
doi: 10.1080/13803395.2016.1230181
Freeman, S. C., Kerby, C. R., Patel, A., Cooper, N. J., Quinn, T., Sutton, A. J. (2019). Development of an interactive web-based tool to conduct and interrogate meta-analysis of diagnostic test accuracy studies: MetaDTA. BMC Medical Research Methodology, 19, 81. https://crsu.shinyapps.io/dta_ma/
Grant, A. (2018). Detection of feigned symptoms of ADHD, depression, and anxiety in college students (Unpublished master’s thesis). Saint Louis, MO: Saint Louis University.
Green, P. (2003). Green’s word memory test for Microsoft Windows: User’s manual. Green’s Publishing.
Green, P. (2004). Green’s medical symptom validity test (MSVT) for Microsoft Windows: User’s manual. Green’s Publishing.
Greiffenstein, M. F., Baker, W. J., & Gola, T. (1994). Validation of malingered amnesia measures with a large clinical sample. Psychological Assessment, 6, 218–224.
doi: 10.1037/1040-3590.6.3.218
Greve, K. W., Heinly, M. T., Bianchini, K. J., & Love, J. M. (2009). Malingering detection with the Wisconsin Card Sorting Test in mild traumatic brain injury. The Clinical Neuropsychologist, 23, 343–362.
doi: 10.1080/13854040802054169
Guyatt, G., Rennie, D., Meade, M., & Cook, D. (Eds.). (2015). Users’ guides to the medical literature: A manual for evidence-based clinical practice (3rd ed.). AMA Press.
Harbord, R. M., & Whiting, P. (2009). Metandi: Meta-analysis of diagnostic accuracy using hierarchical logistic regression. The Stata Journal, 9(2), 211–229.
doi: 10.1177/1536867X0900900203
Hardy, R. J., & Thompson, S. G. (1998). Detecting and describing heterogeneity in meta-analysis. Statistics in Medicine, 17(8), 841–856. https://doi.org/10.1002/(sici)1097-0258(19980430)17:8%3c841::aid-sim781%3e3.0.co;2-d
doi: 10.1002/(sici)1097-0258(19980430)17:8<841::aid-sim781>3.0.co;2-d
pubmed: 9595615
Hayward, L., Hall, W., Hunt, M., & Zubrick, S. R. (1987). Can localized brain impairment be simulated on neuropsychological test profiles? Australian and New Zealand Journal of Psychiatry, 21, 87–93.
doi: 10.3109/00048678709160904
Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., & Millis, S. R. (2009). American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23, 1093–1129.
doi: 10.1080/13854040903155063
Institute of Medicine. (2015). Psychological testing in the service of disability determination: Report brief. Institute of Medicine.
Iverson, G. L., Lange, R. T., Green, P., & Franzen, M. D. (2002). Detecting exaggeration and malingering with the Trail Making Test. The Clinical Neuropsychologist, 16, 398–406.
doi: 10.1076/clin.16.3.398.13861
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., & Moher, D. (2009). The PRISMA Statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and Elaboration. PLoS Medicine, 6(7), e1000100.
doi: 10.1371/journal.pmed.1000100
Lippa, S. M. (2018). Performance validity testing in neuropsychology: A clinical guide, critical review, and update on a rapidly evolving literature. The Clinical Neuropsychologist, 32(3), 391–421. https://doi.org/10.1080/13854046.2017.1406146
doi: 10.1080/13854046.2017.1406146
pubmed: 29182444
Lu, P. H., Boone, K. B., Cozolino, L., & Mitchell, C. (2003). Effectiveness of the Rey-Osterrieth Complex Figure Test and the Meyers and Meyers Recognition Trial in the detection of suspect effort. The Clinical Neuropsychologist, 17, 426–440.
doi: 10.1076/clin.17.3.426.18083
Macaskill, P., Gatsonis, C., Deeks, J., Harbord, R., & Takwoingi, Y. (2010). Analysing and presenting results. In J. Deeks, P. Bossuyt & C. Gatsonis (Eds.), Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 1.0 (pp. 1–61). The Cochrane Collaboration. http://methods.cochrane.org/
Martin, P. K., Schroeder, R. W., & Odland, A. P. (2015). Neuropsychologists’ validity testing beliefs and practices: A survey of North American professionals. The Clinical Neuropsychologist, 29, 741–776.
doi: 10.1080/13854046.2015.1087597
Martin, P. K., Schroeder, R. W., Olsen, D., Maloy, H., Boettcher, A., Ernst, N., & Okut, H. (2020). A systematic review and meta-analysis of the Test of Memory Malingering in adults: Two decades of deception detection. The Clinical Neuropsychologist, 34, 88–119.
doi: 10.1080/13854046.2019.1637027
Miller, J. B., Millis, S. R., Rapport, L. J., Bashem, J. R., Hanks, R. A., & Axelrod, B. N. (2011). Detection of insufficient effort using the Advanced Clinical Solutions for the Wechsler Memory Scale, Fourth Edition. The Clinical Neuropsychologist, 25, 160–172.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group. (2008). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA Statement. PLoS Medicine, 6, e100097.
Neale, A. C., Ovsiew, G. P., Resch, Z. J., & Soble, J. R. (2020). Feigning or forgetfulness: The effect of memory impairment severity on Word Choice Test performance. The Clinical Neuropsychologist. Advance online publication. https://doi.org/10.1080/13854046.2020.1799076
Ovsiew, G. P., Resch, Z. J., Nayar, K., Williams, C. P., & Soble, J. R. (2020). Not so fast! Limitations of processing speed and working memory indices as embedded performance validity tests in a mixed neuropsychiatric sample. Journal of Clinical and Experimental Neuropsychology, 42(5), 473–484. https://doi.org/10.1080/13803395.2020.1758635
doi: 10.1080/13803395.2020.1758635
pubmed: 32498648
Patrick, S. D., Rapport, L. J., Kanser, R. J., Hanks, R. A., & Bashem, J. R. (2020). Performance validity assessment using response time on the Warrington Recognition Memory Test. The Clinical Neuropsychologist: Advance online publication.
doi: 10.1080/13854046.2020.1716997
Pearson, N. (2009). Advanced clinical solutions for WAIS-IV and WMS-IV: Administration and scoring manual. The Psychological Corporation.
Pliskin, J. I., De-Dios-Stern, S., Resch, Z. J., Saladino, K. F., Ovsiew, G. P., Carter, D. A., & Soble, J. R. (2020). Comparing the psychometric properties of 8 embedded performance validity tests in the Rey Auditory Verbal Learning Test, Wechsler Memory Scale Logical Memory, and Brief Visuospatial Memory Test-Revised recognition trials for detecting invalid neuropsychological test performance. Assessment. Advance online publication. https://doi.org/10.1177/1073191120929093
Resch, Z. J., Pham, A. T., Abramson, D. A., White. D. J., DeDios-Stern, S., Ovsiew, G. P., Castillo, L., & Soble, J. R. (2020). Examining independent and combined accuracy of embedded performance validity tests in the California Verbal Learning Test-II and Brief Visuospatial Memory Test-Revised for detecting invalid performance. Applied Neuropsychology: Adult. Advance online publication. https://doi.org/10.1080/23279095.2020.1742718
Rey, A. (1964). L’examen Clinique en psychologie. Presses Universitaires de France.
Rogers, R. (Ed.) (2008). Clinical assessment of malingering and deception (3rd ed.). Guilford Press.
Root, J. C., Robbins, R. N., Chang, L., & Van Gorp, W. G. (2006). Detection of inadequate effort on the California Verbal Learning Test-Second Edition: Forced choice recognition and critical item analysis. Journal of the International Neuropsychological Society, 12, 688–696.
doi: 10.1017/S1355617706060838
Schroeder, R. W., Martin, P. K., Heinrichs, R. J., & Baade, L. E. (2019). Research methods in performance validity testing studies: Criterion grouping approach impacts study outcomes. The Clinical Neuropsychologist, 33, 466–477.
doi: 10.1080/13854046.2018.1484517
Soble, J. R., Alverson, W. A., Phillips, J. I., Critchfield, E. A., Fullen, C., O’Rourke, J. J. F., et al. (2020). Strength in numbers or quality over quantity? Examining the importance of criterion measure selection to define validity groups in performance validity test (PVT) research. Psychological Injury and Law, 13, 44–56. https://doi.org/10.1007/s12207-019-09370-w
doi: 10.1007/s12207-019-09370-w
Soble, J. R., Bain, K. M., Bailey, K. C., Kirton, J. W., Marceaux, J. M., Critchfield, E. A., et al. (2019). Evaluating the accuracy of the Wechsler Memory Scale-Fourth Edition (WMS-IV) Logical Memory embedded validity index for detecting invalid test performance. Applied Neuropsychology: Adult, 26(4), 311–318. https://doi.org/10.1080/23279095.2017.1418744
doi: 10.1080/23279095.2017.1418744
Sopko, K. (2012). Detecting feigned reading learning disability and Attention Deficit Hyperactivity Disorder: The effects of coaching (Unpublished master’s thesis). Saint Louis, MO: Saint Louis University.
Tombaugh, T. N. (1996). Test of memory malingering. Multi-Health Systems.
Vickery, C. D., Berry, D. T., Inman, T. H., Harris, M. J., & Orey, S. A. (2001). Detection of inadequate effort on neuropsychological testing: A meta-analytic review of selected procedures. Archives of Clinical Neuropsychology, 16, 45–73.
pubmed: 14590192
Viechtbauer, W. (2007). Hypothesis tests for population heterogeneity in meta-analysis. The British Journal of Mathematical and Statistical Psychology, 60(Pt 1), 29–60. https://doi.org/10.1348/000711005X64042
doi: 10.1348/000711005X64042
pubmed: 17535578
Webber, T. A., Critchfield, E. A., & Soble, J. R. (2020). Convergent, discriminant, and concurrent validity of nonmemory-based performance validity tests. Assessment, 27(7), 1399–1415. https://doi.org/10.1177/1073191118804874
doi: 10.1177/1073191118804874
pubmed: 30293445
Webber, T. A., & Soble, J. R. (2018). Utility of various WAIS-IV Digit Span indices for identifying noncredible performance validity among cognitively impaired and unimpaired examinees. The Clinical Neuropsychologist, 32(4), 657–670. https://doi.org/10.1080/13854046.2017.1415374
doi: 10.1080/13854046.2017.1415374
pubmed: 29256334
White, D. J., Korinek, D., Bernstein, M. T., Ovsiew, G. P., Resch, Z. J., & Soble, J. R. (2020). Cross-validation of non-memory-based embedded performance validity tests for detecting invalid performance among patients with and without neurocognitive impairment. Journal of Clinical and Experimental Neuropsychology, 42, 459–472. https://doi.org/10.1080/13803395.2020.1758634
doi: 10.1080/13803395.2020.1758634
pubmed: 32397824
Whiting, P. F., Rutjes, A. W., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M., Sterne, J. A., Bossuyt, P. M., & QUADAS-2 Group. (2011). QUADAS-2: A revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine, 155(8), 529–536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
doi: 10.7326/0003-4819-155-8-201110180-00009
Young, J. C., Roper, B. L., & Arentsen, T. J. (2016). Validity testing and neuropsychology practice in the VA healthcare system: results from recent practitioner survey. The Clinical Neuropsychologist, 30, 497–514.
doi: 10.1080/13854046.2016.1159730
Zuccato, B. G., Tyson, B. T., & Erdodi, L. A. (2018). Early bird fails the PVT? The effects of timing artifacts on performance validity tests. Psychological Assessment, 30, 1491–1498.
doi: 10.1037/pas0000596