Online Training of Community Therapists in Observational Coding of Family Therapy Techniques: Reliability and Accuracy.
Evidence-based intervention
Family therapy
Observational coding
Online training
Reliability
Usual care
Journal
Administration and policy in mental health
ISSN: 1573-3289
Titre abrégé: Adm Policy Ment Health
Pays: United States
ID NLM: 8914574
Informations de publication
Date de publication:
01 2022
01 2022
Historique:
accepted:
14
07
2021
pubmed:
24
7
2021
medline:
2
2
2022
entrez:
23
7
2021
Statut:
ppublish
Résumé
A foundational strategy to promote implementation of evidence-based interventions (EBIs) is providing EBI training to therapists. This study tested an online training system in which therapists practiced observational coding of mock video vignettes demonstrating family therapy techniques for adolescent behavior problems. The study compared therapists ratings to gold-standard scores to measure therapist reliability (consistency across vignettes) and accuracy (approximation to gold scores); tested whether reliability and accuracy improved during training; and tested therapist-level predictors of overall accuracy and change in accuracy over time. Participants were 48 therapists working in nine community behavioral health clinics. The 32-exercise training course provided online instruction (about 15 min/week) in 13 core family therapy techniques representing three modules: Family Engagement, Relational Orientation, Interactional Change. Therapist reliability in rating technique presence (i.e., technique recognition) remained moderate across training; reliability in rating extensiveness of technique delivery (i.e., technique judgment) improved sharply over time, from poor to good. Whereas therapists on average overestimated extensiveness for almost every technique, their tendency to give low-accuracy scores decreased. Therapist accuracy improved significantly over time only for Interactional Change techniques. Baseline digital literacy and submission of self-report checklists on use of the techniques in their own sessions predicted coding accuracy. Training therapists to be more reliable and accurate coders of EBI techniques can potentially yield benefits in increased EBI self-report acumen and EBI use in daily practice. However, training effects may need to improve from those reported here to avail meaningful impact on EBI implementation.Trial Registration: The parent clinical trial is registered at www.ClinicalTrials.gov , ID: NCT03342872 (registration date: 11.10.17).
Identifiants
pubmed: 34297259
doi: 10.1007/s10488-021-01152-4
pii: 10.1007/s10488-021-01152-4
pmc: PMC8298690
doi:
Banques de données
ClinicalTrials.gov
['NCT03342872']
Types de publication
Journal Article
Observational Study
Research Support, N.I.H., Extramural
Langues
eng
Sous-ensembles de citation
IM
Pagination
139-151Subventions
Organisme : NIDA NIH HHS
ID : R34 DA044740
Pays : United States
Informations de copyright
© 2021. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.
Références
Baldwin, S. A., Christian, S., Berkeljon, A., & Shadish, W. R. (2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy, 38, 281–304.
pubmed: 22283391
Becker, E. M., & Jensen-Doss, A. (2014). Therapist attitudes towards computer-based trainings. Administration and Policy in Mental Health and Mental Health Services Research, 41, 845–854.
pubmed: 24150441
Beidas, R. S., Cross, W., & Dorsey, S. (2014). Show me, don’t tell me: Behavioral rehearsal as a training and analogue fidelity tool. Cognitive and Behavioral Practice, 21, 1–11.
pubmed: 25382963
Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17, 1–30.
Beidas, R. S., Williams, N. J., Becker-Haimes, E. M., Aarons, G. A., Barg, F. K., Evans, A. C., Jackson, K., Jones, D., Hadley, T., Hoagwood, K., & Marcus, S. C. (2019). A repeated cross-sectional study of clinicians’ use of psychotherapy techniques during 5 years of a system-wide effort to implement evidence-based practices in Philadelphia. Implementation Science, 14, 67.
pubmed: 31226992
pmcid: 6588873
Caron, E. B., & Dozier, M. (2019). Effects of fidelity-focused consultation on clinicians’ implementation: An exploratory multiple baseline design. Administration and Policy in Mental Health and Mental Health Services Research, 46, 445–457.
pubmed: 30783903
Caron, E. B., Muggeo, M. A., Souer, H. R., Pella, J. E., & Ginsburg, G. S. (2020). Concordance between clinician, supervisor and observer ratings of therapeutic competence in CBT and treatment as usual: Does clinician competence or supervisor session observation improve agreement? Behavioural and Cognitive Psychotherapy, 48, 350–363.
pubmed: 31806076
Carroll, K. M., Nich, C., Sifry, R., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug & Alcohol Dependence, 57, 225–238.
Chorpita, B. F., & Daleiden, E. L. (2009). Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials. Journal of Consulting and Clinical Psychology, 77, 566–579.
pubmed: 19485596
Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Laurence Erlbaum Associates.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Damberg, C. L., Sorbero, M. E., Lovejoy, S. L., Martsolf, G. R., Raaen, L., & Mandel, D. (2014). Measuring success in health care value-based purchasing programs: Findings from an environmental scan, literature review, and expert panel discussions. Rand Health Quarterly, 4(3), 9.
pubmed: 28083347
pmcid: 5161317
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–340.
Ehrenreich-May, J., Dimeff, L. A., Woodcock, E. A., Queen, A. H., Kelly, T., Contreras, I. S., Rash, B., Kelley-Brimer, A., Hauschildt, J., Danner, S. M., & Kennedy, S. M. (2016). Enhancing online training in an evidence-based treatment for adolescent panic disorder: A randomized controlled trial. Evidence-Based Practice in Child and Adolescent Mental Health, 1, 241–258.
Gonder, J., Metlay, W., & Shapiro, T. (2018). Testing assumptions: Can performance rating feedback result in objective performance improvements? Journal of Management and Innovation. https://doi.org/10.18059/jmi.v4i2.100
doi: 10.18059/jmi.v4i2.100
Grotevant, H. D., & Carlson, C. I. (1987). Family interaction coding systems: A descriptive review. Family Process, 26, 49–74.
pubmed: 3552722
Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23.
pubmed: 22833776
pmcid: 3402032
Henderson, C. E., Hogue, A., & Dauber, S. (2019). Family therapy techniques and one-year clinical outcomes among adolescents in usual care for behavior problems. Journal of Consulting and Clinical Psychology, 87, 308–312.
pubmed: 30589350
Hogue, A., Bobek, M., Dauber, S., Henderson, C. E., McLeod, B. D., & Southam-Gerow, M. A. (2019a). Core elements of family therapy for adolescent behavior problems: Empirical distillation of three manualized treatments. Journal of Clinical Child and Adolescent Psychology, 48, 29–41.
pubmed: 30657722
pmcid: 6377288
Hogue, A., Bobek, M., MacLean, A., Porter, N., Jensen-Doss, A., & Henderson, C. E. (2020). Measurement training and feedback system for implementation of evidence-based treatment for adolescent externalizing problems: Protocol for a randomized trial of pragmatic clinician training. Trials. https://doi.org/10.1186/s13063-019-3783-8
doi: 10.1186/s13063-019-3783-8
Hogue, A., Dauber, S., Bobek, M., Jensen-Doss, A., & Henderson, C. E. (2019b). Measurement training and feedback system for implementation of family-based services for adolescent substance use: Protocol for a cluster randomized trial of two implementation strategies. Implementation Science, 14, 25.
pubmed: 30866967
pmcid: 6416843
Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., Inclan, J., Reiner, R., & Liddle, H. A. (2008). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment, 35, 137–147.
pubmed: 17997268
Hogue, A., Dauber, S., & Henderson, C. E. (2014). Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research, 41, 126–139.
pubmed: 23124275
Hogue, A., Dauber, S., & Henderson, C. E. (2017). Benchmarking family therapy for adolescent behavior problems in usual care: Fidelity, outcomes, and therapist performance differences. Administration and Policy in Mental Health and Mental Health Services Research, 44, 626–641.
pubmed: 27664141
Hogue, A., Dauber, S., Henderson, C. E., Bobek, M., Johnson, C., Lichvar, E., & Morgenstern, J. (2015). Randomized trial of family therapy versus non-family treatment for adolescent behavior problems in usual care. Journal of Clinical Child and Adolescent Psychology, 44, 954–969.
pubmed: 25496283
Hogue, A., Henderson, C. E., Becker, S. J., & Knight, D. K. (2018). Evidence base on outpatient behavioral treatments for adolescent substance use, 2014–2017: Outcomes, treatment delivery, and promising horizons. Journal of Clinical Child and Adolescent Psychology, 47, 499–526.
pubmed: 29893607
pmcid: 7192024
Hogue, A., Liddle, H. A., & Rowe, C. (1996). Treatment adherence process research in family therapy: A rationale and some practical guidelines. Psychotherapy: Theory, Research, Practice, & Training, 33, 332–345.
Isenhart, C., Dieperink, E., Thuras, P., Fuller, B., Stull, L., Koets, N., & Lenox, R. (2014). Training and maintaining motivational interviewing skills in a clinical trial. Journal of Substance Use, 19, 164–170.
Jensen-Doss, A., Smith, A. M., Walsh, L. M., Ringle, V. M., Casline, E., Patel, Z., Shaw, A. M., Maxwell, C., Hanson, R., & Webster, R. (2020). Preaching to the choir? Predictors of engagement in a community-based learning collaborative. Administration and Policy in Mental Health and Mental Health Services Research, 47, 279–290.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15, 155–163.
pubmed: 27330520
pmcid: 4913118
Kreft, I., & DeLeeuw, J. (1998). Introducing multilevel modeling. Sage Publications.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.
pubmed: 843571
Lyon, A., & Koerner, K. (2016). User-centered design for psychosocial intervention development and implementation. Clinical Psychology: Science and Practice, 23(2), 180–200.
Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research, 19, 181–193.
pubmed: 19396649
pmcid: 2782382
McCart, M. R., & Sheidow, A. J. (2016). Evidence-based psychosocial treatments for adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology, 45, 529–563.
McLeod, B. D., Cox, J. R., Jensen-Doss, A., Herschell, A., Ehrenreich-May, J., & Wood, J. J. (2018). Proposing a mechanistic model of clinician training and consultation. Clinical Psychology: Science and Practice, 25(3), e12260.
Minuchin, S., & Fishman, H. C. (1981). Family therapy techniques. Harvard University Press.
Moran, G., Diamond, G. M., & Diamond, G. S. (2005). The relational reframe and parents’ problem constructions in attachment-based family therapy. Psychotherapy Research, 15, 226–235.
pubmed: 22011152
Muthén, L. K., & Muthén, B. O. (1998–2021). Mplus user's guide (Seventh ed.). Muthen & Muthen.
Riedinger, V., Pinquart, M., & Teubert, D. (2017). Effects of systemic therapy on mental health of children and adolescents: A meta-analysis. Journal of Clinical Child and Adolescent Psychology, 46, 880–894.
pubmed: 26467300
Robbins, M., Feaster, D., Horigian, V., Puccinelli, M., Henderson, C. E., & Szapocznik, J. (2011). Therapist adherence in Brief Strategic Family Therapy for adolescent drug abusers. Journal of Consulting and Clinical Psychology, 79, 43–53.
pubmed: 21261433
pmcid: 3440762
Shrout, P., & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
pubmed: 18839484
Stirman, S. W. (2020). Commentary: Challenges and opportunites in the assessment of fidelity and related constructs. Administration and Policy in Mental Health and Mental Health Services Research, 47, 932–934.
pubmed: 32715432
Stirman, S. W., Bhar, S. S., Spokas, M., Brown, G. K., Creed, T. A., Perivoliotis, D., Farabaugh, D. T., Grant, P. M., & Beck, A. T. (2010). Training and consultation in evidence-based psychosocial treatments in public mental health settings: The access model. Professional Psychology: Research and Practice, 41, 48.
Tanner-Smith, E. E., Wilson, S. J., & Lipsey, M. W. (2013). The comparative effectiveness of outpatient treatment for adolescent substance abuse: A meta-analysis. Journal of Substance Abuse Treatment, 44, 145–158.
pubmed: 22763198
U.S. Department of Health and Human Services. (2020). Telehealth: Delivering care safely during COVID-19. Website: https://www.hhs.gov/coronavirus/telehealth/index.html
Valenstein-Mah, H., Greer, N., McKenzie, L., Hansen, L., Strom, T. Q., Wiltsey Stirman, S., Wilt, T. J., & Kehle-Forbes, S. M. (2020). Effectiveness of training methods for delivery of evidence-based psychotherapies: A systematic review. Implementation Science, 15, 1–17.
Weingardt, K. R. (2004). The role of instructional design and technology in the dissemination of empirically supported, manual-based therapies. Clinical Psychology: Science and Practice, 11, 313–331.
Weingardt, K. R., Cucciare, M. A., Bellotti, C., & Lai, W. P. (2009). A randomized trial comparing two models of web-based training in cognitive–behavioral therapy for substance abuse counselors. Journal of Substance Abuse Treatment, 37, 219–227.
pubmed: 19339136
pmcid: 2771721
Weisz, J. R., Bearman, S., Santucci, L., & Jensen-Doss, A. (2017). Initial test of a principle-guided approach to transdiagnostic psychotherapy with children and adolescents. Journal of Clinical Child and Adolescent Psychology, 46, 44–58.
pubmed: 27442352