Applying behaviour change theory to understand the barriers to implementing routine outcome monitoring.
Child and Adolescent Mental Health Services
behaviour change interventions
implementation
routine outcome monitoring
Journal
The British journal of clinical psychology
ISSN: 0144-6657
Titre abrégé: Br J Clin Psychol
Pays: England
ID NLM: 8105533
Informations de publication
Date de publication:
Sep 2022
Sep 2022
Historique:
revised:
29
06
2021
received:
29
10
2020
pubmed:
29
7
2021
medline:
5
8
2022
entrez:
28
7
2021
Statut:
ppublish
Résumé
Routine outcome monitoring (ROM) is a valuable tool for monitoring client progress and pre-empting deterioration, however, there is considerable variation in how data are collected and recorded and uptake in clinical practice remains low. The aim of this study was to develop a self-report measure of practitioner attitudes to ROM in order to better understand the barriers to successful implementation in Child and Adolescent Mental Health Services (CAMHS). An anonymous survey was completed by 184 CAMHS practitioners in the United Kingdom. The survey was designed using the Capability, Opportunity, and Motivation Model of Behaviour (COM-B). Practitioners who reported using ROM frequently in their clinical work (53%) were compared to those who used ROM infrequently (47%) across dimensions of the COM-B survey subscales. Confirmatory factor analysis confirmed the proposed four-factor structure, showing acceptable model fit, with high factor loadings and good reliability for all subscales. Frequent users of ROM exhibited significantly higher psychological capability, physical opportunity, social opportunity, and motivation, compared to infrequent users F (4, 140) = 14.76, p < .0001; Pillai's Trace = .297, partial η In the hope of improving the successful implementation of ROM, this research provides an evidence-based tool for assessing practitioners' attitudes to ROM, which map on to intervention functions and represent targets for future implementation efforts. The value of routine outcome monitoring (ROM) as a means to measure client progress and to elevate the efficiency and quality of mental health care is well-documented in the research literature, however, uptake in practice remains relatively low. This study applied behaviour change theory to develop a psychometrically sound self-report measure of practitioners' perspectives and practices to understand the barriers to implementation in child and adolescent mental health services in the United Kingdom. The complex and multifaceted nature of the barriers to implementation requires multilevel behaviour change strategies at the client, clinician, and organisational level. Recommendations for practice include the need for integrated, multilevel strategies aimed at improving practitioners' capabilities and motivations, strong organisational leadership and a culture of data gathering and sharing, and implementation interventions, which are tailored to target local barriers.
Sections du résumé
BACKGROUND
BACKGROUND
Routine outcome monitoring (ROM) is a valuable tool for monitoring client progress and pre-empting deterioration, however, there is considerable variation in how data are collected and recorded and uptake in clinical practice remains low. The aim of this study was to develop a self-report measure of practitioner attitudes to ROM in order to better understand the barriers to successful implementation in Child and Adolescent Mental Health Services (CAMHS).
METHODS
METHODS
An anonymous survey was completed by 184 CAMHS practitioners in the United Kingdom. The survey was designed using the Capability, Opportunity, and Motivation Model of Behaviour (COM-B). Practitioners who reported using ROM frequently in their clinical work (53%) were compared to those who used ROM infrequently (47%) across dimensions of the COM-B survey subscales.
RESULTS
RESULTS
Confirmatory factor analysis confirmed the proposed four-factor structure, showing acceptable model fit, with high factor loadings and good reliability for all subscales. Frequent users of ROM exhibited significantly higher psychological capability, physical opportunity, social opportunity, and motivation, compared to infrequent users F (4, 140) = 14.76, p < .0001; Pillai's Trace = .297, partial η
IMPLICATIONS
CONCLUSIONS
In the hope of improving the successful implementation of ROM, this research provides an evidence-based tool for assessing practitioners' attitudes to ROM, which map on to intervention functions and represent targets for future implementation efforts.
PRACTITIONER POINTS
CONCLUSIONS
The value of routine outcome monitoring (ROM) as a means to measure client progress and to elevate the efficiency and quality of mental health care is well-documented in the research literature, however, uptake in practice remains relatively low. This study applied behaviour change theory to develop a psychometrically sound self-report measure of practitioners' perspectives and practices to understand the barriers to implementation in child and adolescent mental health services in the United Kingdom. The complex and multifaceted nature of the barriers to implementation requires multilevel behaviour change strategies at the client, clinician, and organisational level. Recommendations for practice include the need for integrated, multilevel strategies aimed at improving practitioners' capabilities and motivations, strong organisational leadership and a culture of data gathering and sharing, and implementation interventions, which are tailored to target local barriers.
Types de publication
Journal Article
Langues
eng
Sous-ensembles de citation
IM
Pagination
557-578Subventions
Organisme : Child Outcomes Research Consortium (CORC)
Informations de copyright
© 2021 The British Psychological Society.
Références
Aarons, G. A., & Sawitzky, A. C. (2006). Organizational culture and climate and mental health provider attitudes toward evidence-based practice. Psychological Services, 3(1), 61-72. https://doi.org/10.1037/1541-1559.3.1.61
Allison, P. D. (2003). Missing data techniques for structural equation modeling. Journal of Abnormal Psychology, 112, 545-557. https://doi.org/10.1037/0021-843X.112.4.545
Atkins, L., Francis, J., Islam, R., O’Connor, D., Patey, A., Ivers, N., … Grimshaw, J. M. (2017). A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implementation Science, 12(1), 77. https://doi.org/10.1186/s13012-017-0605-9
Batty, M. J., Moldavsky, M., Foroushani, P. S., Pass, S., Marriott, M., Sayal, K., & Hollis, C. (2013). Implementing routine outcome measures in child and adolescent mental health services: From present to future practice. Child and Adolescent Mental Health, 18, 82-87. https://doi.org/10.1111/j.1475-3588.2012.00658.x
Bergman, H., Kornør, H., Nikolakopoulou, A., Hanssen-Bauer, K., Soares-Weiser, K., Tollefsen, T. K., & Bjørndal, A. (2018). Client feedback in psychological therapy for children and adolescents with mental health problems. Cochrane Database of Systematic Reviews, 8(8), CD011729. https://doi.org/10.1002/14651858.CD011729.pub2
Bickman, L., Douglas, S. R., De Andrade, A. R. V., Tomlinson, M., Gleacher, A., Olin, S., & Hoagwood, K. (2016). Implementing a measurement feedback system: A tale of two sites. Administration and Policy in Mental Health and Mental Health Services Research, 43, 410-425. https://doi.org/10.1007/s10488-015-0647-8
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62, 1423-1429. https://doi.org/10.1176/appi.ps.002052011
Bickman, L., Rosof-Williams, J., Salzer, M. S., Summerfelt, W. T., Noser, K., Wilson, S. J., & Karver, M. S. (2000). What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice, 31(1), 70-74. https://doi.org/10.1037/0735-7028.31.1.70
Bjaastad, J. F., Jensen-Doss, A., Moltu, C., Jakobsen, P., Hagenberg, H., & Joa, I. (2019). Attitudes toward standardized assessment tools and their use among clinicians in a public mental health service. Nordic Journal of Psychiatry, 73, 387-396. https://doi.org/10.1080/08039488.2019.1642383
Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25(1), 6-19. https://doi.org/10.1080/10503307.2013.817696
Brattland, H., Koksvik, J. M., Burkeland, O., Gråwe, R. W., Klöckner, C., Linaker, O. M., … Iversen, V. C. (2018). The effects of routine outcome monitoring (ROM) on therapy outcomes in the course of an implementation process: A randomized clinical trial. Journal of Counseling Psychology, 65, 641-652. https://doi.org/10.1037/cou0000286
Callaly, T., Hyland, M., Coombs, T., & Trauer, T. (2006). Routine outcome measurement in public mental health: Results of a clinician survey. Australian Health Review, 30(2), 164-173.
Carlier, I. V. E., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van Der Wee, N. J. A., & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104-110. https://doi.org/10.1111/j.1365-2753.2010.01543.x
Cattell, R. (2012). The scientific use of factor analysis in behavioral and life sciences. Berlin, Germany: Springer Science & Business Media.
Close-Goedjen, J. L., & Saunders, S. M. (2002). The effect of technical support on clinician attitudes toward an outcome assessment instrument. Journal of Behavioral Health Services and Research, 29(1), 99-108. https://doi.org/10.1007/BF02287837
Connors, E. H., Douglas, S., Jensen-Doss, A., Landes, S. J., Lewis, C. C., McLeod, B. D., … Lyon, A. R. (2021). What gets measured gets done: How mental health agencies can leverage measurement-based care for better patient care, clinician supports, and organizational goals. Administration and Policy in Mental Health and Mental Health Services Research, 48, 250-265. https://doi.org/10.1007/s10488-020-01063-w
de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22, 464-474. https://doi.org/10.1080/10503307.2012.673023
De Leo, A., Bayes, S., Bloxsome, D., & Butt, J. (2021). Exploring the usability of the COM-B model and Theoretical Domains Framework (TDF) to define the helpers of and hindrances to evidence-based practice in midwifery. Implementation Science Communications, 2(1), 1-8. https://doi.org/10.1186/s43058-020-00100-x
Delgadillo, J., de Jong, K., Lucock, M., Lutz, W., Rubel, J., Gilbody, S., … McMillan, D. (2018). Feedback-informed treatment versus usual psychological treatment for depression and anxiety: A multisite, open-label, cluster randomised controlled trial. The Lancet Psychiatry, 5, 564-572. https://doi.org/10.1016/S2215-0366(18)30162-7
Delgadillo, J., Overend, K., Lucock, M., Groom, M., Kirby, N., McMillan, D., … de Jong, K. (2017). Improving the efficiency of psychological treatment using outcome feedback technology. Behaviour Research and Therapy, 99, 89-97. https://doi.org/10.1016/j.brat.2017.09.011
Douglas, S., Button, S., & Casey, S. E. (2016). Implementing for sustainability: Promoting use of a measurement feedback system for innovation and quality improvement. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 286-291. https://doi.org/10.1007/s10488-014-0607-8
Douglas, S., Jonghyuk, B., de Andrade, A. R. V., Tomlinson, M. M., Hargraves, R. P., & Bickman, L. (2015). Feedback mechanisms of change: How problem alerts reported by youth clients and their caregivers impact clinician-reported session content. Psychotherapy Research, 25, 678-693. https://doi.org/10.1080/10503307.2015.1059966
Eccles, M., Grimshaw, J., Walker, A., Johnston, M., & Pitts, N. (2005). Changing the behavior of healthcare professionals: The use of theory in promoting the uptake of research findings. Journal of Clinical Epidemiology, 58(2), 107-112. https://doi.org/10.1016/j.jclinepi.2004.09.002
Edbrooke-Childs, J., Wolpert, M., & Deighton, J. (2016). Using Patient Reported Outcome Measures to Improve Service Effectiveness (UPROMISE): Training clinicians to Use Outcome Measures in Child Mental Health. Administration and Policy in Mental Health and Mental Health Services Research, 43, 302-308. https://doi.org/10.1007/s10488-014-0600-2
Enders, C. K., & Bandalos, D. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8, 430-457. https://doi.org/10.1207/S15328007SEM0803_5
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S., & Merry, S. (2018). Beyond the trial: Systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. Journal of Medical Internet Research, 20(6), e199. https://doi.org/10.2196/jmir.9275
Ford, J. K., MacCallum, R. C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, 291-314. https://doi.org/10.1111/j.1744-6570.1986.tb00583.x
Glass, G. V., Peckham, P. D., & Sanders, J. R. (1972). Consequences of failure to meet assumptions underlying the fixed effects analyses of variance and covariance. Review of Educational Research, 42(3), 237-288. https://doi.org/10.3102/00346543042003237
Gleacher, A. A., Olin, S. S., Nadeem, E., Pollock, M., Ringle, V., Bickman, L., … Hoagwood, K. (2016). Implementing a measurement feedback system in community mental health clinics: A case study of multilevel barriers and facilitators. Administration and Policy in Mental Health and Mental Health Services Research, 43, 426-440. https://doi.org/10.1007/s10488-015-0642-0
Gondek, D., Edbrooke-Childs, J., Fink, E., Deighton, J., & Wolpert, M. (2016). Feedback from outcome measures and treatment effectiveness, treatment efficiency, and collaborative practice: A systematic review. Administration and Policy in Mental Health and Mental Health Services Research, 43, 325-343. https://doi.org/10.1007/s10488-015-0710-5
Goodman, R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry and Allied Disciplines, 38, 581-586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates.
Hall, C. L., Moldavsky, M., Baldwin, L., Marriott, M., Newell, K., Taylor, J., … Hollis, C. (2013). The use of routine outcome measures in two child and adolescent mental health services: A completed audit cycle. BMC Psychiatry, 13(1), 270. https://doi.org/10.1186/1471-244X-13-270
Hall, C. L., Taylor, J., Moldavsky, M., Marriott, M., Pass, S., Newell, K., … Hollis, C. (2014). A qualitative process evaluation of electronic session-by-session outcome measurement in child and adolescent mental health services. BMC Psychiatry, 14(1), 1-11. https://doi.org/10.1186/1471-244X-14-113
Hamilton, J. D., & Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child & Adolescent Psychiatry, 47, 1114-1119. https://doi.org/10.1097/CHI.0b013e3181825af8
Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research and Practice, 35(5), 485-491. https://doi.org/10.1037/0735-7028.35.5.485
Hattingh, L., Sim, T. F., Sunderland, B., & Czarniak, P. (2020). Successful implementation and provision of enhanced and extended pharmacy services. Research in Social and Administrative Pharmacy, 16, 464-474. https://doi.org/10.1016/j.sapharm.2019.06.015
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55. https://doi.org/10.1080/10705519909540118
Ionita, G., & Fitzpatrick, M. (2014). Bringing science to clinical practice: A Canadian survey of psychological practice and usage of progress monitoring measures. Canadian Psychology/Psychologie Canadienne, 55(3), 187. https://doi.org/10.1037/a0037355
Janse, P. D., de Jong, K., Veerkamp, C., van Dijk, M. K., Hutschemaekers, G. J. M., & Verbraak, M. J. P. M. (2020). The effect of feedback-informed cognitive behavioral therapy on treatment outcome: A randomized controlled trial. Journal of Consulting and Clinical Psychology, 88(9), 818-828. https://doi.org/10.1037/ccp0000549
Jensen-Doss, A., Douglas, S., Phillips, D. A., Gencdur, O., Zalman, A., & Gomez, N. E. (2020). Measurement-based care as a practice improvement tool: Clinical and organizational applications in youth mental health. Evidence-Based Practice in Child and Adolescent Mental Health, 5(3), 233-250. https://doi.org/10.1080/23794925.2020.1784062
Jensen-Doss, A., Haimes, E. M. B., Smith, A. M., Lyon, A. R., Lewis, C. C., Stanick, C. F., & Hawley, K. M. (2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48-61. https://doi.org/10.1007/s10488-016-0763-0
Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, https://doi.org/10.1080/15374416.2010.517169
Johnston, C., & Gowers, S. (2005). Routine outcome measurement: A survey of UK Child and Adolescent Mental Health Services. Child and Adolescent Mental Health, 10, 133-139. https://doi.org/10.1111/j.1475-3588.2005.00357.x
Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., … Moore, M. (2016). Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database of Systematic Reviews, 7, CD011119.https://doi.org/10.1002/14651858.CD011119.pub2
Kenny, D. A., & Judd, C. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422-431. https://doi.org/10.1037/0033-2909.99.3.422
Lambert, M. J., & Harmon, K. L. (2018). The merits of implementing routine outcome monitoring in clinical practice. Clinical Psychology: Science and Practice, 25(4), 12268. https://doi.org/10.1111/cpsp.12268
Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55, 520-537. https://doi.org/10.1037/pst0000167
Lewis, C. C., Boyd, M., Puspitasari, A., Navarro, E., Howard, J., Kassab, H., … Kroenke, K. (2019). Implementing measurement-based care in behavioral health: A review. JAMA Psychiatry, 76(3), 324. https://doi.org/10.1001/jamapsychiatry.2018.3329
Lewis, C. C., Stanick, C. F., Martinez, R. G., Weiner, B. J., Kim, M., Barwick, M., & Comtois, K. A. (2015). The society for implementation research collaboration instrument review project: A methodology to promote rigorous evaluation. Implementation Science, 10(1), 1-18. https://doi.org/10.1186/s13012-014-0193-x
Little, R. J. A. (1988). A test of missing completely at random for multivariate data with missing values. Journal of the American Statistical Association, 83, 1198-1202. https://doi.org/10.1080/01621459.1988.10478722
Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research, 42(1), 47-60. https://doi.org/10.1007/s10488-014-0543-7
Mackrill, T., & Sørensen, K. M. (2020). Implementing routine outcome measurement in psychosocial interventions - A systematic review. European Journal of Social Work, 23, 790-808. https://doi.org/10.1080/13691457.2019.1602029
Martin, A.-M., Fishman, R., Baxter, L., & Ford, T. (2011). Practitioners’ attitudes towards the use of standardized diagnostic assessment in routine practice: A qualitative study in two child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 16, 407-420. https://doi.org/10.1177/1359104510366284
Meehan, T., Mccombes, S., Hatzipetrou, L., & Catchpoole, R. (2006). Introduction of routine outcome measures: Staff reactions and issues for consideration. Journal of Psychiatric and Mental Health Nursing, 13, 581-587. https://doi.org/10.1111/j.1365-2850.2006.00985.x
Mellor-Clark, J., Cross, S., Macdonald, J., & Skjulsvik, T. (2016). Leading horses to water: Lessons from a decade of helping psychological therapy services use routine outcome measurement to improve practice. Administration and Policy in Mental Health and Mental Health Services Research, 43(3), 279-285. https://doi.org/10.1007/s10488-014-0587-8
Michie, S., Atkins, L., & West, R. (2014). The behaviour change wheel: A guide to designing interventions. In A guide to designing interventions, 1st ed. (pp. 1003-110). Great Britain: Silverback Publishing.
Michie, S., Van Stralen, M. M., & West, R. (2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6(1), 42. https://doi.org/10.1186/1748-5908-6-42
Moltu, C., Veseth, M., Stefansen, J., Nøtnes, J. C., Skjølberg, Å., Binder, P. E., … Nordberg, S. S. (2018). This is what I need a clinical feedback system to do for me: A qualitative inquiry into therapists’ and patients’ perspectives. Psychotherapy Research, 28, 250-263. https://doi.org/10.1080/10503307.2016.1189619
Mundfrom, D. J., Shaw, D. G., & Ke, T. L. (2005). Minimum sample size recommendations for conducting factor analyses. International Journal of Testing, 5(2), 159-168. https://doi.org/10.1207/s15327574ijt0502_4
Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(1), 1-13. https://doi.org/10.1186/s13012-015-0242-0
Norman, S., Dean, S., Hansford, L., & Ford, T. (2014). Clinical practitioner’s attitudes towards the use of Routine Outcome Monitoring within Child and Adolescent Mental Health Services: A qualitative study of two Child and Adolescent Mental Health Services. Clinical Child Psychology and Psychiatry, 19, 576-595. https://doi.org/10.1177/1359104513492348
Østergård, O. K., Randa, H., & Hougaard, E. (2020). The effect of using the Partners for Change Outcome Management System as feedback tool in psychotherapy-A systematic review and meta-analysis. Psychotherapy Research, 30, 195-212. https://doi.org/10.1080/10503307.2018.1517949
Persons, J. B., Koerner, K., Eidelman, P., Thomas, C., & Liu, H. (2016). Increasing psychotherapists’ adoption and implementation of the evidence-based practice of progress monitoring. Behaviour Research and Therapy, 76, 24-31. https://doi.org/10.1016/j.brat.2015.11.004
Phelps, R., Eisman, E. J., & Kohout, J. (1998). Psychological practice and managed care: Results of the CAPP practitioner survey. Professional Psychology: Research and Practice, 29(1), 31-36. https://doi.org/10.1037/0735-7028.29.1.31
Rye, M., Rognmo, K., Aarons, G. A., & Skre, I. (2019). Attitudes towards the use of routine outcome monitoring of psychological therapies among mental health providers: The EBPAS-ROM. Administration and Policy in Mental Health and Mental Health Services Research, 46(6), 833-846. https://doi.org/10.1007/s10488-019-00968-5
Sharples, E., Qin, C., Goveas, V., Gondek, D., Deighton, J., Wolpert, M., & Edbrooke-Childs, J. (2017). A qualitative exploration of attitudes towards the use of outcome measures in child and adolescent mental health services. Clinical Child Psychology and Psychiatry, 22, 219-228. https://doi.org/10.1177/1359104516652929
Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment Failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78, 298-311. https://doi.org/10.1037/a0019247
Solstad, S. M., Castonguay, L. G., & Moltu, C. (2019). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research, 29(2), 157-170. https://doi.org/10.1080/10503307.2017.1326645
Tam, H. E., & Ronan, K. (2017). The application of a feedback-informed approach in psychological service with youth: Systematic review and meta-analysis. Clinical Psychology Review, 55, 41-55. https://doi.org/10.1016/j.cpr.2017.04.005
Trauer, T., Gill, L., Pedwell, G., & Slattery, P. (2006). Routine outcome measurement in public mental health-what do clinicians think? Australian Health, 30(2), 144-147. https://doi.org/10.1071/AH060144
Turchik, J. A., Karpenko, V., & Ogles, B. M. (2007). Further evidence of the utility and validity of a measure of outcome for children and adolescents. Journal of Emotional and Behavioral Disorders, 15(2), 119-128. https://doi.org/10.1177/10634266070150020201
Westmacott, R., & Hunsley, J. (2010). Reasons for terminating psychotherapy: A general population study. Journal of Clinical Psychology, 66, 965-977. https://doi.org/10.1002/jclp.20702
Willis, A., Deane, F. P., & Coombs, T. (2009). Improving clinicians’ attitudes toward providing feedback on routine outcome assessments. International Journal of Mental Health Nursing, 18(3), 211-215. https://doi.org/10.1111/j.1447-0349.2009.00596.x
Wolpert, M., Dalzell, K., Jacob, J., Bloxham, J., Barnard, M., Karwatzki, E., & Martin, K. (2017). Routine outcome monitoring in child and adolescent mental health in the United Kingdom at the individual and the systems level. In T. Rousmaniere, R. K. Goodyear, S. D. Miller & B. E. Wampold (Eds.), The cycle of excellence: Using deliberate practice to improve supervision and training (pp. 145-160). New York, NY: John Wiley and Sons.