Assessing Therapist Competence: Development of a Performance-Based Measure and Its Comparison With a Web-Based Measure

Quality of cognitive-behavioural therapy in routine psychiatric care: therapist adherence and competence, and patient outcomes for depression and anxiety disorders

BACKGROUND

Quality of care is essential for the dissemination of evidence-based practices, yet therapist adherence and competence are seldom assessed. We examined the quality of delivery of cognitive-behavioural therapy (CBT) in routine psychiatric care for depression and anxiety disorders, considering therapist adherence and competence, and therapy effectiveness, as well as their associations.

METHODS

Twenty-nine therapists recruited 85 patients with a principal diagnosis of depression or anxiety disorder from two routine psychiatric outpatient clinics in Stockholm, Sweden. Therapist adherence was assessed mid-CBT by observers and post-CBT by patients and therapists, respectively, using an instrument developed as part of the present study. Therapist competence was assessed using role-plays with a standardised patient. Patients rated symptoms, functional impairment, and global health pre- and post-CBT. Linear mixed models were used to analyse associations.

RESULTS

Therapist adherence was high according to patients, moderate to high according to therapists, and moderate according to observers. Most therapists demonstrated competence in CBT, as assessed using the Cognitive Therapy Scale-Revised (M = 40.5, SD = 6.5; 76% passed the ≥ 36 points competence threshold). Patients improved significantly from pre- to post-CBT across outcome measures (Cohen's ds = 0.80 - 1.36). Neither therapist adherence nor competence was associated with patient outcomes.

CONCLUSIONS

In routine psychiatric care, therapists delivered CBT with adherence, competence, and improvements for patients with depression and anxiety disorders, on par with previous research results in controlled settings. The implications for quality assessment and improvement are discussed.

TRIAL REGISTRATION

ClinicalTrials.gov NCT03625024 10/08/2018.

Recent Advances in Cognitive-Behavioural Therapy for Eating Disorders (CBT-ED)

PURPOSE OF REVIEW

Eating disorders require more effective therapies than are currently available. While cognitive behavioural therapy for eating disorders (CBT-ED) has the most evidence to support its effectiveness, it requires substantial improvement in order to enhance its reach and outcomes, and to reduce relapse rates. Recent years have seen a number of noteworthy developments in CBT-ED, which are summarised in this paper.

RECENT FINDINGS

The key advances identified here include: improvements in the efficiency and availability of CBT-ED; expansion of applicability to younger cases across durations of eating disorder; and new methodologies. There have been important recent advances in the field of CBT-ED. However, it is important to stress that there remain gaps in our evidence base and clinical skills, and suggestions are made for future research and clinical directions.

Assessing clinician competence in the delivery of cognitive-behavioural therapy for eating disorders: development of the Cognitive-Behavioural Therapy Scale for Eating Disorders (CBTS-ED)

Evidence-based cognitive-behaviour therapy for eating disorders (CBT-ED) differs from other forms of CBT for psychological disorders, making existing generic CBT measures of therapist competence inadequate for evaluating CBT-ED. This study developed and piloted the reliability of a novel measure of therapist competence in this domain-the Cognitive Behaviour Therapy Scale for Eating Disorders (CBTS-ED). Initially, a team of CBT-ED experts developed a 26-item measure, with general (i.e. present in every session) and specific (context- or case-dependent) items. To determine statistical properties of the measure, nine CBT-ED experts and eight non-experts independently observed six role-played mock CBT-ED therapy sessions, rating the therapists' performance using the CBTS-ED. The inter-item consistency (Cronbach's alpha and McDonald's omega) and inter-rater reliability (ICC) were assessed, as appropriate to the clustering of the items. The CBTS-ED demonstrated good internal consistency and moderate/good inter-rater reliability for the general items, at least comparable to existing generic CBT scales in other domains. An updated version is proposed, where five of the 16 "specific" items are reallocated to the general group. These preliminary results suggest that the CBTS-ED can be used effectively across both expert and non-expert raters, though less experienced raters might benefit from additional training in its use.

Knowledge of Problem Solving (KOPS) Scale: Design and Evaluation of a Digitally Administered Competence Measure for a Common Practice Element in Task-Shared Youth Mental Health Interventions

There are few reliable and feasible quality assurance methods to support scaling up of psychological interventions delivered by non-specialist providers. This paper reports on the phased development and validation of a digitally administered Knowledge of Problem Solving (KOPS) measure to assess competencies associated with a "task-shared" problem-solving intervention for adolescents with diverse mental health problems in India. Phase 1 established key competencies required to deliver the intervention, followed by item generation for a corresponding knowledge-based competency measure that could be administered efficiently through e-learning systems. In phase 2, items were refined based on responses from an "experienced" reference sample comprising 17 existing counsellors with direct experience of the problem-solving intervention, and a "novice" sample with 14 untrained university students and NGO staff. In phase 3, we evaluated two parallel versions of the measure in a validation sample (N = 277) drawn from universities and NGOs. The resulting 17-item measure was structured around a hypothetical case, followed by multiple-choice questions that asked about the most appropriate response to a practice-based scenario. The difficulty level of the test items was well matched to the ability level of participants (i.e. most items being of moderate difficulty and few items being easy or difficult). Only one item showed a negative discrimination index and was removed from the 17-item forms. The final 16-item version of the KOPS measure provides a scalable digital method to assess key psychotherapeutic competencies among non-specialists, particularly in relation to a transdiagnostic problem-solving intervention. Similar formats could be deployed more widely alongside e-learning programmes to expand the global workforce capable of delivering evidence-based psychological interventions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s41347-023-00356-9.

The Impact of an Online Post-Graduate Interdisciplinary Mental Health Programme on Graduates' Confidence and Practice

This study aimed to examine the experiences of graduates of online interdisciplinary postgraduate mental health programmes in Australia. The program was delivered in 6-week terms. Seven graduates from diverse backgrounds were interviewed about their experiences with the course and its impact on their practice, confidence, professional identity, views on mental health service users, and their motivation for additional learning. The interviews were recorded and transcribed and underwent thematic content analysis. The graduates reported an increase in confidence and knowledge after completing the course, which led to a change in their views and attitudes towards service users. They appreciated the examination of psychotherapies and motivational interviewing, and applied their newly acquired skills and knowledge in their practice. The course was found to have improved their clinical practice. This study highlights a departure from traditional pedagogical approaches in mental health skill acquisition, as the entire program was delivered online. There is a need for further research to determine who might benefit most from this mode of delivery and to verify the competencies acquired by graduates in real-world situations. Online mental health courses are a feasible option and have been positively received by graduates. To enable graduates to participate in transforming mental health services, systemic change and recognition of their capabilities, particularly those from non-traditional backgrounds, is required. The results of this study suggest the potential for online postgraduate programs to play a significant role in transforming mental health services.

Adaptation and translation of a scalable measure for assessing health worker competence to deliver a brief psychological intervention: A case study from central India of its comparison to a performance-based measure

Efforts to reduce the global burden of common mental disorders have focused on scaling up evidence-based training programs for non-specialist providers to deliver brief psychological interventions. To evaluate these provider training programs, appropriate and scalable assessments of competency need to be developed alongside them. We followed a systematic approach for the cultural adaptation and translation into Hindi of a valid, English, multiple-choice applied knowledge measure to assess non-specialists' competence to deliver a brief psychological intervention for depression in rural India. We then explored the relationship between the performance of 30 non-specialist providers on the same written measure compared with a structured performance-based measure consisting of two role-plays. The results of the multiple-choice assessment had an overall mean score of 37.40 (SD = 11.31) compared to the mean scores of role-play A (the easier role-play) of 43.25 (SD = 14.50) and role-play B (the more difficult role-play) of 43.25 (SD = 13.00). Role-play performance-based measures and written applied knowledge measures represent different approaches with unique strengths and challenges to measuring competence. Scaling up training programs requires the development of scalable methods for competency assessment. Exploring the relationship between these two measures, our team found no apparent differences between the two modes of assessment. Continued comparison of these approaches is needed to determine the consistency of outcomes across the two formats and to link the scores on these measures with clinical performance as reflected by the quality of care and patient outcomes.Trial Registration: ClinicalTrials.gov Identifier: NCT04157816; 8th November 2019.

Development of competence in cognitive behavioural therapy and the role of metacognition among clinical psychology and psychotherapy students

BACKGROUND

There is a paucity of research on therapist competence development following extensive training in cognitive behavioural therapy (CBT). In addition, metacognitive ability (the knowledge and regulation of one's cognitive processes) has been associated with learning in various domains but its role in learning CBT is unknown.

AIMS

To investigate to what extent psychology and psychotherapy students acquired competence in CBT following extensive training, and the role of metacognition.

METHOD

CBT competence and metacognitive activity were assessed in 73 psychology and psychotherapy students before and after 1.5 years of CBT training, using role-plays with a standardised patient.

RESULTS

Using linear mixed modelling, we found large improvements of CBT competence from pre- to post-assessment. At post-assessment, 72% performed above the competence threshold (36 points on the Cognitive Therapy Scale-Revised). Higher competence was correlated with lower accuracy in self-assessment, a measure of metacognitive ability. The more competent therapists tended to under-estimate their performance, while less competent therapists made more accurate self-assessments. Metacognitive activity did not predict CBT competence development. Participant characteristics (e.g. age, clinical experience) did not moderate competence development.

CONCLUSIONS

Competence improved over time and most students performed over the threshold post-assessment. The more competent therapists tended to under-rate their competence. In contrast to what has been found in other learning domains, metacognitive ability was not associated with competence development in our study. Hence, metacognition and competence may be unrelated in CBT or perhaps other methods are required to measure metacognition.

The why, what, when, who and how of assessing CBT competence to support lifelong learning

Assessment of cognitive behaviour therapy (CBT) competence is a critical component in ensuring optimal clinical care, supporting therapists’ skill acquisition, and facilitating continuing professional development. This article provides a framework to support trainers, assessors, supervisors and therapists when making decisions about selecting and implementing effective strategies for assessing CBT competence. The framework draws on the existing evidence base to address five central questions: Why assess CBT competence?; What is CBT competence?; When should CBT competence be assessed?; Who is best placed to assess CBT competence?; and How should CBT competence be assessed? Various methods of assessing CBT competence are explored and the potential benefits and challenges are outlined. Recommendations are made about which approach to use across different contexts and how to use these effectively to facilitate the acquisition, enhancement and evaluation of CBT knowledge and skills.

Key learning aims

After reading this article you will be able to: (1)

Role-Play Assessment of Therapist Adherence and Skill in Implementation of Trauma-Focused Cognitive-Behavioral Therapy

Numerous efforts are underway to train clinicians in evidence-based practices. Unfortunately, the field has few practical measures of therapist adherence and skill with which to judge the success of these training and implementation efforts. One possible assessment method is using behavioral rehearsal, or role-play, as an analogue for therapist in-session behavior. The current study describes aspects of reliability, validity and utility of a behavioral role-play assessment developed to evaluate therapist adherence and skill in implementing Trauma-Focused Cognitive-Behavioral Therapy (TF-CBT). TF-CBT role-play assessments were conducted with a sample of 43 therapists as part of a larger training study. The TF-CBT role-play assessments were independently coded for TF-CBT adherence and skill by a certified TF-CBT trainer and three clinical psychology doctoral students. Findings indicated good interrater reliability for the individual items (ICC: M = .71, SD = .15). Regarding utility, 67.19% (n = 43/64) of contacted therapists completed the role-play assessment, which took an average of 30 min (M = 31.42, SD = 5.65) to complete and 60 min (M = 62.84, SD = 11.31) to code. Therapists with a master's degree were more likely to complete the role-play assessment than those with other degrees but no other differences in demographic variables, practice characteristics, or TF-CBT knowledge or training were found between participants and nonparticipants. Role-play assessments may offer an alternative to observational coding for assessing therapist adherence and skill, particularly in contexts where session recordings are not feasible.

Assessing health worker competence to deliver a brief psychological treatment for depression: development and validation of a scalable measure

Increased interest in disseminating and implementing psychological treatments has focused on the need for evidence-based training programs for providers, especially those without specialized training. To evaluate provider-training programs, validated outcome measures are necessary; however, the scalable measurement of training outcomes has been largely overlooked. Current methods of assessing providers' ability to deliver psychological treatments are generally time-consuming and costly, representing a major bottleneck in scaling up mental health care for commonly occurring disorders such as depression. The present study describes the development and initial validation of a scalable measure for assessing provider competence in delivering a brief behavioral activation treatment for depression, called the Healthy Activity Program, adapted for primary care settings. The measure focuses on testing knowledge about the treatment and applied knowledge regarding how to skillfully deliver the treatment, both essential features of competence. The measure was tested on a sample of 531 respondents with a variety of educational levels and professional backgrounds and found to meet the requirements of the Rasch model. Three versions of the measure each of equal difficulty were derived to allow repeat testing of training outcomes over time. A scalable measure of provider competence is an essential first step towards supporting the wider dissemination and implementation of brief psychological interventions for depression, especially in low-resource settings.

Digital Training for Non-Specialist Health Workers to Deliver a Brief Psychological Treatment for Depression in Primary Care in India: Findings from a Randomized Pilot Study

Introduction: Task sharing holds promise for scaling up depression care in countries such as India, yet requires training large numbers of non-specialist health workers. This pilot trial evaluated the feasibility and acceptability of a digital program for training non-specialist health workers to deliver a brief psychological treatment for depression. Methods: Participants were non-specialist health workers recruited from primary care facilities in Sehore, a rural district in Madhya Pradesh, India. A three-arm randomized controlled trial design was used, comparing digital training alone (DGT) to digital training with remote support (DGT+), and conventional face-to-face training. The primary outcome was the feasibility and acceptability of digital training programs. Preliminary effectiveness was explored as changes in competency outcomes, assessed using a self-reported measure covering the specific knowledge and skills required to deliver the brief psychological treatment for depression. Outcomes were collected at pre-training and post-training. Results: Of 42 non-specialist health workers randomized to the training programs, 36 including 10 (72%) in face-to-face, 12 (86%) in DGT, and 14 (100%) in DGT+ arms started the training. Among these participants, 27 (64%) completed the training, with 8 (57%) in face-to-face, 8 (57%) in DGT, and 11 (79%) in DGT+. The addition of remote telephone support appeared to improve completion rates for DGT+ participants. The competency outcome improved across all groups, with no significant between-group differences. However, face-to-face and DGT+ participants showed greater improvement compared to DGT alone. There were numerous technical challenges with the digital training program such as poor connectivity, smartphone app not loading, and difficulty navigating the course content-issues that were further emphasized in follow-up focus group discussions with participants. Feedback and recommendations collected from participants informed further modifications and refinements to the training programs in preparation for a forthcoming large-scale effectiveness trial. Conclusions: This study adds to mounting efforts aimed at leveraging digital technology to increase the availability of evidence-based mental health services in primary care settings in low-resource settings.

Development and evaluation of a measure of treatment knowledge in guided self-help for eating disorders in a sample of healthcare students and professionals

PURPOSE

The current study describes the development of a measure designed to assess treatment-specific competence in guided self-help (GSH) for eating disorders. The aim is to assess healthcare professionals' understanding of a popular treatment manual and associated material.

METHODS

After initial item development from a review of relevant literature, a range of healthcare staff and students (N = 127) completed a knowledge questionnaire. From these data, estimates of psychometric properties were made and a subset of the original sample completed the measure again after 6 weeks.

RESULTS

The final questionnaire consists of 40 items, demonstrating acceptable content validity, internal consistency, and reliability. Significant differences in the number of questions answered correctly were observed between experts in GSH and those with less experience.

CONCLUSIONS

This questionnaire offers a means of assessing therapist knowledge of GSH which demonstrates good psychometric properties. Further testing of this instrument is required to establish its full applicability.

LEVEL OF EVIDENCE

Level IV.

Innovative Models in Mental Health Delivery Systems: Task Sharing Care with Non-specialist Providers to Close the Mental Health Treatment Gap

PURPOSE OF REVIEW

Most people do not have access to adequate mental health care, and lack of skilled human resources is a major factor. We provide a narrative review of approaches to implementing task sharing-engaging non-specialist providers-to deliver mental health care.

RECENT FINDINGS

There is strong evidence both for the effectiveness of task sharing as a means of delivering care for a range of conditions across settings and for the effectiveness of non-specialist providers and health workers in delivering elements of culturally adapted psychosocial and psychological interventions for common and severe mental disorders. Key approaches to facilitate task sharing of care include balanced care, collaborative care, sustained training and supervision, use of trans-diagnostic interventions based on a dimensional approach to wellness and illness, and the use of emerging digital technologies. Non-specialist providers and health workers are well positioned to deliver evidence-based interventions for mental disorders, and a variety of delivery approaches can support, facilitate, and sustain this innovation. These approaches should be used, and evaluated, to increase access to mental health services.

Increasing the Availability of Psychological Treatments: A Multinational Study of a Scalable Method for Training Therapists

BACKGROUND

One of the major barriers to the dissemination and implementation of psychological treatments is the scarcity of suitably trained therapists. A highly scalable form of Web-centered therapist training, undertaken without external support, has recently been shown to have promise in promoting therapist competence.

OBJECTIVE

The aim of this study was to conduct an evaluation of the acceptability and effectiveness of a scalable independent form of Web-centered training in a multinational sample of therapists and investigate the characteristics of those most likely to benefit.

METHODS

A cohort of eligible therapists was recruited internationally and offered access to Web-centered training in enhanced cognitive behavioral therapy, a multicomponent, evidence-based, psychological treatment for any form of eating disorder. No external support was provided during training. Therapist competence was assessed using a validated competence measure before training and after 20 weeks.

RESULTS

A total of 806 therapists from 33 different countries expressed interest in the study, and 765 (94.9%) completed a pretraining assessment. The median number of training modules completed was 15 out of a possible 18 (interquartile range, IQR: 4-18), and 87.9% (531/604) reported that they treated at least one patient during training as recommended. Median pretraining competence score was 7 (IQR: 5-10, range: 0-19; N=765), and following training, it was 12 (IQR: 9-15, range: 0-20; N=577). The expected change in competence scores from pretraining to posttraining was 3.5 (95% CI 3.1-3.8; P<.001). After training, 52% (300/574) of therapists with complete competence data met or exceeded the competence threshold, and 45% (95% CI 41-50) of those who had not met this threshold before training did so after training. Compliance with training predicted both an increase in competence scores and meeting or exceeding the competence threshold. Expected change in competence score increased for each extra training module completed (0.19, 95% CI 0.13-0.25), and those who treated a suitable patient during training had an expected change in competence score 1.2 (95% CI 0.4-2.1) points higher than those who did not. Similarly, there was an association between meeting the competence threshold after training and the number of modules completed (odds ratio, OR=1.11, 95% CI 1.07-1.15), and treating at least one patient during training was associated with competence after training (OR=2.2, 95% CI 1.2-4.1).

CONCLUSIONS

Independent Web-centered training can successfully train large numbers of therapists dispersed across a wide geographical area. This finding is of importance because the availability of a highly scalable method of training potentially increases the number of people who might receive effective psychological treatments.