Field supervisors' assessments of trainee performance: Are they reliable and valid?

The Performance Diagnostic Checklist - Human Services: Guidance for Assessment Administration

The Performance Diagnostic Checklist - Human Services (PDC-HS) is an assessment designed to assess the environmental variables contributing to employee performance concerns in human-service settings. Recent research has demonstrated that interventions indicated by the PDC-HS result in improved employee performance across several human-service settings and that the assessment has acceptable reliability and validity. Although PDC-HS-indicated interventions have been effective at increasing employee performance, there is a need for additional guidance when using the assessment given the limited nature of the original administration guidelines. Thus, the purpose of the current manuscript is to introduce additional guidance for use of the PDC-HS across a variety of situations.

Developmental trajectories of competency attainment amongst clinical psychology trainees across field placements

OBJECTIVE

This research aimed to describe the developmental trajectories of clinical psychology trainees across competency domains over multiple placements.

METHOD

Competency reviews of 252 trainees were completed at mid-placement and end-placement for up to four consecutive placements by 143 field supervisors. Competency was measured across multiple domains using the Clinical Psychology Practicum Competencies Rating Scale.

RESULTS

There was an overall ascending stepped pattern occurring across all competency domains from early to late placements. The starting point of competency ratings varied across domains with the largest discrepancy between Response to supervision (highest) and Intervention competencies (lowest). There were significant increases from mid-placement to end-placement for all competencies within each of the four placements. There were few significant decreases in competency between different placements and these were largely restricted to the transition from placement one to placement two.

CONCLUSIONS

This research has the potential to be used as a benchmark to indicate expected levels of competency attainment for trainees across their training, allowing for early identification of difficulties and more targeted remediation plans.

Brief Online Training with Standardised Vignettes Reduces Inflated Supervisor Ratings of Trainee Practitioner Competencies

Objective

Supervisor assessments of trainee competence are integral to ensuring that clinical psychology trainees reach competency benchmarks. The commonly used Clinical Psychology Practicum Competencies Rating Scale (CΨPRS) has been shown to elicit inflated ratings of competency. Hence, the aim of this study is to examine whether brief supervisor training reduces ratings by providing objective criteria with which supervisors can assess trainee competency.

Method

The ratings included were of 124 psychology trainees from nine Australian university clinical programmes. Of 170 supervisors, 32 completed the online training immediately prior to commencing the CΨPRS. Training required supervisors to rate the competency level described in five standardised vignettes (Beginner through to Competent). Vignette ratings, as determined by a panel of expert supervisors, were provided as feedback. A sixth calibration vignette was also rated (no feedback provided). Firstly, CΨPRS ratings from the trained and untrained supervisors were compared. Secondly, the difference between supervisor and expert ratings of the calibration vignettes were compared across trained and untrained groups.

Results

Trained supervisors provided lower CΨPRS ratings than untrained supervisors. In addition, trained supervisors (vs untrained supervisors) provided ratings of the calibration vignette that more accurately matched the ratings provided by the expert panel.

Conclusions

Brief online training using standardised vignettes was associated with lower CΨPRS ratings. The standardised vignettes helped calibrate supervisors’ ratings and likely attuned supervisors to the skills and competency levels that are expected at particular developmental stages. As a consequence, training appeared to reduce ratings, arguably resulting in more accurate assessments of trainee performance.

Trainee-therapists are not all equal: Examination of therapeutic efficiency, effectiveness and early client dropout after 12 months of clinical training

OBJECTIVE

Contemporary research demonstrates the feasibility of assessing therapeutic performance of trainee-therapists through the use of objective measures of client treatment outcome. Further, significant variation between individual therapists based on their client treatment outcomes has been demonstrated. This study sets out to determine whether a reliable composite measure of therapeutic efficiency, effectiveness and early dropout can be developed and used to objectively compare trainee-therapists against each other.

DESIGN AND METHODS

Treatment outcomes of 611 clients receiving treatment from 58 trainee-therapists enrolled in a professional training programme were tracked with the OQ-45.2 over a 6-year period to assess therapeutic efficiency, therapeutic effectiveness and early client dropout.

RESULTS

Significant variation between trainee-therapists was observed for each index. Findings of a moderately strong correlation between therapeutic efficiency and effectiveness enabled the ranking of trainee-therapists based upon a composite measure of these indexes. A non-significant correlation was found between early client dropout and measures of therapeutic effectiveness and efficiency.

CONCLUSIONS

The findings stress the importance of utilizing objective measures to track the treatment outcomes. Despite all trainee-therapists being enrolled in the same training programme, significant variation between trainee-therapists' therapeutic efficiency and effectiveness was found to exist.

PRACTITIONER POINTS

Developing of potential benchmarking tools that enable trainee-therapists, supervisors and educational institutions to quickly assess therapeutic performance can become part of a holistic assessment of a trainee-therapist's clinical development. Despite an inherent optimistic belief that therapists do not cause harm, there appears to be a small and significant proportion of trainee-therapists who consistently evidence little therapeutic change. Considerable variability in trainee-therapists' therapeutic efficiency and effectiveness can exist in the one training programme. Early client dropout may not be associated with therapists' therapeutic effectiveness and efficiency.

Treatment adherence and competency ratings among therapists, supervisors, study-related raters and external raters in a clinical trial of a 12-step facilitation for stimulant users

This study investigated the correspondence among four groups of raters on adherence to STAGE-12, a manualized 12-step facilitation (TSF) group and individual treatment targeting stimulant abuse. The four rater groups included the study therapists, supervisors, study-related ("TSF expert") raters, and non-project related ("external") raters. Results indicated that external raters rated most critically mean adherence - the mean of all the adherence items - and global performance. External raters also demonstrated the highest degree of reliability with the designated expert. Therapists rated their own adherence lower, on average, than did supervisors and TSF expert raters, but therapist ratings also had the poorest reliability. Findings highlight the challenges in developing practical, but effective methods of fidelity monitoring for evidence based practice in clinical settings. Recommendations based on study findings are provided.

Assessing Trainee Clinical Psychologists' Clinical Competence

Within UK clinical psychology training, traditionally used methods for assessing the development of trainees' clinical competence have taken the form of evaluations of trainee knowledge regarding their clinical practice, either in the form of written assignments or via discussion in the supervisory relationship. Whilst direct observation of trainees is expected and required by supervisors on practice placements, frequency of observation varies, and supervisors can experience difficulties occupying both supervisory and evaluatory roles. In addition, calls from within the clinical psychology training community have expressed a need for a consistent and formalised in vivo assessment of trainees' clinical competency (Gallichan & Mitchell, 2008). This article reports on a psychometric evaluation of the Clinical Skills Assessment Rating Form (CSA-RF), a seven domain measure of clinical competence developed at the University of Leicester Doctoral Programme in Clinical Psychology for use with live or videotaped clinical material. Results suggest that the CSA-RF has acceptable levels of internal consistency, although interrater reliability requires further investigation. Factor analysis revealed a five domain structure. From this, a revised CSA-RF has been developed which aims to be a valid and practicable initial means by which trainees' clinical competence can be assessed.