The use of standardised patients to assess clinical competence: does practice make perfect?

The assessment of medical competencies

OBJECTIVE

To describe the most widely used tools in the assessment of medical competencies, analyse their prevalence of use, their advantages and disadvantages and propose an appropriate model for our context.

METHODS

We conducted a narrative review of articles from MEDLINE, following the PRISM protocol, and analysed a total of 62 articles.

RESULTS

The assessment of competencies is heterogeneous, especially in the educational and professional settings. The specific and technical competencies acquired during university education are mainly assessed using the objective structured clinical assessment. In the professional setting, core competencies are assessed using the 360° technique.

CONCLUSIONS

We need a rigorous empiric comparison of the efficiency of the tools according to the type of competency. We propose a competency management model for the «undergraduate/graduate/active professional» continuum, whose goal is to improve training and professional practice and thereby increase the quality of patient care.

Development of an Emergency Medicine Simulation Fellowship Consensus Curriculum: Initiative of the Society for Academic Emergency Medicine Simulation Academy

OBJECTIVES

There is currently no consolidated list of existing simulation fellowship programs in emergency medicine (EM). In addition, there are no universally accepted or expected standards for core curricular content. The objective of this project is to develop consensus-based core content for EM simulation fellowships to help frame the critical components of such training programs.

METHODS

This paper delineates the process used to develop consensus curriculum content for EM simulation fellowships. EM simulation fellowship curricula were collected. Curricular content was reviewed and compiled by simulation experts and validated utilizing survey methodology, and consensus was obtained using a modified Delphi methodology.

RESULTS

Fifteen EM simulation fellowship curricula were obtained and analyzed. Two rounds of a modified Delphi survey were conducted. The final proposed core curriculum content contains 47 elements in nine domains with 14 optional elements.

CONCLUSION

The proposed consensus content will provide current and future fellowships a foundation on which to build their own specific and detailed fellowship curricula. Such standardization will ultimately increase the transparency of training programs for future trainees and potential employers.

Adapting the helpful responses questionnaire to assess communication skills involved in delivering contingency management: preliminary psychometrics

A paper/pencil instrument, adapted from Miller and colleagues' (1991) Helpful Responses Questionnaire (HRQ), was developed to assess clinician skill with core communicative aspects involved in delivering contingency management (CM). The instrument presents a single vignette consisting of six points of client dialogue to which respondents write 'what they would say next.' In the context of an implementation/effectiveness hybrid trial, 19 staff clinicians at an opiate treatment program completed serial training outcome assessments before, following, and three months after CM training. Assessments included this adaptation of the HRQ, a multiple-choice CM knowledge test, and a recorded standardized patient encounter scored for CM skillfulness. Study results reveal promising psychometric properties for the instrument, including strong scoring reliability, internal consistency, concurrent and predictive validity, test-retest reliability and sensitivity to training effects. These preliminary findings suggest the instrument is a viable, practical method to assess clinician skill in communicative aspects of CM delivery.

Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients

OBJECTIVE

The goal of measuring therapist adherence is to determine whether a therapist can perform a given treatment. Yet, the evaluation of therapist behaviors in most clinical trials is limited. Typically, randomized trials have few therapists and minimize therapist variability through training and supervision. Furthermore, therapist adherence is confounded with uncontrolled differences in patients across therapists. Consequently, the extent to which adherence measures capture differences in actual therapist adherence versus other sources of variance is unclear.

METHOD

We estimated intra-class correlations (ICCs) for therapist adherence in sessions with real and standardized patients (RPs and SPs), using ratings from a motivational interviewing (MI) dissemination trial (Baer et al., 2009) in which 189 therapists recorded 826 sessions with both patient types. We also examined the correlations of therapist adherence between SP and RP sessions, and the reliability of therapist level adherence scores with generalizability coefficients (GCs).

RESULTS

ICCs for therapist adherence were generally large (average ICC for SPs = .44; average ICC for RPs = .40), meaning that a given therapist's adherence scores were quite similar across sessions. Both ICCs and GCs were larger for SP sessions compared to RPs on global measures of MI adherence, such as Empathy and MI Spirit. Correlations between therapist adherence with real and standardized patients were moderate to large on 3 of 5 adherence measures.

CONCLUSION

Differences in therapist-level adherence ratings were substantial, and standardized patients have promise as tools to evaluate therapist behavior.

The communication skills course for second year medical students at Hannover Medical School: An evaluation study based on students' self-assessments

In the model medical curriculum HannibaL at Hannover Medical School (MHH, Hannover, Germany), communication skills in taking case histories and disclosing diagnoses (breaking bad news) are assessed through an objective structured clinical examination (OSCE). This is part of the examinations which at the MHH represent the equivalent to the First Part of the Medical Examinations. The second year doctor-patient communication course preparing for these examinations was evaluated during the 2009/10 academic year.

Using questionnaires specific to the learning objectives, learning needs were assessed, pre-post comparisons of self-assessed competencies were performed and key teaching methods were evaluated (5-point Likert scales, “5”=fully agree). At T0 (start of the course) 267 students participated (response rate: 93.7%), of which 180 filled out the T1 questionnaire during the last session of the course (67.4%). Within-subject analyses of variance and paired t-tests were conducted.

The highest learning needs were found for the “to show how”-items regarding history taking and disclosing diagnoses (M=4.4). The T1-T0 comparisons showed the greatest improvements for history taking (“to know how”: mean difference = +1.7, “to show how”: +1.8, p<.0001 as with all tests) and the “to know how”-item regarding the disclosure of diagnoses (+1.6), followed by the “to show how”-items on disclosing a diagnosis (+1.4), shared decision making (+1.2), self-assessing one’s own strengths/weaknesses (+1.0) and confidently approaching new patients (+0.7). Students with T0 values of 1 or 2 on the respective scales improved on average by 2.2 points across all items, students with the value of 3 by 1.1, and from 4 or 5 by 0.1. Methodically, the use of simulated patients was rated the most helpful (M=4.8, 87% with the scale value 5).

This doctor-patient communication course is associated with substantial improvements regarding all key learning objectives. Regarding methods, the deployed simulated patients (2-4 per 10-student-course group in 3 of the 7 course sessions, respectively) were rated the most helpful. The present evaluation calls for both further development of the doctor-patient communication curriculum at the MHH and joint activities across medical schools, which are discussed at the end of the paper.

Enhancing objectivity in pastoral education: use of standardized patients in video simulation

Measurement of student learning outcomes in clinical pastoral education has historically been assessed through student self-report instruments and subjective evaluation by CPE supervisors. Valid quantitative measurement is limited. This study utilizes standardized patients and a behavioral interview checklist to measure the impact of an initial summer unit of CPE training. Results demonstrate measurable change in student interview style behavior suggesting that simulation utilizing standardized patients is a valid objective measure of students' behavioral communication styles.

The reliability of summative judgements based on objective structured clinical examination cases distributed across the clinical year

CONTEXT

Objective structured clinical examinations (OSCEs) can be used for formative and summative evaluation. We sought to determine the generalisability of students' summary scores aggregated from formative OSCE cases distributed across 5 clerkships during Year 3 of medical school.

METHODS

Five major clerkships held OSCEs with 2-4 cases each during their rotations. All cases used 15-minute student-standardised patient encounters and performance was assessed using clinical and communication skills checklists. As not all students completed every clerkship or OSCE case, the generalisability (G) study was an unbalanced student x (case : clerkship) design. After completion of the G study, a decision (D) study was undertaken and phi (phi) values for different cut-points were calculated.

RESULTS

The data for this report were collected over 2 academic years involving 262 Year 3 students. The G study found that 9.7% of the score variance originated from the student, 3.1% from the student-clerkship interaction, and 87.2% from the student-case nested within clerkship effect. Using the variance components from the G study, the D study suggested that if students completed 3 OSCE cases in each of the 5 different clerkships, the reliability of the aggregated scores would be 0.63. The phi, calculated at a cut-point 1 standard deviation below the mean, would be approximately 0.85.

CONCLUSIONS

Aggregating case scores from low stakes OSCEs within clerkships results in a score set that allows for very reliable decisions about which students are performing poorly. Medical schools can use OSCE case scores collected over a clinical year for summative evaluation.