The challenge of creating new OSCE measures to capture the characteristics of expertise

Simulation-based summative assessment in healthcare: an overview of key principles for practice

BACKGROUND

Healthcare curricula need summative assessments relevant to and representative of clinical situations to best select and train learners. Simulation provides multiple benefits with a growing literature base proving its utility for training in a formative context. Advancing to the next step, "the use of simulation for summative assessment" requires rigorous and evidence-based development because any summative assessment is high stakes for participants, trainers, and programs. The first step of this process is to identify the baseline from which we can start.

METHODS

First, using a modified nominal group technique, a task force of 34 panelists defined topics to clarify the why, how, what, when, and who for using simulation-based summative assessment (SBSA). Second, each topic was explored by a group of panelists based on state-of-the-art literature reviews technique with a snowball method to identify further references. Our goal was to identify current knowledge and potential recommendations for future directions. Results were cross-checked among groups and reviewed by an independent expert committee.

RESULTS

Seven topics were selected by the task force: "What can be assessed in simulation?", "Assessment tools for SBSA", "Consequences of undergoing the SBSA process", "Scenarios for SBSA", "Debriefing, video, and research for SBSA", "Trainers for SBSA", and "Implementation of SBSA in healthcare". Together, these seven explorations provide an overview of what is known and can be done with relative certainty, and what is unknown and probably needs further investigation. Based on this work, we highlighted the trustworthiness of different summative assessment-related conclusions, the remaining important problems and questions, and their consequences for participants and institutions of how SBSA is conducted.

CONCLUSION

Our results identified among the seven topics one area with robust evidence in the literature ("What can be assessed in simulation?"), three areas with evidence that require guidance by expert opinion ("Assessment tools for SBSA", "Scenarios for SBSA", "Implementation of SBSA in healthcare"), and three areas with weak or emerging evidence ("Consequences of undergoing the SBSA process", "Debriefing for SBSA", "Trainers for SBSA"). Using SBSA holds much promise, with increasing demand for this application. Due to the important stakes involved, it must be rigorously conducted and supervised. Guidelines for good practice should be formalized to help with conduct and implementation. We believe this baseline can direct future investigation and the development of guidelines.

The role of training in student examiner rating performance in a student-led mock OSCE

INTRODUCTION

Peer assessments are increasingly prevalent in medical education, including student-led mock Objective Structured Clinical Examinations (OSCE). While there is some evidence to suggest that examiner training may improve OSCE assessments, few students undergo training before becoming examiners. We sought to evaluate an examiner training programme in the setting of a student-led mock OSCE.

METHODS

A year‑2 mock OSCE comprised of history taking (Hx) and physical examination (PE) stations was conducted involving 35 year‑3 (Y3) student examiners and 21 year‑5 (Y5) student examiners who acted as reference examiners. Twelve Y3 student-examiners attended an OSCE examiner training programme conducted by senior faculty. During the OSCE, Y3 and Y5 student examiners were randomly paired to grade the same candidates and scores were compared. Scores for checklist rating (CR) and global rating (GR) domains were assigned for both Hx and PE stations.

RESULTS

There was moderate to excellent correlation between Y3 and Y5 student examiners for both Hx (ICC 0.71-0.96) and PE stations (ICC 0.71-0.88) across all domains. For both Hx and PE stations, GR domain had poorer correlation than CR domains. Examiner training resulted in better correlations for PE but not Hx stations. Effect sizes were lower than the minimum detectible effect (MDE) sizes for all comparisons made.

DISCUSSION

Y3 student examiners are effective substitutes for Y5 student examiners in a Y2 mock OSCE. Our findings suggest that examiner training may further improve marking behaviour especially for PE stations. Further studies with larger sample sizes are required to further evaluate the effects of dedicated examiner training.

Competency-Based Education in Minimally Invasive and Robotic Colorectal Surgery

Minimally invasive and robotic techniques have become increasingly implemented into surgical practice and are now an essential part of the foundational skills of training colorectal surgeons. Over the past 5 years there has been a shift in the surgical educational paradigm toward competency-based education (CBE). CBE recognizes that trainees learn at different rates but regardless, are required to meet a competent threshold of performance prior to independent practice. Thus, CBE attempts to replace the traditional "time" endpoint of training with "performance." Although conceptually sensible, implementing CBE has proven challenging. This article will define competence, outline appropriate assessment tools to assess technical skill, and review the literature on the number of cases required to achieve competence in colorectal procedures while outlining the barriers to implementing CBE.

Evaluation of a new e-learning resource for calibrating OSCE examiners on the use of rating scales

INTRODUCTION

Rating scales have been described as better at assessing behaviours such as professionalism during Objective Structured Clinical Examinations (OSCEs). However, there is an increased need to train and calibrate staff on their use prior to student assessment.

MATERIAL AND METHODS

An online e-learning package was developed and made available to all examiners at the Institute of Dentistry at the University of Aberdeen. The package included videos of three OSCE stations (medical emergency, rubber dam placement and handling a complaint) which were recorded in two different scenarios; (excellent and unsatisfactory candidate). These videos were recorded to meet a pre-defined marking score. The examiners were required to mark the six videos using pre-set marking criteria (checklist and rating scales). The rating scales included professionalism, general clinical ability and/or communication skills. For each video, examiners were given four possible options (unsatisfactory, borderline, satisfactory or excellent), and they were provided with a description for each domain. They were also required to complete a questionnaire to gather their views on the use of this e-learning environment.

RESULTS

Fifteen examiners completed the task. The total scores given were very similar to the expected scores for the medical emergency and complaint stations; however, this was not the case for the rubber dam station (P-value .017 and .036). This could be attributed to some aspects of the placement of the rubber dam being unclear as commented on in the examiners questionnaires. There was consistency in the selection of marks on the rating scales (inter-examiner correlation ranged between 0.916 and 0.979).

CONCLUSION

Further studies are required on the field of e-learning training to calibrate examiners for practical assessment; however, this study provides preliminary evidence to support the use of videos as part of an online training package to calibrate OSCE examiners on the use of rating scales.

A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE

BACKGROUND

Faculty members need to assess resident performance using the Accreditation Council for Graduate Medical Education Milestones.

OBJECTIVE

In this randomized study we used an objective structured clinical examination (OSCE) around the disclosure of an adverse event to determine whether use of a checklist improved the quality of milestone assessments by faculty.

METHODS

In 2013, a total of 20 anesthesiology faculty members from 3 institutions were randomized to 2 groups to assess 5 videos of trainees demonstrating advancing levels of competency on the OSCE. One group used milestones alone, and the other used milestones plus a 13-item checklist with behavioral anchors based on ideal performance. We classified faculty ratings as either correct or incorrect with regard to the competency level demonstrated in each video, and then used logistic regression analysis to assess the effect of checklist use on the odds of correct classification.

RESULTS

Thirteen of 20 faculty members rated assessing performance using milestones alone as difficult or very difficult. Checklist use was associated with significantly greater odds of correct classification at entry level (odds ratio [OR] = 9.2, 95% confidence interval [CI] 4.0-21.2) and at junior level (OR = 2.7, 95% CI 1.3-5.7) performance. For performance at other competency levels checklist use did not affect the odds of correct classification.

CONCLUSIONS

A majority of anesthesiology faculty members reported difficulty with assessing a videotaped OSCE of error disclosure using milestones as primary assessment tools. Use of the checklist assisted in correct assessments at the entry and junior levels.

Peer assessment in the objective structured clinical examination: A scoping review

BACKGROUND

The objective structured clinical examination (OSCE), originally designed with experts assessing trainees' competence, is more frequently employed with an element of peer assessment and feedback. Although peer assessment in higher education has been studied, its role in OSCEs has not reviewed.

AIMS

The aim of this study is to conduct a scoping review and explore the role of peer assessment and feedback in the OSCE.

METHODS

Electronic database and hand searching yielded 507 articles. Twenty-one full records were screened, of which 13 were included in the review. Two independent reviewers completed each step of the review.

RESULTS

Peer-based OSCEs are used to assess students' accuracy in assessing OSCE performance and to promote learning. Peer examiners (PE) tend to award better global ratings and variable checklist ratings compared to faculty and provide high-quality feedback. Participating in these OSCEs is perceived as beneficial for learning.

CONCLUSIONS

Peer assessment and feedback can be used to gauge PE reliability and promote learning. Teachers using these OSCEs must use methodology which fits their purpose. Competency-based education calls for diversification of assessment practices and asks how assessment impacts learning; the peer-based OSCE responds to these demands and will become an important practice in health professions education.

Individualised Expert Feedback is Not Essential for Improving Basic Clinical Skills Performance in Novice Learners: A Randomized Trial

OBJECTIVE

To determine whether unsupervised video feedback (UVF) is as effective as direct expert feedback (DEF) in improving clinical skills performance for medical students learning basic surgical skills-intravenous cannulation, catheterization, and suturing.

BACKGROUND

Feedback is a vital component of the learning process, yet great variation persists in its quality, quantity, and methods of delivery. The use of video technology to assist in the provision of feedback has been adopted increasingly.

METHODS

A prospective, blinded randomized trial comparing DEF, an expert reviewing students' performances with subsequent improvement suggestions, and UVF, students reviewing their own performance with an expert teaching video, was carried out. Medical students received an initial teaching lecture on intravenous cannulation, catheterization, and suturing and were then recorded performing the task. They subsequently received either DEF or UVF before reperforming the task. Students' recordings were additionally scored by 2 blinded experts using a validated proforma.

RESULTS

A total of 71 medical students were recruited. Cannulation scores improved 4.3% with DEF and 9.5% with UVF (p = 0.044), catheterization scores improved 8.7% with DEF and 8.9% with UVF (p = 0.96), and suturing improved 15.6% with DEF and 13.2% with UVF (p = 0.54). Improvement from baseline scores was significant in all cases (p < 0.05).

CONCLUSION

Video-assisted feedback allows a significant improvement in clinical skills for novices. No significant additional benefit was demonstrated from DEF, and a similar improvement can be obtained using a generic expert video and allowing students to review their own performance. This could have significant implications for the design and delivery of such training.

Performance Assessment in Medical Education

The assessment of clinical competence is becoming increasingly complex, patient cen tered, and student driven. Traditionally, clini cal evaluation methods consisted primarily of faculty observations, oral examinations, and multiple-choice tests. Increased faculty work load, discontent with traditional methods of clinical skill assessment, and developments in the fields of psychology and education have led to the formation of new modalities, namely performance assessments. The literature per taining to the performance assessment with standardized patients is reviewed. Based on this literature, several areas for the future direction of performance assessment are pro posed, including (a) toward evidence-based locally developed assessments, (b) toward an understanding of educational outcomes and noncognitive assessment factors, and (c) toward more student-driven assessments.

Demonstrating Medical Student Competency in Palliative Care: Development and Evaluation of a New Objective Structured Clinical Examination Station

BACKGROUND

The observed structured clinical examination (OSCE) is an important tool to assess clinical competencies; however, there are no reported palliative care OSCEs for medical student assessment.

OBJECTIVE

We aimed to develop, implement, and evaluate the characteristics of a palliative care OSCE for fourth-year medical students.

METHODS

We created a representative case and a checklist of 14 history items from three core palliative care competency domains. Subjects were fourth-year medical students who had completed our school's longitudinal palliative care curriculum. Measurements were students' scores compiled from the standardized patient's (SP) tally of the checklist results. We determined inter-rater reliability between the SP and a remote observer. Measurements included the difficulty and discrimination index, internal consistency reliability, factor analysis, and relationships between palliative care scores and composite seven station OSCE scores.

RESULTS

In the implementation year, 95 students scored an average of 74% (standard deviation [SD] = 13%) on the 14 history items. There was 95% agreement in ratings on items between the SP and the remote observer. The Cronbach's alpha was 0.53, demonstrating moderate internal consistency. The palliative care scores correlated with overall OSCE communication scores (R = 0.29, p = 0.01) and history scores (R = 0.61, p = 0.01).

CONCLUSIONS

A new OSCE to assess palliative care competencies was feasible to implement with high inter-rater reliability, evidence supporting validity, and moderate internal consistency. We believe this OSCE would prove useful to assess students' primary palliative care competency and to evaluate curricula in palliative care.

To observe or not to observe peers when learning physical examination skills; that is the question

BACKGROUND

Learning physical examination skills is an essential element of medical education. Teaching strategies include practicing the skills either alone or in-group. It is unclear whether students benefit more from training these skills individually or in a group, as the latter allows them to observing their peers. The present study, conducted in a naturalistic setting, investigated the effects of peer observation on mastering psychomotor skills necessary for physical examination.

METHODS

The study included 185 2nd-year medical students, participating in a regular head-to-toe physical examination learning activity. Students were assigned either to a single-student condition (n = 65), in which participants practiced alone with a patient instructor, or to a multiple-student condition (n = 120), in which participants practiced in triads under patient instructor supervision. The students subsequently carried out a complete examination that was videotaped and subsequently evaluated. Student's performance was used as a measure of learning.

RESULTS

Students in the multiple-student condition learned more than those who practiced alone (81% vs 76%, p < 0.004). This result possibly derived from a positive effect of observing peers; students who had the possibility to observe a peer (the second and third students in the groups) performed better than students who did not have this possibility (84% vs 76%, p <. 001). There was no advantage of observing more than one peer (83.7% vs 84.1%, p > .05).

CONCLUSIONS

The opportunity to observe a peer during practice seemed to improve the acquisition of physical examination skills. By using small groups instead of individual training to teach physical examination skills, health sciences educational programs may provide students with opportunities to improve their performance by learning from their peers through modelling.

Structured training on box trainers for first year surgical residents: does it improve retention of laparoscopic skills? A randomized controlled study

BACKGROUND AND AIM

Structured training on box trainers in laparoscopic skills in the initial years of residency has been used and found to be effective. Although there are studies that confirm immediate improvement after training, there is a lack of well-designed trials addressing the crucial issue of retention of these skills over time. The purpose of this study is to assess improvement in laparoscopic skills of surgical trainees after structured training on box trainers, compared with traditional training (observing and assisting laparoscopic procedures in the operation rooms) immediately and after 5 months.

METHODS

Forty surgical residents in their first 2 months of residency training were randomized to either structured training on box trainers, in addition to traditional training, or to traditional training alone. Groups were equivalent with regards to demographics, previous operative experience, and baseline skills. Structured training consisted of 4 sessions with 6 tasks on box trainers under supervision and self practice. Task-based objective structured practical examinations (OSPE) were completed before and after each task. At the end of the training, residents were assessed by a blinded faculty member with the global operative assessment of laparoscopic skills (GOALS) rating scale. Residents also completed a satisfaction questionnaire. Focus group discussions were conducted for both groups. The GOALS were repeated for both the groups at the end of 5 months to assess retention of skills.

RESULTS

The mean GOALS score was significantly higher for the structured training group (mean/SD 20.35 + 0.74) compared with the traditional training group (mean/SD 16.35 + 1.75, p < 0.01) at the end of 5 months. The mean global rating scale (GRS) score was significantly higher (Pre 7.55 + 0.99 vs. Post 16.4 + 0.68, p < 0.01) for the structured training group at the end of course. Residents in the structured training group had significantly improved skills immediately after the training and had better retention of skills at the end of five months.

CONCLUSIONS

Structured training on box trainers, in addition to traditional training, compared with traditional training alone, leads to better skills and improved confidence of residents. There is significant retention of skills at the end of 5 months. These results provide support for incorporation of structured training with box trainers for laparoscopic skills into surgical training programs.

The Postoperative Pain Assessment Skills pilot trial

BACKGROUND⁄

OBJECTIVES

Pain-related misbeliefs among health care professionals (HCPs) are common and contribute to ineffective postoperative pain assessment. While standardized patients (SPs) have been effectively used to improve HCPs' assessment skills, not all centres have SP programs. The present equivalence randomized controlled pilot trial examined the efficacy of an alternative simulation method - deteriorating patient-based simulation (DPS) - versus SPs for improving HCPs' pain knowledge and assessment skills.

METHODS

Seventy-two HCPs were randomly assigned to a 3 h SP or DPS simulation intervention. Measures were recorded at baseline, immediate postintervention and two months postintervention. The primary outcome was HCPs' pain assessment performance as measured by the postoperative Pain Assessment Skills Tool (PAST). Secondary outcomes included HCPs knowledge of pain-related misbeliefs, and perceived satisfaction and quality of the simulation. These outcomes were measured by the Pain Beliefs Scale (PBS), the Satisfaction with Simulated Learning Scale (SSLS) and the Simulation Design Scale (SDS), respectively. Student's t tests were used to test for overall group differences in postintervention PAST, SSLS and SDS scores. One-way analysis of covariance tested for overall group differences in PBS scores.

RESULTS

DPS and SP groups did not differ on post-test PAST, SSLS or SDS scores. Knowledge of pain-related misbeliefs was also similar between groups.

CONCLUSIONS

These pilot data suggest that DPS is an effective simulation alternative for HCPs' education on postoperative pain assessment, with improvements in performance and knowledge comparable with SP-based simulation. An equivalence trial to examine the effectiveness of deteriorating patient-based simulation versus standardized patients is warranted.

Evaluation of medical student performance on objective structured clinical exams with standardized patients with and without disabilities

PURPOSE

To investigate whether medical students' performance on a family medicine clerkship objective structured clinical exam (OSCE) differed when the standardized patient (SP) had a disability versus when the SP did not have a disability.

METHOD

SPs with spinal cord injury (SP-SCI), SPs with intellectual disability (SP-ID), and SPs without a disability participated separately in two OSCE scenarios that were administered by the University of South Carolina School of Medicine's Department of Family and Preventive Medicine from 2007 to 2009. OSCE scores were determined based on the number of critical actions completed by the student, and scores were analyzed to determine differences among scenarios.

RESULTS

Students scored lower in history, physical exam, lab tests, and interpersonal skills with an SP-SCI, and lower in history, physical exam, and lab tests with an SP-ID than did students interacting with SPs without a disability. The odds ratio for ordering a hemoglobin A1c in one scenario was 4.16 times higher in cases when the SP did not have a disability (95% confidence interval [CI] 1.78-9.17, P = .001). In the second scenario, the odds ratio was 3.08 times higher for ordering a urinalysis (95% CI 1.34-7.08, P = .006) and was 2.15 times higher for providing lifestyle counseling (95% CI 1.04-4.44, P = .038) in students interacting with SPs without a disability.

CONCLUSIONS

Students performed better when the SP did not have a disability. This suggests that greater emphasis should be placed on teaching appropriate care of patients with a disability.

An exploration of student nurses' thoughts and experiences of using a video-recording to assess their performance of cardiopulmonary resuscitation (CPR) during a mock objective structured clinical examination (OSCE)

Cardiopulmonary resuscitation (CPR) is an essential skill taught within undergraduate nursing programmes. At the author's institution, students must pass the CPR objective structured clinical examination (OSCE) before progressing to second year. However, some students have difficulties developing competence in CPR and evidence suggests that resuscitation skills may only be retained for several months. This has implications for practice as nurses are required to be competent in CPR. Therefore, further opportunities for students to develop these skills are necessary. An action research project was conducted with six students who were assessed by an examiner at a video-recorded mock OSCE. Students self-assessed their skills using the video and a checklist. Semi-structured interviews were conducted to compare checklist scores, and explore students' thoughts and experiences of the OSCE. The findings indicate that students may need to repeat this exercise by comparing their previous and current performances to develop both their self-assessment and CPR skills. Although there were some differences between the examiner's and student's checklist scores, all students reported the benefits of participating in this project, e.g. discussion and identification of knowledge and skills deficits, thus emphasising the benefits of formative assessments to prepare students for summative assessments and ultimately clinical practice.

Information-gathering patterns associated with higher rates of diagnostic error

Diagnostic errors are an important source of medical errors. Problematic information-gathering is a common cause of diagnostic errors among physicians and medical students. The objectives of this study were to (1) determine if medical students' information-gathering patterns formed clusters of similar strategies, and if so (2) to calculate the percentage of incorrect diagnoses in each cluster. A total of 141 2nd year medical students completed a computer case simulation. Each student's information-gathering pattern included the sequence of history, physical examination, and ancillary testing items chosen from a predefined list. We analyzed the patterns using an artificial neural network and compared percentages of incorrect diagnoses among clusters of information-gathering patterns. We input patterns into a 35 x 35 self organizing map. The network trained for 10,000 epochs. The number of students at each neuron formed a surface that was statistically smoothed into clusters. Each student was assigned to one cluster, the cluster that contributed the largest value to the smoothed function at the student's location in the grid. Seven clusters were identified. Percentage of incorrect diagnoses differed significantly among clusters (Range 0-42%, Chi (2) = 13.62, P = .034). Distance of each cluster from the worst performing cluster was used to rank clusters. This rank was compared to rank determined by percentage incorrect. We found a high positive correlation (Spearman Correlation = .893, P = .007). Clusters closest to the worst performing cluster had the highest percentages of incorrect diagnoses. Patterns of information-gathering were distinct and had different rates of diagnostic error.

The assessment of individual cognitive expertise and clinical competency: a research agenda

There is a large push to utilize evidence-based practices in medical education. At the same time, credentialing bodies are evaluating the use of simulation technologies to assess the competency and safety of its practitioners. At the 2008 Academic Emergency Medicine Consensus Conference on "The Science of Simulation in Healthcare," our breakout session critically evaluated several issues important to the use of simulation in emergency physician (EP) assessment. In this article, we discuss five topics felt to be most critical to simulation-based assessment (SBA). We then offer more specific research questions that would help to define and implement a SBA program in emergency medicine (EM).

Unintentional failure to assess for experience in senior undergraduate OSCE scoring

CONTEXT

One goal of undergraduate assessment is to test students' (future) performance. In the area of skills testing, the objective structured clinical examination (OSCE) has been of great value as a tool with which to test a number of skills in a limited time, with bias reduction and improved reliability. But can OSCEs measure undergraduate internship expertise in basic clinical skills?

METHODS

Undergraduate students (n = 32) were given a questionnaire listing 182 basic clinical skills. We asked them to score the number of times they had performed each skill during their internships (a 12-month period in Year 6). We assessed the students at the end of Year 5 (before the start of their internships) and again at the start of Year 7 (undergraduate training takes 7 years in Belgium, with internships during Year 6), using a 14-station OSCE assessing basic clinical skills. Global ratings were used to score performance. The relationship between internship experience and the OSCE Year 7 score was analysed using a linear regression model, controlling for variation in OSCE scores from Year 5. A multi-level analysis was performed considering students as level-1 units and stations as level-2 units.

RESULTS

Year 7 OSCE scores (post-internships) were not affected by the number of times that students practised basic medical skills during their internships.

DISCUSSION

Scores on OSCEs do not seem to reflect clinical expertise acquired during internships. Other more integrated assessment methods may prove to be more valid for testing final undergraduate skills levels.

Low- to high-fidelity simulation - a continuum of medical education?

CONTEXT

Changes in medical training and culture have reduced the acceptability of the traditional apprenticeship style training in medicine and influenced the growth of clinical skills training. Simulation is an educational technique that allows interactive, and at times immersive, activity by recreating all or part of a clinical experience without exposing patients to the associated risks. The number and range of commercially available technologies used in simulation for education of health care professionals is growing exponentially. These range from simple part-task training models to highly sophisticated computer driven models.

AIM

This paper will review the range of currently available simulators and the educational processes that underpin simulation training. The use of different levels of simulation in a continuum of training will be discussed. Although simulation is relatively new to medicine, simulators have been used extensively for training and assessment in many other domains, most notably the aviation industry. Some parallels and differences will be highlighted.