Clinical Assessment and Management Examination--Outpatient (CAMEO): its validity and use in a surgical milestones paradigm

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.

Needs assessment can guide creation of a "resident-optimized clinic" in surgery

BACKGROUND

There is limited data on deliberate teaching of residents in the clinic setting; we sought to investigate the clinic experience at our institution and improve education through creation of a novel "Resident-Optimized Clinic" (ROC).

METHODS

An online survey was sent separately to residents and faculty. Based on the results of this survey a modified ROC was developed to try to improve the obstacles to learning in clinic.

RESULTS

Qualitative analysis revealed the barriers in clinic were inconsistencies in expectations, lack of autonomy, time, and facility limitations. The modified ROC was rated positively with 100% of participants expressing they had sufficient time and autonomy; and 90% felt the environment was optimized for teaching.

CONCLUSIONS

Multiple themes have been identified as problematic for the clinic education experience. The ROC was rated positively by trainees suggesting thoughtful intervention to improve clinic results in a better clinic experience and more educational gain from the clinic environment.

A Pilot Study to Gauge Effectiveness of Standardized Patient Scenarios in Assessing General Surgery Milestones

PURPOSE

Some General Surgery Milestones can be difficult to assess in traditional clinical settings and especially difficult to assess in junior residents. The purpose of this pilot study was to METHODS: A total of 9 categorical interns participated in a comprehensive, 4-module, SP scenario designed to evaluate and manage right upper quadrant pain. SP checklist scores (SP%) were converted to Milestone-equivalent scores for direct comparison (SP-C). Milestone scores were analyzed from 3 different sources: SP, faculty (FAC), and CCC. Interns completed course evaluations at the end of each session. Spearman's rho was used to determine correlations. Wilcoxon signed rank tests were used to test for differences between scores from different sources.

RESULTS

Individual intern Milestone scores from the 3 sources (SP-C, FAC, and CCC) did not correlate. All 7 mean Milestone scores from SPs were significantly higher than from FAC and CCC. FAC and CCC scores were statistically equivalent except for Systems-Based Practice 1 (SBP1) and Patient Care 3 (PC3) where CCC scores were significantly higher than FAC. Mean SP% scores for PC1 were significantly lower than for PROF1, MK1, MK2, and ICS1 (p < 0.05). Interns felt the modules were moderately to very useful.

CONCLUSIONS

Developing an SP scenario for Milestones evaluation is feasible. SPs, faculty observers, and CCC each use different data to provide a unique source of Milestone assessment. SP scenarios may be ideally suited to assess specific resident strengths and weaknesses and provide individualized feedback, thus augmenting traditional evaluations. Additional SP scenarios, assessing a broader range of skills and Milestones, are advisable for more reliable estimates of resident performance.

A New Instrument for Assessing Resident Competence in Surgical Clinic: The Ottawa Clinic Assessment Tool

BACKGROUND

The shift toward competency-based medical education has created a demand for feasible workplace-based assessment tools. Perhaps, more important than competence to assess an individual patient is the ability to successfully manage a surgical clinic. Trainee performance in clinic is a critical component of learning to manage a surgical practice, yet no assessment tool currently exists to assess daily performance in outpatient clinics for surgery residents. The development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT), is described here to address this gap.

STUDY DESIGN

A consensus group of experts was gathered to generate dimensions of performance reflective of a competent "generalist" surgeon in clinic. A 6-month pilot study of the OCAT was conducted in orthopedics, general surgery, and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. In all, 2 subsequent feedback sessions and a survey for staff and residents evaluated the OCAT for clarity and utility.

RESULTS

The OCAT is a 9-item tool, with a global assessment item and 2 short-answer questions. Among the 2 divisions, 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of validity. Analysis of feedback indicated that the entrustability rating scale was useful for surgeons and residents and that the items could be correlated with individual competencies.

CONCLUSIONS

Multiple sources of validity evidence collected in this study demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

Development and implementation of a clinical pathway approach to simulation-based training for foregut surgery

INTRODUCTION

Contemporary demands on resident education call for integration of simulation. We designed and implemented a simulation-based curriculum for Post Graduate Year 1 surgery residents to teach technical and nontechnical skills within a clinical pathway approach for a foregut surgery patient, from outpatient visit through surgery and postoperative follow-up.

METHODS

The 3-day curriculum for groups of 6 residents comprises a combination of standardized patient encounters, didactic sessions, and hands-on training. The curriculum is underpinned by a summative simulation "pathway" repeated on days 1 and 3. The "pathway" is a series of simulated preoperative, intraoperative, and postoperative encounters in following up a single patient through a disease process. The resident sees a standardized patient in the clinic presenting with distal gastric cancer and then enters an operating room to perform a gastrojejunostomy on a porcine tissue model. Finally, the resident engages in a simulated postoperative visit. All encounters are rated by faculty members and the residents themselves, using standardized assessment forms endorsed by the American Board of Surgery.

RESULTS

A total of 18 first-year residents underwent this curriculum. Faculty ratings of overall operative performance significantly improved following the 3-day module. Ratings of preoperative and postoperative performance were not significantly changed in 3 days. Resident self-ratings significantly improved for all encounters assessed, as did reported confidence in meeting the defined learning objectives.

CONCLUSIONS

Conventional surgical simulation training focuses on technical skills in isolation. Our novel "pathway" curriculum targets an important gap in training methodologies by placing both technical and nontechnical skills in their clinical context as part of managing a surgical patient. Results indicate consistent improvements in assessments of performance as well as confidence and support its continued usage to educate surgery residents in foregut surgery.