Remediation as a Road to Competency: Strategies for Early Identification of the Struggling Resident and Generating the Remediation Plan

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills

OBJECTIVE

The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor.

DESIGN

The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed.

SETTING

The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles.

RESULTS

The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial.

CONCLUSIONS

ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Learning, judgement and workforce tensions impact failure to fail in a medical postgraduate setting: A supervisor perspective

PURPOSE

Medical training organisations have a duty to prepare medical graduates for future safe, competent practice. Decisions about underperformance are high stakes at the postgraduate level and failure to fail can occur. We aimed to explore this concept from a systems and supervisor perspective.

METHOD

Supervisors of specialist physician trainees were invited to provide written feedback on failure to fail as part of a broader anonymous supervisor survey. They were provided with a trigger statement and responded in free-text format. A deductive content analysis was undertaken through the lenses of supervisor and institution.

RESULTS

Of 663 supervisors who responded to the broader survey, 373 (56%) provided feedback on the failure to fail trigger statement. Analyses indicated an interplay between trainee and supervisor characteristics, and broader system elements. System elements that contributed to failure to fail trainees included lack of longitudinal monitoring and quality of assessment information. Supervisor characteristics included confident, conflicted and avoidant behaviours towards underperforming trainees.

CONCLUSIONS

Individual and system challenges that contributed to failure to fail were identified in this study, and we propose a three-way tension among learning, judgement and workforce. Three potential mitigation strategies have been identified to reduce failure to fail, namely a stage-based approach to remediation, faculty development in supervisory skills and improved assessment-for-learning processes.

Emotion in remediation: A scoping review of the medical education literature

OBJECTIVES

Remediation can be crucial and high stakes for medical learners, and experts agree it is often not optimally conducted. Research from other fields indicates that explicit incorporation of emotion improves education because of emotion's documented impacts on learning. Because this could present an important opportunity for improving remediation, we aimed to investigate how the literature on remediation interventions in medical education discusses emotion.

METHODS

The authors used Arksey and O'Malley's framework to conduct a scoping literature review of records describing remediation interventions in medical education, using PubMed, CINAHL Complete, ERIC, Web of Science and APA PsycInfo databases, including all English-language publications through 1 May 2020 meeting search criteria. They included publications discussing remediation interventions either empirically or theoretically, pertaining to physicians or physician trainees of any level. Two independent reviewers used a standardised data extraction form to report descriptive information; they reviewed included records for the presence of mentions of emotion, described the mentions and analysed results thematically.

RESULTS

Of 1644 records, 199 met inclusion criteria and were reviewed in full. Of those, 112 (56%) mentioned emotion in some way; others focused solely on cognitive aspects of remediation. The mentions of emotion fell into three themes based on when the emotion was cited as present: during regular coursework or practice, upon referral for remediation and during remediation. One-quarter of records (50) indicated potential intentional incorporation of emotion into remediation programme design, but they were non-specific as to how emotions related to the learning process itself.

CONCLUSION

Even though emotion is omnipresent in remediation, medical educators frequently do not factor emotion into the design of remediation approaches and rarely explicitly utilise emotion to improve the learning process. Applications from other fields may help medical educators leverage emotion to improve learning in remediation, including strategies to frame and design remediation.

Remediation of Underperformance in Surgical Trainees - A Scoping Review

OBJECTIVE

Surgical trainees with significant underperformance require formal support to return to an expected standard, termed remediation. The aim of this scoping review was to define remediation interventions, approaches, and contexts.

DESIGN

Following scoping review protocols, we set out to identify the evidence-base for remediation of surgical trainees, outline key concepts and uncover areas to stimulate further research.

RESULTS

From a screen of 80 articles, 24 reported on remediation of surgical trainees. Most were from medical journals (n = 21, 88%) and published in the United States (n = 20, 83%). Ten articles (41%) reported outcomes of remediation of a trainee cohort and 7 (19%) were survey reports from surgical directors. The remainder were a mix of commentaries, editorials or reviews. Thirteen articles (54%) described trainees with deficiencies in multiple competencies, 8 articles (33%) had a singular focus on academic performance, and 1 article (3%) on technical skills. All articles used targeted individualized remediation strategies, a range of intervention methods (some multimodal) and recommended a 6- to 12-month period of remediation (n = 7, 29%). The program director was often the only supervisor (n = 12, 50%). One article reported trainees' perspective of the process and one used educational theory to inform remediation.

CONCLUSIONS

Data with clearly reported outcomes were limited, but we found that targeted, individualized, multimodal and long-term remediation covering a range of competencies have been reported in the literature for surgical trainees. There is a need for development of explicit frameworks, strengthen the support for supervisors and trainees and further apply educational theory to develop better interventions that remediate deficiencies for all competencies.

Remediation for surgical trainees: recommendations from a narrative review

BACKGROUND

Remediation involves formalized support for surgical trainees with significant underperformance to return to expected standards. There is a need to understand current evidence of remediation for surgical trainees to inform practice and justify investment of resources.

METHODS

Following the principles of a systematic review, we conducted a narrative analysis to make recommendations for remediation of underperforming surgical trainees.

RESULTS

From a review of 55 articles on remediation of trainees in medical and surgical sub-specialities, we have identified system and process level recommendations. Remediation is reported as long-term, complex and resource-intensive. Establishing a defined and standardized remediation framework enables co-ordination of multi-modal interventions. System level recommendations aim to consolidate protocols via developing better assessment, intervention and re-evaluation modalities whilst also strengthening support to supervisors conducting the remediation. Process level recommendations should be tailored for the specific needs of each trainee, aiming to be proactive with interventions within a programmatic framework. Regular reassessment is required, and long-term follow-up shows that remediation efforts are often successful.

CONCLUSION

While remediation within a programmatic framework is complex, it is often a successful approach to return surgical trainees to their expected standard. Future directions involve applying learning theories, encouraging research methods and to develop integrated collaborate protocols and support to synergize efforts.

The Clinical Competency Committee in Adult Cardiothoracic Anesthesiology-Perspectives From Program Directors Around the United States

The clinical competency committee offers a fellowship program a structured approach to assess the clinical performance of each trainee in a comprehensive fashion This special article examines the structure and function of this important committee in detail. Furthermore, the strategies for the optimal functioning of this committee are also discussed as a way to enhance the overall quality of the fellowship program.

Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment.

METHOD

The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time.

RESULTS

Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations.

CONCLUSIONS

Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.