Ways to Write a Milestone: Approaches to Operationalizing the Development of Competence in Graduate Medical Education

Importance and fitness of family medicine milestone project from America to Taiwan

BACKGROUND

The six core competencies of ACGME - patient care (PC), medical knowledge (MK), systems-based practice (SBP), practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal and communication skills (ICS) - represent domains in which physicians must ultimately demonstrate competence. Although the ACGME's six core competencies have been applied in Taiwan with the milestone project, the application of the six core competences in the Family Medicine milestones for residency training have not yet been established.

METHODS

We recruited 61 family medicine physicians from 25 hospitals from four major geographic areas for a Delphi round one survey and 72 physicians from 27 hospitals for a Delphi round two survey. With 5-point scales, the Cronbach's alphas for both importance and fitness were 0.98 in round one. In round two, the Cronbach's alphas were 0.86 and 0.83 for importance and fitness. The mode and quartile deviation in Delphi method, importance-performance analysis (IPA), and importance-performance matrix analysis (IPMA) were used for three stages IPA process.

RESULTS

In IPA, a total of 72.7% (16/22) of the sub-competencies exhibited high importance and fitness, with a mean score ≥ 4.7; the exceptions were PC-1 (cares for acutely ill or injured patients), MK-2 (critical thinking skills in patient care), SBP-1 (cost-conscious medical care), PBLI-3 (improves systems), PROF-3 (humanism/cultural proficiency), and ICS-4 (utilizes technology). In IPMA, the performance value of six core competencies for FMM-Taiwan was 92.6 when considering the importance and fitness indices of the 22 sub-competencies.

CONCLUSIONS

The accordance of the ACGME's milestones to Taiwan was acceptable to good and related milestones could been developed for residency training.

Development of an integrated milestone assessment tool across multiple early-adopter programs for breaking bad news: a pilot project

BACKGROUND

The transition of the Accreditation Council for Graduate Medical Education (ACGME) to milestone assessment creates opportunities for collaboration and shared assessments across graduate medical programs. Breaking bad news is an essential communication skill that is a common milestone across almost every medical specialty. The purpose of this study was to develop and pilot an integrated milestone assessment (IMA) tool for breaking bad news using ACGME milestone criteria and to compare the IMA tool with the existing SPIKES protocol.

METHODS

The IMA tool was created using sub-anchors in professionalism and interpersonal communication skills that are applicable to every specialty and to the ability to break bad news. Two cases of breaking bad news, designed to be "easy" and "intermediate" in difficulty, were used to assess basic skills in breaking bad news in first-year medical residents from six residency specialties. Eight standardized patients were trained to portray the cases in sessions held in November 2013 and May 2014. Standardized patients completed an assessment checklist to evaluate each resident's performance in breaking bad news based on their use of the SPIKES protocol and IMA tool. Residents answered post-encounter questions about their training and comfort in breaking bad news. The association between SPIKES and IMA scores was investigated by simple linear regression models and Spearman rank correlations.

RESULTS

There were 136 eligible medical residents: 108 (79.4%) participated in the first session and 97 (71.3%) participated in the second session, with 96 (70.6%) residents participating in both sessions. Overall, we were able to identify residents that performed at both extremes of the assessment criteria using the integrated milestone assessment (IMA) and the SPIKES protocol. Interestingly, residents rated themselves below "comfortable" on average.

CONCLUSION

We developed an integrated milestone assessment (IMA) that was better than the SPIKES protocol at assessing the skill of breaking bad news. This collaborative assessment tool can be used as supplement tool in the era of milestone transformation. We aim assess our tool in other specialties and institutions, as well as assess other shared milestones across specialties.

Evaluation of bias and gender/racial concordance based on sentiment analysis of narrative evaluations of clinical clerkships using natural language processing

There is increasing interest in understanding potential bias in medical education. We used natural language processing (NLP) to evaluate potential bias in clinical clerkship evaluations. Data from medical evaluations and administrative databases for medical students enrolled in third-year clinical clerkship rotations across two academic years. We collected demographic information of students and faculty evaluators to determine gender/racial concordance (i.e., whether the student and faculty identified with the same demographic). We used a multinomial log-linear model for final clerkship grades, using predictors such as numerical evaluation scores, gender/racial concordance, and sentiment scores of narrative evaluations using the SentimentIntensityAnalyzer tool in Python. 2037 evaluations from 198 students were analyzed. Statistical significance was defined as P < 0.05. Sentiment scores for evaluations did not vary significantly by student gender, race, or ethnicity (P = 0.88, 0.64, and 0.06, respectively). Word choices were similar across faculty and student demographic groups. Modeling showed narrative evaluation sentiment scores were not predictive of an honors grade (odds ratio [OR] 1.23, P = 0.58). Numerical evaluation average (OR 1.45, P < 0.001) and gender concordance between faculty and student (OR 1.32, P = 0.049) were significant predictors of receiving honors. The lack of disparities in narrative text in our study contrasts with prior findings from other institutions. Ongoing efforts include comparative analyses with other institutions to understand what institutional factors may contribute to bias. NLP enables a systematic approach for investigating bias. The insights gained from the lack of association between word choices, sentiment scores, and final grades show potential opportunities to improve feedback processes for students.

Pediatric Pulmonary Milestones 2.0: Development, Lessons Learned, and Future Directions

Pediatric pulmonology fellowship training programs are required by the Accreditation Council for Graduate Medical Education to report Pediatric Subspecialty Milestones biannually to track fellow progress. However, several issues, such as lack of subspecialty-specific context and ambiguous language, have raised concerns about their validity and applicability to use for fellow assessment and curriculum development. In this Perspective, we briefly share the process of the Pediatric Pulmonology Milestones 2.0 Work Group in creating new specialty-specific Milestones and tailoring information on the Harmonized Milestones to pediatric pulmonologists, with the goal of improving the Milestones' utility for stakeholders, including pulmonology fellows, faculty, program directors, and accrediting bodies. In addition, we created a supplemental guide to better link the Milestones to pulmonary-specific scenarios to create a shared mental model between stakeholders and remove a potential detriment to validity. Through the process, a number of guiding principles were clarified, including: 1) every Milestone should be able to be assessed independently, without overlap with other Milestones; 2) there should be clear developmental progression from one Milestone to the next; 3) Milestones should be based on the unique skills expected of pediatric pulmonologists; and 4) health equity should be a core component to highlight as a top priority to all stakeholders. In this Perspective, we describe these principles that guided formulation of the Pediatric Pulmonary Milestones to help familiarize the pediatric pulmonary community with the new Milestones. In addition, we share lessons learned and challenges in our process to inform other specialties that may soon participate in this process.

Milestones 2.0: An advancement in competency-based assessment for dermatology

In 2013, Next Accreditation System and Milestones became the competency-based assessment framework required for all specialties accredited by the Accreditation Council for Graduate Medical Education. Dermatology residency programs implemented Milestones 1.0 in the 2013-2014 academic year. The Accreditation Council for Graduate Medical Education committed to review and revise Milestones 1.0 within 3 to 5 years. Subsequently, feedback from key stakeholders influenced the goals for revision, including reducing complexity, enhancing community engagement, and providing additional resources for programs. In 2019, the Dermatology Milestones 2.0 work group streamlined the specialty-specific patient care and medical knowledge subcompetencies. The harmonized milestones allowed for greater uniformity across specialties in systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication and skills. The work group developed a supplemental guide with specialty-specific context to help program directors, clinical competency committee members, and other faculty understand individual milestones. Dermatology Milestones 2.0 reduces the number of subcompetencies from 28 to 21. Milestones 2.0 represents an advancement in competency-based assessment for dermatology. The first year of reporting for Dermatology Milestones 2.0 is 2021.

Rheumatology Milestones 2.0: A Roadmap for Competency-Based Medical Training of Rheumatology Fellows in the 21st Century

OBJECTIVE

Since 2014, rheumatology fellows have been assessed not only based on their ability to provide patient care and possession of medical knowledge but also on their skill in serving as patient advocates, navigators of health systems, and members of a health care team. Such assessments have been carried out through the use of competency-based milestones from the Accreditation Council of Graduate Medical Education (ACGME). However, a needs assessment has demonstrated interest in more context validity and subspecialty relevance since the development of the ACGME internal medicine (IM) subspecialty reporting milestones. The ACGME thus created a milestones working group, and the present study was undertaken to develop Rheumatology Milestones 2.0 as well as a supplemental guide to assist with implementation.

METHODS

The working group, consisting of 7 rheumatology program directors, 2 division directors, a community practice rheumatologist, a rheumatology fellow in training, and a public member who is a rheumatology patient, was overseen by the ACGME vice president for milestones development and met through three 12-hour, in-person meetings to compose the rheumatology specialty milestones and supplemental guide within the ACGME Milestones 2.0 project.

RESULTS

Informed by the needs assessment data and stakeholders, the working group revised and adapted the ACGME IM subspecialty reporting milestones to create a rheumatology-specific set of milestones and a supplemental guide for their implementation.

CONCLUSION

The Rheumatology Milestones 2.0 provides a specialty-specific, competency-based evaluation tool that can be used by program directors, clinical competency committees, and others to assess the competencies of rheumatology fellows during training and help measure readiness for independent practice.

Sleep Medicine Milestones 2.0: designed for our field

NONE

The Accreditation Council for Graduate Medical Education published the first sleep medicine milestones in 2015. However, these milestones were the same among all internal medicine fellowship programs; they were not specific to the specialty. Based on stakeholder feedback, the Accreditation Council for Graduate Medical Education called for the creation of specialty-specific milestones. Herein, we outline the history of Accreditation Council for Graduate Medical Education reporting milestones; the identification of knowledge, skills, and attitudes that define the practice of sleep medicine; and the creation of the supplemental guide and sleep medicine-specific milestones (Sleep Medicine Milestones 2.0) to assess developmental progression during fellowship training.

ACGME Milestones 2.0: why and what's new for cytopathology?

BACKGROUND

Primary stakeholders in the Accreditation Council for Graduate Medical Education (ACGME) Milestones Project are: ACGME, Residency Programs, Residents, Fellowship Programs, Fellows, and Certification Boards. The intent of the Milestones is to describe the educational and professional developmental trajectory of a trainee from the first stages of their postgraduate education through the completion of their clinical training. The Milestones 2.0 project includes changes made based on experience with Milestones 1.0.

METHODS

The ACGME solicited volunteers to participate in the development of subspecialty Milestones 2.0. The workgroup was charged with reviewing/making any additions to the four "Harmonized Milestones", developing subspecialty specific milestones for the Patient Care and Medical Knowledge competencies, and creating a supplemental guide. The Milestones were finalized following review of input from an open comment period.

RESULTS

The Cytopathology Milestones 2.0 will go into effect July 2021. They include additional subcompetencies in the 4 harmonized competency areas and cytopathology-specific edits to the patient care and medical knowledge subcompetencies. Although the number of subcompetencies has increased from 18 to 21, within each subcompetency, the number of milestone trajectories has decreased. Additionally, within each subcompetency, the wording has been streamlined. A supplemental guide was created and Milestones 1.0 were compared to 2.0; however, curriculum mapping has been left to programs to develop.

CONCLUSIONS

The ultimate goal of the Cytopathology Milestones 2.0 is to provide better real-time documentation of the progress of cytopathology fellows. The expected outcome is to produce highly competent cytopathologists, improving the care they provide, regardless of the program at which they trained.

Preparing Anesthesiology Residents for Operating Room Communication Challenges: A New Approach for Conflict Resolution Training

BACKGROUND

The hierarchical culture in high-stake areas such as operating rooms (ORs) may create volatile communication challenges. This unfunded exploratory study sought to establish whether a conflict resolution course was effective in preparing anesthesiology residents to handle and deescalate disagreements that may arise in the clinical environment, especially when challenging a surgeon.

METHODS

Thirty-seven anesthesiology residents were assessed for ability to deescalate conflict. Nineteen had completed a conflict resolution course, and 18 had not. The 2-hour course used 10 videotaped vignettes that showed attending anesthesiologists, patients, and surgeons challenging residents in a potentially confrontational situation. Guided review of the videos and discussions was focused on how the resident could optimally engage in conflict resolution. To determine efficacy of the conflict resolution course, we used simulation-based testing. The setting was a simulated OR with loud music playing (75-80 dB) under the control of the surgeon. The music was used as a tool to create a potential, realistic confrontation with the surgeon to test conflict resolution skills. The initial evaluation of the resident was whether they ignored the music, asked for the surgeon to turn it off, or attempted to turn it off themselves. The second evaluation was whether the resident attempted to deescalate (eg, calmly negotiate for the music to be turned off or down) when the surgeon was scripted to adamantly refuse. Two trained observers evaluated residents' responses to the surgeon's refusal.

RESULTS

Of the residents who experienced the confrontational situation and had not yet taken the conflict resolution course, 1 of 5 (20.0%; 95% CI, 0.5-71.6) were judged to have deescalated the situation. In comparison, of those who had taken the course, 14 of 15 (93.3%; 95% CI, 68.1-99.8) were judged to have deescalated the situation (P = .002). Only 2 of 19 (10.5%; 95% CI, 1.3-33.1) of those who completed the course ignored the music on entering the OR versus 10 of 18 (55.6%; 95% CI, 30.8-78.5) who did not complete the course (P = .004).

CONCLUSIONS

This study suggests that a conflict resolution course may improve the ability of anesthesiology residents to defuse clinical conflicts. It also demonstrated the effectiveness of a novel, simulation-based assessment of communication skills used to defuse OR confrontation.

Core Competency Self-Assessment Enhances Critical Review of Complications and Entrustable Activities

BACKGROUND

The Accreditation Council for Graduate Medical Education has defined six core competencies (CCs) that every successful physician should possess. However, the assessment of CC achievement among trainees is difficult. This project was designed to prospectively evaluate the impact of resident identification of CC as a component of morbidity review on error identification and standard of care (SOC) assessments. The platform was assessed for its reliability as a measure of resident critical analysis of complication causality across postgraduate year (PGY).

MATERIALS AND METHODS

A total of 1945 general surgery cases with complications were assessed for error identification and SOC management between January 1, 2016, and December 31, 2018. CC identification was additionally assessed between January 1, 2019, and December 31, 2019, and included 708 general surgery cases. Data were evaluated for error assessments and overall SOC management. PGY4 and 5 residents were compared for number of cases and complications reviewed, severity, error causation, and CC relevance.

RESULTS

Study groups were equivalent by Clavien-Dindo scores. Error identification significantly increased in all categories: diagnostic (P < 0.001), technical (P < 0.05), judgment (P < 0.001), system (P < 0.001), and communication (P < 0.001). Overall SOC assessments validated by a supervising surgical quality officer were unchanged. An increased exposure to cases with severe complications, error causation, and CC relevance was noted across PGY.

CONCLUSIONS

The addition of CC assessment into morbidity review appears to improve the critical thinking of evaluating residents by increasing the identification of management errors. Used as an element of prospective self-assessment, teaching residents to identify CC principles in cases with complications may assist in learner progression toward clinical competence and critical thinking.

Developing National Conceptual Understanding to Describe Entry-to-Practice Dietetics Competence

OBJECTIVE

Develop consensus on entrustable professional activities (EPAs) and milestones for students, practice educators, and academics that need shared understanding to assess entry-level competence in dietetics.

DESIGN

An iterative, pragmatic, 3 key-phase approach involving workshops, a consensus survey with job description and consultation.

SETTING

Australia.

PARTICIPANTS

A total of 58 academics and 43 practitioners with expertise in competency-based assessment in dietetics were purposefully selected to participate in 4 national workshops. Of those selected, 36 completed a survey that sought consensus on the EPAs and milestones drafted at the workshops (36% response rate).

PHENOMENON OF INTEREST

Support for competency-based assessment.

ANALYSIS

Level of agreement on the survey was set at ≥70%. These comments were triangulated with content analysis from 98 entry-level job descriptions.

RESULTS

Consensus on 4 key tasks of the profession with associated descriptions of the level of performance were included to implement a nutrition intervention; facilitate a food, nutrition, and/or lifestyle conversation; perform professional activities and projects; and work as part of a team.

CONCLUSIONS AND IMPLICATIONS

Identification of the key minimum tasks of a dietitian upon entry-to-practice, together with developmental descriptors of performance, provides clarity in the expected progression and end point of assessment. This resource will support competency-based assessment decisions. The international transferability of this warrants further investigation.

Urology Milestones 2.0: The Future Looks Bright

PURPOSE OF REVIEW

One of the major functions of the Accreditation Council for Graduate Medical Education (ACGME) is to accredit all approved residency programs. This accreditation system is based on both common and program-specific requirements that form the foundation of all ACGME-accredited training programs. Embedded within the program requirements are the essential elements of the Competencies and Milestones. In this review article, we hope to provide the reader with an overview of the current Milestones and a preview of what lies ahead.

RECENT FINDINGS

Milestones for resident education were implemented approximately 7 years ago. The milestones were intended to create a logical trajectory of professional growth which could be measured and tracked for each sub-specialty. However, substantial variability in both content and developmental progression was seen in many specialties. The ACGME has been actively reviewing the Milestones to insure that there exists harmony across all specialties. Much has been learned about the milestones since their implementation. As educators, we need to provide a robust and reproducible system for all to use. The future of resident education, Milestones 2.0, will provide the necessary groundwork for a more user friendly system that will allow adequate evaluation of our trainees.

A practical framework for remediating unprofessional behavior and for developing professionalism competencies and a professional identity

The relatively new term "Professional Identity Formation" (PIF) complements behavior-based and attitude-based perspectives on professionalism. Unprofessional behavior and its remediation should also be addressed from this perspective. However, a framework is needed to guide discussion and remediation of unprofessional behavior, which can encompass behavior-based, attitude-based, and identity-based perspectives on professionalism. To this end, the authors propose a multi-level professionalism framework which describes, apart from professional behavior, more levels which influence professional performance: environment, competencies, beliefs, values, identity, and mission. The different levels can provide tools for educators to address and discuss unprofessional behavior with their students in a comprehensive way. By reflecting on all the different levels of the framework, educators guard themselves against narrowing the discussion to either professional behavior or professional identity. The multi-level professionalism framework can help educators and students to gain a better understanding of the root of unprofessional behavior, and of remediation strategies that would be appropriate. For despite the recent emphasis on PIF, unprofessional behavior and its remediation will remain important issues in medical education.

A Checklist to Help Faculty Assess ACGME Milestones in a Video-Recorded OSCE

BACKGROUND

Faculty members need to assess resident performance using the Accreditation Council for Graduate Medical Education Milestones.

OBJECTIVE

In this randomized study we used an objective structured clinical examination (OSCE) around the disclosure of an adverse event to determine whether use of a checklist improved the quality of milestone assessments by faculty.

METHODS

In 2013, a total of 20 anesthesiology faculty members from 3 institutions were randomized to 2 groups to assess 5 videos of trainees demonstrating advancing levels of competency on the OSCE. One group used milestones alone, and the other used milestones plus a 13-item checklist with behavioral anchors based on ideal performance. We classified faculty ratings as either correct or incorrect with regard to the competency level demonstrated in each video, and then used logistic regression analysis to assess the effect of checklist use on the odds of correct classification.

RESULTS

Thirteen of 20 faculty members rated assessing performance using milestones alone as difficult or very difficult. Checklist use was associated with significantly greater odds of correct classification at entry level (odds ratio [OR] = 9.2, 95% confidence interval [CI] 4.0-21.2) and at junior level (OR = 2.7, 95% CI 1.3-5.7) performance. For performance at other competency levels checklist use did not affect the odds of correct classification.

CONCLUSIONS

A majority of anesthesiology faculty members reported difficulty with assessing a videotaped OSCE of error disclosure using milestones as primary assessment tools. Use of the checklist assisted in correct assessments at the entry and junior levels.