Milestones and Competency-Based Medical Education in Internal Medicine

Clinician Educator Milestones: Assessing and Improving Educators' Skills

ABSTRACT

The importance of the clinician educator (CE) role in delivery of competency-based medical education is well recognized. There is, however, no formal mechanism to identify when faculty have the knowledge, skills, and attitudes to be successful CEs. In 2020, the Accreditation Council for Graduate Medical Education, Accreditation Council for Continuing Medical Education, Association of American Medical Colleges, and American Association of Colleges of Osteopathic Medicine convened a workgroup of 18 individuals representing multiple medical specialties and diverse institutions in the United States, including nonphysician educators, a medical student, and a resident, to develop a set of competencies, subcompetencies, and milestones for CEs.A 5-step process was used to create the Clinician Educator Milestones (CEMs). In step 1, the workgroup developed an initial CEM draft. Through brainstorming, 141 potential education-related CE tasks were identified. Descriptive statements for each competency and developmental trajectories for each subcompetency were developed and confirmed by consensus. The workgroup then created a supplemental guide, assessment tools, and additional resources. In step 2, a diverse group of CEs were surveyed in 2021 and provided feedback on the CEMs. In step 3, this feedback was used by the workgroup to refine the CEMs. In step 4, the second draft of the CEMs was submitted for public comment, and the CEMs were finalized. In step 5, final CEMs were released for public use in 2022.The CEMs consist of 1 foundational domain that focuses on commitment to lifelong learning, 4 additional domains of competence for CEs in the learning environment, and 20 subcompetencies. These milestones have many potential uses for CEs, including self-assessment, constructing learning and improvement plans, and designing systematic faculty development efforts. The CEMs will continue to evolve as they are applied in practice and as the role of CEs continues to grow and develop.

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons' Early Outcomes

Importance

National data on the development of competence during training have been reported using the Accreditation Council for Graduate Medical Education (ACGME) Milestones system. It is now possible to consider longitudinal analyses that link Milestone ratings during training to patient outcomes data of recent graduates.

Objective

To evaluate the association of in-training ACGME Milestone ratings in a surgical specialty with subsequent complication rates following a commonly performed operation, endovascular aortic aneurysm repair (EVAR).

Design, Setting, and Participants

This study of patient outcomes followed EVAR in the Vascular Quality Initiative (VQI) registry (4213 admissions from 208 hospitals treated by 327 surgeons). All surgeons included in this study graduated from ACGME-accredited training programs from 2015 through 2019 and had Milestone ratings 6 months prior to graduation. Data were analyzed from December 1, 2021, through September 15, 2023. Because Milestone ratings can vary with program, they were corrected for program effect using a deviation score from the program mean.

Exposure

Milestone ratings assigned to individual trainees 6 months prior to graduation, based on judgments of surgical competence.

Main Outcomes and Measures

Surgical complications following EVAR for patients treated by recent graduates during the index hospitalization, obtained using the nationwide Society for Vascular Surgery Patient Safety Organization's VQI registry, which includes 929 participating centers in 49 US states.

Results

The study included outcomes for 4213 patients (mean [SD] age, 73.25 [8.74] years; 3379 male participants [80.2%]). Postoperative complications included 9.5% major (400 of 4213 cases) and 30.2% minor (1274 of 4213 cases) complications. After adjusting for patient risk factors and site of training, a significant association was identified between individual Milestone ratings of surgical trainees and major complications in early surgical practice in programs with lower mean Milestone ratings (odds ratio, 0.50; 95% CI; 0.27-0.95).

Conclusions and Relevance

In this study, Milestone assessments of surgical trainees were associated with subsequent clinical outcomes in their early career. Although these findings represent one surgical specialty, they suggest Milestone ratings can be used in any specialty to identify trainees at risk for future adverse patient outcomes when applying the same theory and methodology. Milestones data should inform data-driven educational interventions and trainee remediation to optimize future patient outcomes.

Medical School Rankings: Time to End or Time to Amend?

ABSTRACT

The recent decisions of several medical schools to no longer participate in the rankings published annually by U.S. News & World Report have added greater visibility to the issues surrounding medical school rankings than ever before. While these announcements garnered significant attention in academic medicine and from the lay public, the authors believe these events are unlikely to eliminate rankings in medical education. In this article, the authors outline the potential harms of rankings in higher education, provide a rationale for why they will likely endure despite recent scrutiny, and offer suggestions for prospective students and the academic medical community to engage with rankings in a more productive way. Criticisms of medical school rankings have been noted for decades, including their use of subjective information and data with questionable relevance to the quality of students' education or the outcomes of their educational experience, methodological concerns, and the potential harms of ranking systems (such as schools diverting their focus and resources away from initiatives that benefit students to improve their ranking). At the same time, rankings are ubiquitous in today's culture and are powerful in human decision-making, so there is reason to believe that medical school rankings may weather the current storm. Given these concerns, the authors suggest students continue to use a variety of resources to obtain information about medical schools and consider how each school fits with their own educational needs. The authors also encourage medical schools, medical education organizations, and governing medical bodies to suggest different metrics that reflect quality in medical education and that are of importance to applicants.

Predicting Performance at Graduation From Early ACGME Milestone Ratings: Longitudinal Learning Analytics in Professionalism and Communication in Vascular Surgery

OBJECTIVE

Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories.

DESIGN

This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings.

SETTING

All ACGME-accredited vascular surgery training programs within the United States.

PARTICIPANTS

All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.

RESULTS

The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%).

CONCLUSIONS

As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

Milestones 2.0: An advancement in competency-based assessment for dermatology

In 2013, Next Accreditation System and Milestones became the competency-based assessment framework required for all specialties accredited by the Accreditation Council for Graduate Medical Education. Dermatology residency programs implemented Milestones 1.0 in the 2013-2014 academic year. The Accreditation Council for Graduate Medical Education committed to review and revise Milestones 1.0 within 3 to 5 years. Subsequently, feedback from key stakeholders influenced the goals for revision, including reducing complexity, enhancing community engagement, and providing additional resources for programs. In 2019, the Dermatology Milestones 2.0 work group streamlined the specialty-specific patient care and medical knowledge subcompetencies. The harmonized milestones allowed for greater uniformity across specialties in systems-based practice, practice-based learning and improvement, professionalism, and interpersonal communication and skills. The work group developed a supplemental guide with specialty-specific context to help program directors, clinical competency committee members, and other faculty understand individual milestones. Dermatology Milestones 2.0 reduces the number of subcompetencies from 28 to 21. Milestones 2.0 represents an advancement in competency-based assessment for dermatology. The first year of reporting for Dermatology Milestones 2.0 is 2021.

Association Between Clinical Competencies and Mental Health Symptoms Among Frontline Medical Staff During the COVID-19 Outbreak: A Cross-Sectional Study

Background

In China, mental health of frontline medical staff might be influenced by clinicians' ability to handle the outbreak of coronavirus disease 2019 (COVID-19). Few studies to-date have addressed the association between clinicians' competencies and mental health in this context. This cross-sectional study was to examine the prevalence of mental health symptoms among frontline medical staff that fought against the COVID-19 outbreak, and explore the associations between their competencies, and separate and concurrent depressive and anxiety symptoms.

Methods

A total of 623 frontline medical staff was included in this study. Competencies, depressive symptoms, and anxiety symptoms were assessed using a self-reported short form of the Chinese clinical physicians' competency model, Patient Health Questionnaire-9, and Generalized Anxiety Disorder-7 questionnaire, respectively. Logistic regression models were used to evaluate the associations between one SD increase in competency scores and the prevalence of mental health problems.

Results

The prevalence of depressive, anxiety, and comorbid depressive and anxiety symptoms was 40.93, 31.78, and 26.00%, respectively. Among the medical staff with higher total competency scores, the prevalence of depressive [odds ratios (ORs) = 0.67, 95% confidence intervals (CIs): 0.55-0.81], anxiety (OR = 0.68, 95% CI: 0.56-0.83), and comorbid anxiety and depressive symptoms (OR = 0.69, 95% CI: 0.55-0.83) was lower than among their lower-scoring counterparts. Subgroup analyses stratified by core competency scores revealed similar associations as the main analyses.

Conclusion

The present findings highlight the association between high core competency scores and lower prevalence of depressive, anxiety, and comorbid anxiety and depressive symptoms.

An Empirical Investigation Into Milestones Factor Structure Using National Data Derived From Clinical Competency Committees

PURPOSE

To investigate whether milestone data obtained from clinical competency committee (CCC) ratings in a single specialty reflected the 6 general competency domains framework.

METHOD

The authors examined milestone ratings from all 275 U.S. Accreditation Council for Graduate Medical Education-accredited categorical obstetrics and gynecology (OBGYN) programs from July 1, 2018, to June 30, 2019. The sample size ranged from 1,371 to 1,438 residents from 275 programs across 4 postgraduate years (PGYs), each with 2 assessment periods. The OBGYN milestones reporting form consisted of 28 subcompetencies under the 6 general competency domains. Milestone ratings were determined by each program's CCC. Intraclass correlations (ICCs) and design effects were calculated for each subcompetency by PGY and assessment period. A multilevel confirmatory factor analysis (CFA) perspective was used, and the pooled within-program covariance matrix was obtained to compare the fit of the 6-domain factor model against 3 other plausible models.

RESULTS

Milestone ratings from 5,618 OBGYN residents were examined. Moderate to high ICCs and design effects greater than 2.0 were prevalent among all subcompetencies for both assessment periods, warranting the use of the multilevel approach in applying CFA to the milestone data. The theory-aided split-patient care (PC) factor model, which used the 6 general competency domains but also included 3 factors within the PC domain (obstetric technical skills, gynecology technical skills, and ambulatory care), was consistently shown as the best-fitting model across all PGYs by assessment period conditions, except for one.

CONCLUSIONS

The findings indicate that in addition to using the 6 general competency domains framework in their rating process, CCCs may have further distinguished the PC competency domain into 3 meaningful factors. This study provides internal structure validity evidence for the milestones within a single specialty and may shed light on CCCs' understanding of the distinctive content embedded within the milestones.

A Pilot Study on the Feasibility of Developing and Implementing a Mobile App for the Acquisition of Clinical Knowledge and Competencies by Medical Students Transitioning from Preclinical to Clinical Years

Due to the COVID-19 pandemic and the consequent restrictions, universities have had to adapt their curricula substantially to new schemes in which remote learning is of the essence. In this study, we assess the feasibility of developing a mobile app supplementary to the distant teaching paradigm for the “Cardiology” module of the “General Pathology” subject in undergraduate Medical Education, and we evaluate its impact and acceptability. A cohort of volunteer second-year medical students (n = 44) had access to the app, and their opinions on its utility (1−10) were collected. Additionally, the students were invited to refer their expected satisfaction (1−10) with a blended learning methodology overlapping this new tool with the traditional resources. The average expected satisfaction had been compared to the average satisfaction obtained by just the traditional methodology in other modules from the same subject. Through a qualitative approach, we defined the strengths and weaknesses of the tool. Seventy-seven percent of the participants rated at 8/10 or more the potential learning value of the application and, if used as a supplement to traditional teaching, it would also statistically improve the satisfaction of students (6.52 vs. 8.70, p < 0.001). Similarly, the qualitative data corroborated the benefits of such innovation. Multidisciplinary collaborations are encouraged to develop teaching innovations, although further research should aim to better define the effectiveness of learning with these resources.

Establishing competency-based measures for Department of Veterans Affairs post-graduate nurse practitioner residencies

BACKGROUND

In the past decade, numerous nurse residency models have been created and implemented nationwide; however, validated specialty-specific competency standards have not been established to evaluate Nurse Practitioner (NP) resident core competencies.

PURPOSE

To report the specialty-specific competency assessment tool devised to assess Department of Veterans Affairs (VA) NP residents' competencies and discuss the VA NP residency program's effectiveness in expanding new graduate NP knowledge and skills in the veteran-centric care setting.

METHODS

The VA Nursing Academic Partnership NP residency faculty established and piloted a web-based Nurse Practitioner Resident Competency Assessment (NPRCA) instrument for the comprehensive, specialty-specific assessment of individual NP resident's skill competencies across 24 areas.

RESULTS

The VA specialty-specific competency assessment instrument demonstrates strong internal consistency. The robust VA NP residency program enhances new graduate NP competencies.

CONCLUSIONS

The VA NP residency model can further the goal of standardizing clinical competencies in NP residency programs.

Female Residents Give Themselves Lower Scores Than Male Colleagues and Faculty Evaluators on ACGME Milestones

OBJECTIVE

Orthopedic surgery is one of the specialties with the lowest number of women residents and practicing surgeons. The gender discrepancy in orthopedic residency training may drive a competency bias. We asked whether female orthopedic surgery residents score themselves lower on the Accreditation Council for Graduate Medical Education (ACGME) Milestones than their male counterparts, and lower than their faculty evaluators.

DESIGN

We conducted a retrospective review of ACGME Milestone data from faculty and residents over a 4-year period. The data were analyzed using a snapshot of PGY2 (n = 20 residents) and PGY4 (n = 19 residents) scores, and using a Generalized Estimation Equation (GEE) to account for additional data points from the same residents over the 4-year data collection period.

SETTING

Assessment scores were compiled from a single orthopedic surgery residency at Oregon Health & Science University from 2014 to 2017.

PARTICIPANTS

The residency program has 5 residents in each program year (PGY1 through PGY5); a total of 25 residents during each year of the study were included.

RESULTS

On average, female residents scored themselves lower than both their male counterparts and their faculty mentors. Female PGY2 self-evaluation scores were lower than males in both patient care (p = 0.005) and medical knowledge (p < 0.001). When the GEE model was applied to 99 responses from 41 residents over a 4-year period, there were no gender-related differences in resident self-evaluation scores and in faculty scores of male and female residents, with the exception of meniscal tear. For this milestone, faculty rated female residents lower than males. Furthermore, the differences between faculty evaluation scores and resident self-evaluation scores were significantly lower for males than for females for 4 of the clinical domains, as well as the systems-based practice domains of cost and communication.

CONCLUSIONS

Our results indicate female residents are at risk for a competency bias during training, as reflected by evaluations using the ACGME Milestones.

Fine-needle aspiration performance during cytopathology fellowship: what do the ACGME case logs show us?

INTRODUCTION

Cytopathology fellows are required to enter their fine-needle aspiration (FNA) case numbers in an online data collection system, the Accreditation Council for Graduate Medical Education (ACGME) Case Log system. This study reviewed this data to examine trends in FNA case numbers during fellowship training.

METHODS

A retrospective review of the ACGME Accreditation Data System (ADS) FNA Case Log data was performed for academic years 2006-2019. For 2006-2016, total and average numbers of FNAs performed per academic year were available. After 2016, data also included the number of programs and trainees, national averages, standard deviation, minimum, median, maximum, and percentiles for the number of FNAs performed.

RESULTS

The number of FNAs documented by cytopathology fellows has gradually increased from 2006 (average 10.9) to 2013 (average 18.6) and dramatically increased in 2014 (average 38.0). Averages have remained greater than 30 FNAs documented per academic year since 2014, with some variation. However, a decline was observed in 2019, likely due to the COVID-19 pandemic.

CONCLUSIONS

FNA procedures reported in the ACGME Case Log System indicate vast differences in cytopathology fellowship educational experiences and settings. After logging FNAs becoming an ACGME requirement in 2013, the average number of FNAs has been greater than 30 per year and provides some guidance for programs with respect to the number of FNAs being reported by cytopathology fellows nationally.

Stages of Milestones Implementation: A Template Analysis of 16 Programs Across 4 Specialties

BACKGROUND

Since 2013, US residency programs have used the competency-based framework of the Milestones to report resident progress and to provide feedback to residents. The implementation of Milestones-based assessments, clinical competency committee (CCC) meetings, and processes for providing feedback varies among programs and warrants systematic examination across specialties.

OBJECTIVE

We sought to determine how varying assessment, CCC, and feedback implementation strategies result in different outcomes in resource expenditure and stakeholder engagement, and to explore the contextual forces that moderate these outcomes.

METHODS

From 2017 to 2018, interviews were conducted of program directors, CCC chairs, and residents in emergency medicine (EM), internal medicine (IM), pediatrics, and family medicine (FM), querying their experiences with Milestone processes in their respective programs. Interview transcripts were coded using template analysis, with the initial template derived from previous research. The research team conducted iterative consensus meetings to ensure that the evolving template accurately represented phenomena described by interviewees.

RESULTS

Forty-four individuals were interviewed across 16 programs (5 EM, 4 IM, 5 pediatrics, 3 FM). We identified 3 stages of Milestone-process implementation, including a resource-intensive early stage, an increasingly efficient transition stage, and a final stage for fine-tuning.

CONCLUSIONS

Residency program leaders can use these findings to place their programs along an implementation continuum and gain an understanding of the strategies that have enabled their peers to progress to improved efficiency and increased resident and faculty engagement.

Remediation for surgical trainees: recommendations from a narrative review

BACKGROUND

Remediation involves formalized support for surgical trainees with significant underperformance to return to expected standards. There is a need to understand current evidence of remediation for surgical trainees to inform practice and justify investment of resources.

METHODS

Following the principles of a systematic review, we conducted a narrative analysis to make recommendations for remediation of underperforming surgical trainees.

RESULTS

From a review of 55 articles on remediation of trainees in medical and surgical sub-specialities, we have identified system and process level recommendations. Remediation is reported as long-term, complex and resource-intensive. Establishing a defined and standardized remediation framework enables co-ordination of multi-modal interventions. System level recommendations aim to consolidate protocols via developing better assessment, intervention and re-evaluation modalities whilst also strengthening support to supervisors conducting the remediation. Process level recommendations should be tailored for the specific needs of each trainee, aiming to be proactive with interventions within a programmatic framework. Regular reassessment is required, and long-term follow-up shows that remediation efforts are often successful.

CONCLUSION

While remediation within a programmatic framework is complex, it is often a successful approach to return surgical trainees to their expected standard. Future directions involve applying learning theories, encouraging research methods and to develop integrated collaborate protocols and support to synergize efforts.

Constructing Approaches to Entrustable Professional Activity Development that Deliver Valid Descriptions of Professional Practice

Issue: Entrustable Professional Activities (EPAs) describe the core tasks health professionals must be competent performing prior to promotion and/or moving into unsupervised practice. When used for learner assessment, they serve as gateways to increased responsibility and autonomy. It follows that identifying and describing EPAs is a high-stakes form of work analysis aiming to describe the core work of a profession. However, hasty creation and adoption of EPAs without rigorous attention to content threatens the quality of judgments subsequently made from using EPA-based assessment tools. There is a clear need for approaches to identify validity evidence for EPAs themselves prior to their deployment in workplace-based assessment. Evidence: For EPAs to realize their potential in health professions education, they must first be constructed to reflect accurately the work of that profession or specialty. If the EPAs fail to do so, they cannot predict a graduate's readiness for or future performance in professional practice. Evaluating the methods used for identification, description, and adoption of EPAs through a construct validity lens helps give leaders and stakeholders of EPA development confidence that the EPAs constructed are, in fact, an accurate representation of the profession's work. Implications: Application of a construct validity lens to EPA development impacts all five commonly followed steps in EPA development: selection of experts; identification of candidate EPAs; iterative revisions; evaluation of proposed EPAs; and formal adoption of EPAs into curricula. It allows curricular developers to avoid pitfalls, bias, and common mistakes. Further, construct validity evidence for EPA development provides assurance that the EPAs adopted are appropriate for use in workplace-based assessment and entrustment decision-making.

Professionalism and inter-communication skills (ICS): a multi-site validity study assessing proficiency in core competencies and milestones in medical learners

BACKGROUND

Interpersonal and Communication Skills (ICS) and Professionalism milestones are challenging to evaluate during medical training. Paucity in proficiency, direction and validity evidence of assessment tools of these milestones warrants further research. We validated the reliability of the previously-piloted Instrument for Communication skills and Professionalism Assessment (InCoPrA) in medical learners.

METHODS

This validity approach was guided by the rigorous Kane's Framework. Faculty-raters and standardized patients (SPs) used their respective InCoPrA sub-component to assess distinctive domains pertinent to ICS and Professionalism through multiple expert-built simulated-scenarios comparable to usual care. Evaluations included; inter-rater reliability of the faculty total score; the correlation between the total score by the SPs; and the average of the total score by two-faculty members. Participants were surveyed regarding acceptability, realism, and applicability of this experience.

RESULTS

Eighty trainees and 25 faculty-raters from five medical residency training sites participated. ICC of the total score between faculty-raters was generally moderate (ICC range 0.44-0.58). There was on average a moderate linear relationship between the SPs and faculty total scores (Pearson correlations range 0.23-0.44). Majority of participants ascertained receiving a meaningful, immediate, and comprehensive patient-faculty feedback.

CONCLUSIONS

This work substantiated that InCoPrA was a reliable, standardized, evidence-based, and user-friendly assessment tool for ICS and Professionalism milestones. Validating InCoPrA showed generally-moderate agreeability and high acceptability. Using InCoPrA also promoted engaging all stakeholders in medical education and training-faculty, learners, and SPs-using simulation-media as pathway for comprehensive feedback of milestones growth.

Introducing Summative Progress Testing in Radiology Residency: Little Change in Residents' Test Results After Transitioning from Formative Progress Testing

INTRODUCTION

Educational effects of transitioning from formative to summative progress testing are unclear. Our purpose was to investigate whether such transitioning in radiology residency is associated with a change in progress test results.

METHODS

We investigated a national cohort of radiology residents (N > 300) who were semi-annually assessed through a mandatory progress test. Until 2014, this test was purely formative for all residents, but in 2014/2015, it was transitioned (as part of a national radiology residency program revision) to include a summative pass requirement for new residents. In 7 posttransitioning tests in 2015-2019, including summatively and formatively tested residents who followed the revised and pre-transitioning residency program, respectively, we assessed residents' relative test scores and percentage of residents that reached pass standards.

RESULTS

Due to our educational setting, most posttransitioning tests had no residents in the summative condition in postgraduate year 4-5, nor residents in the formative condition in year 0.5-2. Across the 7 tests, relative test scores in postgraduate year 1-3 of the summative resident group and year 3.5-4.5 of the formative group differed significantly (p < 0.01 and p < 0.05, respectively, Kruskal-Wallis test). However, scores fluctuated without consistent time trends and without consistent differences between both resident groups. Percentage of residents reaching the pass standard did not differ significantly across tests or between groups.

DISCUSSION

Transitioning from formative to summative progress testing was associated with overall steady test results of the whole resident group in 4 post-transitioning years. We do not exclude that transitioning may have positive educational effects for resident subgroups.

An Appreciative Inquiry Approach to the Core Competencies: Taking it From Theory to Practice

OBJECTIVE

To operationalize the surgical core competencies by using a qualitative inquiry strategy to explore how surgical competence is behaviorally demonstrated by faculty.

DESIGN

Categorical general and vascular surgery residents completed a survey soliciting opinions regarding which faculty were deemed most representative of each core competency. The surveys served as a theoretical sample, as surgeons selected were then interviewed, and interviews transcribed. A qualitative research approach using grounded theory coding methods was used for transcript analysis. Iterative coding was performed, and emergent themes were then extracted from transcript analysis.

SETTING

Southern Illinois University School of Medicine, Department of Surgery in Springfield, IL, a tertiary academic center.

PARTICIPANTS

Fourteen of 19 residents completed the survey (74% response rate). Two surgeons were selected for each competency. A total of 7 interviews were performed, with 4 surgeons being chosen for 2 competencies.

RESULTS

Emergent themes revealed that competent surgeons shared qualities that drove their development and execution of each competency. These qualities included self-awareness, a selfless character, responsibility and ownership, context awareness, reliance on relationships and community, and a pattern of habit formation and discipline. Additionally, the competencies were noted to be pursued in an interrelated and interdependent fashion.

CONCLUSIONS

Surgeons deemed competent in any core domain shared common qualities. Further study exploring how each of these is identified, developed and taught is warranted. The competencies are an inter-related matrix whose development and execution correlates with foundational personal disciplines.

Integrating Specialty-Specific Clinical Anatomy Education into the Post-Clerkship Curriculum

Anatomy is a foundational science mainstay of undergraduate medical school education, particularly in the pre-clerkship curriculum. During the post-clerkship curriculum, students closer to graduate medical education may benefit from a focused concentration on human anatomy related to their specific clinical interests. Here, we describe a course for post-clerkship students that uniquely incorporates a multimodal approach of dissection, didactics, and clinical correlation to radiologic imaging, with the opportunity to personalize student learning on a specialty-specific anatomic region. The course increased students' confidence of anatomical knowledge and its clinical relevance. Other institutions may benefit from establishing a similar multimodal integrated post-clerkship anatomy curriculum.

Miles to Go Before We Sleep. Reforming the Pulmonary and Critical Care Milestones to Improve Trainee Assessment

The Accreditation Council for Graduate Medical Education (ACGME) Milestones are a systematic assessment framework for medical trainees within the six core competencies of practice. Their use by internal medicine subspecialties, including semiannual reports to the ACGME, was mandated beginning in 2014. The Milestones, which were based on specific, observable behaviors, improved upon the prior subjective, global comparisons of each fellow with an “average” fellow in his or her field and served the goals of competency-based medical education. However, the original set of Milestones has proven challenging to apply and interpret. Part of the challenge stems from the use of identical Milestones across all medicine subspecialties, which led to unclear relevance of the patient care and medical knowledge domains to the practice of pulmonary and critical care. This also precluded their use for individualized feedback or development of a learning plan for fellows. In addition, verbose behavioral descriptors, which were designed to provide specificity, ultimately led to rater fatigue among assessors and clinical competency committees. Therefore, the ACGME convened committees for each of the medical subspecialties to revise the original Milestones in an effort to improve subspecialty relevance, minimize educational jargon, and simplify the current iteration. New patient care and medical knowledge Milestones were created to be subspecialty specific and improve utility. The remaining four Milestones were developed as a common set of shorter Milestones, harmonized across specialties. For pulmonary, critical care, and combined fellowship programs, the resulting Milestones 2.0 aims to simplify the use, implementation, and interpretation of this framework for program directors, trainees, and society.

Objective Assessment and Standard Setting for Basic Flexible Ureterorenoscopy Skills Among Urology Trainees Using Simulation-Based Methods

Objective: To objectively assess the performance of graduating urology residents performing flexible ureterorenoscopy (fURS) using a simulation-based model and to set an entrustability standard or benchmark for use across the educational spectrum. Methods: Chief urology residents and attending endourologists performed a standardized fURS task (ureterorenoscopy and repositioning of stones) using a Boston Scientific^© Lithovue ureteroscope on a Cook Medical^© URS model. All performances were video-recorded and blindly scored by both endourology experts and crowd-workers (C-SATS) using the Ureteroscopic Global Rating Scale, plus an overall entrustability score. Validity evidence supporting the scores was collected and categorized. The Borderline Group (BG) method was used to set absolute performance standards for the expert and crowdsourced ratings. Results: A total of 44 participants (40 chief residents, 4 faculties) completed testing. Eighty-three percent of participants had performed >50 fURS cases at the time of the study. Only 47.7% (mean score 12.6/20) and 61.4% (mean score 12.4/20) of participants were deemed "entrustable" by experts and crowd-workers, respectively. The BG method produced entrustability benchmarks of 11.8/20 for experts and 11.4/20 for crowd-worker ratings, resulting in pass rates of 56.9% and 61.4%. Conclusion: Using absolute standard setting methods, benchmark scores were set to identify trainees who could safely carry out fURS in the simulated setting. Only 60% of residents in our cohort were rated as entrustable. These findings support the use of benchmarks to earlier identify trainees requiring remediation.

A Theoretical Framework and Competency-Based Approach to Training in Guideline Development

BACKGROUND

There is increasing requirement to develop guidelines using transparent, standardized, and rigorous methods. Consequently, a better understanding of the knowledge, skills, and expertise necessary for guideline development is needed. The aim of this manuscript is to describe a theoretical framework of knowledge and skills that are required for individuals to serve on a guideline panel in varying capacities.

METHODS

Based on an iterative process and review of published manuscripts focused on guideline development, we identified competencies, subcompetencies, and milestones.

RESULTS

Using a competency-based approach to training and the Dreyfus model of skill acquisition, we identified three core competencies: (1) facilitate the development of guideline structure and setup, (2) make judgments about the quality or certainty of the evidence, and (3) transform evidence to a recommendation. Level 1 focuses on recognizing and acknowledging the importance of a specific skill or behavior. Levels 2 and 3 require learners to demonstrate progressive acquisition of knowledge and application to specific behaviors. Level 4 represents the individual who has acquired the requisite knowledge and can function independently, while level 5 represents the mastery/aspirational level.

DISCUSSION

We propose a preliminary competency-based education framework that will (1) help standardize the qualifications needed for individuals to serve on guideline panels in varying capacities or (2) help with curricula development for teaching and training of guideline panel members. This framework can also help enable guideline-producing organizations to identify guideline methodologists with the relevant and appropriate level of knowledge and skills to lead guidelines. Validation of the framework and further refinement of the competencies and milestones will be required before widespread adoption.

Diagnostic Testing During Pediatric Hospitalizations: The Role of Attending In-House Coverage and Daytime Exposure

OBJECTIVE

Overuse of diagnostic tests is of particular concern for pediatric academic medical centers. Our objective was to measure variation in testing based on proportion of hospitalization during the day versus night and the association between attending in-house coverage on the teaching service and test utilization for hospitalized pediatric patients.

METHODS

Electronic health record data from 11,567 hospitalizations to a large, Northeastern, academic pediatric hospital were collected between January 2007 and December 2010. The patient-level dataset included orders for laboratory and imaging tests, information about who placed the order, and the timing of the order. Using a cross-sectional effect modification analysis, we estimated the difference in test utilization attributable to attending in-house coverage.

RESULTS

We found that admission to the teaching service was independently associated with higher utilization of laboratory and imaging tests. However, the number of orders was 0.76 lower (95% confidence interval:-1.31 to -0.21, P = .006) per 10% increase in the proportion in the share of the hospitalization that occurred during daytime hours on the teaching services, which is attributable to direct attending supervision.

CONCLUSIONS

Direct attending care of hospitalized pediatric patients at night was associated with slightly lower diagnostic test utilization.

Orthopaedic Surgery Residency Milestones: Initial Formulation and Future Directions

Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.

Using Longitudinal Milestones Data and Learning Analytics to Facilitate the Professional Development of Residents: Early Lessons From Three Specialties

PURPOSE

To investigate the effectiveness of using national, longitudinal milestones data to provide formative assessments to identify residents at risk of not achieving recommended competency milestone goals by residency completion. The investigators hypothesized that specific, lower milestone ratings at earlier time points in residency would be predictive of not achieving recommended Level (L) 4 milestones by graduation.

METHOD

In 2018, the investigators conducted a longitudinal cohort study of emergency medicine (EM), family medicine (FM), and internal medicine (IM) residents who completed their residency programs from 2015 to 2018. They calculated predictive values and odds ratios, adjusting for nesting within programs, for specific milestone rating thresholds at 6-month intervals for all subcompetencies within each specialty. They used final milestones ratings (May-June 2018) as the outcome variables, setting L4 as the ideal educational outcome.

RESULTS

The investigators included 1,386 (98.9%) EM residents, 3,276 (98.0%) FM residents, and 7,399 (98.0%) IM residents in their analysis. The percentage of residents not reaching L4 by graduation ranged from 11% to 31% in EM, 16% to 53% in FM, and 5% to 15% in IM. Using a milestone rating of L2.5 or lower at the end of post-graduate year 2, the predictive probability of not attaining the L4 milestone graduation goal ranged from 32% to 56% in EM, 32% to 67% in FM, and 15% to 36% in IM.

CONCLUSIONS

Longitudinal milestones ratings may provide educationally useful, predictive information to help individual residents address potential competency gaps, but the predictive power of the milestones ratings varies by specialty and subcompetency within these 3 adult care specialties.

A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence

PURPOSE

To investigate whether clinical competency committees (CCCs) were consistent in applying milestone ratings for first-year residents over time or whether ratings increased or decreased.

METHOD

Beginning in December 2013, the Accreditation Council for Graduate Medical Education (ACGME) initiated a phased-in requirement for reporting milestones; emergency medicine (EM), diagnostic radiology (DR), and urology (UR) were among the earliest reporting specialties. The authors analyzed CCC milestone ratings of first-year residents from 2013 to 2016 from all ACGME-accredited EM, DR, and UR programs for which they had data. The number of first-year residents in these programs ranged from 2,838 to 2,928 over this time period. The program-level average milestone rating for each subcompetency was regressed onto the time of observation using a random coefficient multilevel regression model.

RESULTS

National average program-level milestone ratings of first-year residents decreased significantly over the observed time period for 32 of the 56 subcompetencies examined. None of the other subcompetencies showed a significant change. National average in-training examination scores for each of the specialties remained essentially unchanged over the time period, suggesting that differences between the cohorts were not likely an explanatory factor.

CONCLUSIONS

The findings indicate that CCCs tend to become more stringent or maintain consistency in their ratings of beginning residents over time. One explanation for these results is that CCCs may become increasingly comfortable in assigning lower ratings when appropriate. This finding is consistent with an increase in confidence with the milestone rating process and the quality of feedback it provides.

Rising to the Challenge: Residency Programs' Experience With Implementing Milestones-Based Assessment

BACKGROUND

Changes to assessment efforts following the shift to milestones-based assessment in the ACGME Next Accreditation System have not been fully characterized.

OBJECTIVE

This study describes themes in initial milestones-based assessment practices with the goal of informing continued implementation and optimization of milestones-based assessment.

METHODS

Semistructured interviews were conducted with 15 residency program leaders in 6 specialties at 8 academic medical centers between August and December 2016. We explored what was retained, what was added, and what was changed from pre-milestones assessment efforts. We also examined the perceived impact of the shift to milestones-based assessment on the programs. Thematic analysis began after the first 5 interviews and ended once thematic sufficiency was reached. Two additional authors reviewed the codes, offered critical input, and informed the formation and naming of the final themes.

RESULTS

Three themes were identified: (1) program leaders faced challenges to effective implementation; (2) program leaders focused on adaptability and making milestones work in what felt like a less than ideal situation for them; and (3) despite challenges, program leaders see value and utility in their efforts to move to milestones-based assessment. We describe a number of strategies that worked for programs during the transition, with perceived benefits acknowledged.

CONCLUSIONS

While adaptation to milestones has occurred and benefits are noted, negative impacts and challenges (eg, perceived lack of implementation guidance and faculty development resources) persist. There are important lessons learned (eg, utilizing implementation experiences formatively to improve curricula and assessment) in the transition to milestones-based assessment.

A crossover comparative study to assess efficacy of competency based medical education (CBME) and the traditional structured (TS) method in selected competencies of living anatomy of first year MBBS curriculum: A pilot study

BACKGROUND

Competency based medical education (CBME) is outcome based teaching methodology where a student learns a set of measurable competencies for early clinical exposure. Inspite of ample resources on CBME, there are limited studies on its implementation. This study will try to demonstrate improvement in the performance of students using CBME as a teaching tool over the traditional structured method (TS).

METHODS

Forty student volunteers were chosen and divided into two groups. The crossover design exposed the group of students to CBME and TS spread over two periods with a wash out period in between. The intervention group was exposed to selected list of competencies in living Anatomy with feedbacks and formative assessments. The summative assessments were held at the end of each period.

RESULTS

The mean scores of CBME and TS in group 1 is 130.625 and 113.65 while in group 2 is 139.425 and 112.075 respectively. The treatment and period effect is significant. Estimate of treatment effect is 22.1625. The average improvement in treatment scores is by 11%. Two tailed paired sample T test reveals significant improvement in the scores post intervention.

CONCLUSION

CBME method produces better performance of the students in the competencies of living anatomy.

Assessing an assessment: The review and redesign of a competency-based mid-degree evaluation

Introduction

Little has been published about competency-based education in academic medicine, in particular how competencies are or should be assessed. This paper re-examines a competency-based assessment for M.S. students in clinical research, and “assesses the assessment” 4 years into its implementation.

Methods

Data were gathered from student surveys and interviews with program advisors, and common themes were identified. We then made refinements to the assessment, and student surveys were administered to evaluate the impact of the changes.

Results

Research results suggested the need to improve communication, time the assessment to align with skills development and opportunities for planning, streamline, and clarify expectations with examples and templates. After implementing these changes, data suggest that student satisfaction has improved without any reduction in academic rigor.

Conclusion

The effective implementation of competency-based training in clinical and translational research requires the development of a scholarly literature on effective methods of assessment. This paper contributes to that nascent body of research.

A review of eye tracking for understanding and improving diagnostic interpretation

Inspecting digital imaging for primary diagnosis introduces perceptual and cognitive demands for physicians tasked with interpreting visual medical information and arriving at appropriate diagnoses and treatment decisions. The process of medical interpretation and diagnosis involves a complex interplay between visual perception and multiple cognitive processes, including memory retrieval, problem-solving, and decision-making. Eye-tracking technologies are becoming increasingly available in the consumer and research markets and provide novel opportunities to learn more about the interpretive process, including differences between novices and experts, how heuristics and biases shape visual perception and decision-making, and the mechanisms underlying misinterpretation and misdiagnosis. The present review provides an overview of eye-tracking technology, the perceptual and cognitive processes involved in medical interpretation, how eye tracking has been employed to understand medical interpretation and promote medical education and training, and some of the promises and challenges for future applications of this technology.

Qualitative study to identify the perception and challenges faced by the faculty of community medicine in the implementation of competency-based medical education for postgraduate students

Objectives

To identify perception about the key aspects of competency-based medical education (CBME) among community medicine faculty members and to ascertain the various challenges faced by them during its implementation.

Methods

A descriptive qualitative study of 2 months’ duration was conducted among the faculty members of the community medicine department. Non-probability purposive sampling was employed in the study. Free listing was done initially to elicit the views of faculty members to meet the intended objectives. Visual Anthropac software was used to identify the salient variables using Smith’s Salience Score, and then pile sorting was done to identify the association between the salient variables.

Results

Three faculty members participated in the free listing and pile sorting. A total of 20 responses were obtained pertaining to the key aspects of CBME, of which 12 were identified as the salient variables depending on the cut-off value of 0.125 (Smith’s Salience Score) and subjected to pile sorting. Similarly, eight challenges were identified in the implementation of the programme during the free listing, and all were included in the second stage of pile sorting. Cognitive maps were drawn to understand the relationship between the key aspects of CBME and involved challenges separately.

Conclusion

On employing the free listing and pile sorting methods, formulation of entrustable professional activities and their assessment using appropriate tools were the identified crucial areas in CBME, while the lack of sensitisation of stakeholders and inadequate planning were identified as the predominant challenges in the implementation of CBME.

What we measure … and what we should measure in medical education

CONTEXT

As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society.

OBJECTIVES

We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role.

METHODS

We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies.

DISCUSSION

As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.

Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions

BACKGROUND

Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.

OBJECTIVE

We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.

METHODS

We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015-2016. Individual CCC members biannually reported characteristics of their review process and Accreditation Council for Graduate Medical Education milestone levels and recommended supervisory role categorizations assigned to residents. Relationships among characteristics of CCC member reviews, mean milestone levels, and supervisory role categorizations were analyzed using mixed-effects linear regression, reported as mean differences with 95% confidence intervals (CIs), and Bayesian mixed-effects ordinal regression, reported as odds ratios (ORs) and 95% credible intervals (CrIs).

RESULTS

A total of 155 CCC members participated. Members who provided milestones or other professional development feedback after CCC meetings assigned significantly lower mean milestone levels (mean 1.4 points; CI -2.2 to -0.6; P < .001) and were significantly less likely to recommend supervisory responsibility in any setting (OR = 0.23, CrI 0.05-0.83) compared with CCC members who did not. Members recommended less supervisory responsibility when they reviewed more residents (OR = 0.96, 95% CrI 0.94-0.99) and participated in more review cycles (OR = 0.22, 95% CrI 0.07-0.63).

CONCLUSIONS

This study explored the association between characteristics of individual CCC member reviews and their summative assessment decisions about residents. Further study is needed to gain deeper understanding of factors influencing CCC members' summative assessment decisions.

National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-up

PURPOSE

To evaluate validity evidence for internal medicine milestone ratings across programs for three resident cohorts by quantifying "not assessable" ratings; reporting mean longitudinal milestone ratings for individual residents; and correlating medical knowledge ratings across training years with certification examination scores to determine predictive validity of milestone ratings for certification outcomes.

METHOD

This retrospective study examined milestone ratings for postgraduate year (PGY) 1-3 residents in U.S. internal medicine residency programs. Data sources included milestone ratings, program characteristics, and certification examination scores.

RESULTS

Among 35,217 participants, there was a decreased percentage with "not assessable" ratings across years: 1,566 (22.5%) PGY1s in 2013-2014 versus 1,219 (16.6%) in 2015-2016 (P = .01), and 342 (5.1%) PGY3s in 2013-2014 versus 177 (2.6%) in 2015-2016 (P = .04). For individual residents with three years of ratings, mean milestone ratings increased from around 3 (behaviors of an early learner or advancing resident) in PGY1 (ranging from a mean of 2.73 to 3.19 across subcompetencies) to around 4 (ready for unsupervised practice) in PGY3 (mean of 4.00 to 4.22 across subcompetencies, P < .001 for all subcompetencies). For each increase of 0.5 units in two medical knowledge (MK1, MK2) subcompetency ratings, the difference in examination scores for PGY3s was 19.5 points for MK1 (P < .001) and 19.0 for MK2 (P < .001).

CONCLUSIONS

These findings provide evidence of validity of the milestones by showing how training programs have applied them over time and how milestones predict other training outcomes.

Competency Crosswalk: A Multispecialty Review of the Accreditation Council for Graduate Medical Education Milestones Across Four Competency Domains

PURPOSE

To identify common and overlapping themes among the interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP) milestones of the transitional year and 26 specialties.

METHOD

In May 2017, milestones were accessed from the Accreditation Council for Graduate Medical Education specialties website. A thematic analysis of the ICS, PBLI, PROF, and SBP milestones was performed to determine unique and common themes across these competencies and across specialties. Keywords from the common program requirements were initially applied as codes to the milestones. Codes were then grouped into common themes.

RESULTS

Twenty-two themes were identified: 15 (68%) were unique to a given competency (3 related to ICS, 4 related to PBLI, 5 related to PROF, and 3 related to SBP), and 7 (32%) appeared in the milestones of more than one core competency. Eleven themes (50%) were used by 20 or more specialties, and 6 themes (27%) by 10 or fewer specialties. No theme was present across all specialties.

CONCLUSIONS

The ICS, PBLI, PROF, and SBP milestones contain multiple themes with areas of overlap among these four competencies and substantial variability across specialties. This variability may create differential expectations of residents across specialties, complicate faculty development, and make sharing assessment tools difficult. The thematic analysis provides important insights into how individual specialties interpret and operationalize the ICS, PBLI, PROF, and SBP competency domains and can inform future revisions of milestones to enable harmonization and shared understanding of these competencies across specialties where appropriate.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Step Up-Not On-The Step 2 Clinical Skills Exam: Directors of Clinical Skills Courses (DOCS) Oppose Ending Step 2 CS

Recently, a student-initiated movement to end the United States Medical Licensing Examination Step 2 Clinical Skills and the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation has gained momentum. These are the only national licensing examinations designed to assess clinical skills competence in the stepwise process through which physicians gain licensure and certification. Therefore, the movement to end these examinations and the ensuing debate merit careful consideration. The authors, elected representatives of the Directors of Clinical Skills Courses, an organization comprising clinical skills educators in the United States and beyond, believe abolishing the national clinical skills examinations would have a major negative impact on the clinical skills training of medical students, and that forfeiting a national clinical skills competency standard has the potential to diminish the quality of care provided to patients. In this Perspective, the authors offer important additional background information, outline key concerns regarding the consequences of ending these national clinical skills examinations, and provide recommendations for moving forward: reducing the costs for students, exploring alternatives, increasing the value and transparency of the current examinations, recognizing and enhancing the strengths of the current examinations, and engaging in a national dialogue about the issue.

Competency-Based Medical Education and the Ghost of Kuhn: Reflections on the Messy and Meaningful Work of Transformation

The transition, if not transformation, to outcomes-based medical education likely represents a paradigm shift struggling to be realized. Paradigm shifts are messy and difficult but ultimately meaningful if done successfully. This struggle has engen dered tension and disagreements, with many of these disagreements cast as either-or polarities. There is little disagreement, however, that the health care system is not effectively achieving the triple aim for all patients. Much of the tension and polarity revolve around how more effectively to prepare students and residents to work in and help change a complex health care system.Competencies were an initial attempt to facilitate this shift by creating frameworks of essential abilities needed by physicians. However, implementation of competencies has proven to be difficult. Entrustable professional activities (EPAs) in undergraduate and graduate medical education and Milestones in graduate medical education are recent concepts being tried and studied as approaches to guide the shift to outcomes. Their primary purpose is to help facilitate implementation of an outcomes-based approach by creating shared mental models of the competencies, which in turn can help to improve curricula and assessment. Understanding whether and how EPAs and Milestones effectively facilitate the shift to outcomes has been and will continue to be an iterative and ongoing reflective process across the entire medical education community using lessons from implementation and complexity science. In this Invited Commentary, the author reflects on what got the community to this point and some sources of tension involved in the struggle to move to outcomes-based education.

Thresholds and interpretations: How clinical competency committees identify pediatric residents with performance concerns

BACKGROUND

Clinical competency committee (CCC) identification of residents with performance concerns is critical for early intervention.

METHODS

Program directors and 94 CCC members at 14 pediatric residency programs responded to a written survey prompt asking them to describe how they identify residents with performance concerns. Data was analyzed using thematic analysis.

RESULTS

Six themes emerged from analysis and were grouped into two domains. The first domain included four themes, each describing a path through which residents could meet or exceed a concern threshold:1) written comments from rotation assessments are foundational in identifying residents with performance concerns, 2) concerning performance extremes stand out, 3) isolated data points may accumulate to raise concern, and 4) developmental trajectory matters. The second domain focused on how CCC members and program directors interpret data to make decisions about residents with concerns and contained 2 themes: 1) using norm- and/or criterion-referenced interpretation, and 2) assessing the quality of the data that is reviewed.

CONCLUSIONS

Identifying residents with performance concerns is important for their education and the care they provide. This study delineates strategies used by CCC members across several programs for identifying these residents, which may be helpful for other CCCs to consider in their efforts.

It's not just what you know: junior trainees' approach to follow-up and documentation

CONTEXT

In teaching hospitals, junior trainees (first-year residents and third-year medical students) are responsible for patient follow-up and documentation under the supervision of senior team members. In order to support trainees in their role, supervisors need to understand how trainees approach these tasks and how they can be coached to develop best practices.

OBJECTIVES

The purpose of our study was to explore the range of practices used by junior trainees in clinical settings.

METHODS

Constructivist grounded theory was used to guide the collection and analysis of data on follow-up and documentation during 34 observation periods with 17 junior trainees. Data sources included field notes, field interviews and de-identified copies of patient charts. We also held two focus groups with four attending physicians in each.

RESULTS

We were able to describe three interrelated characteristics that influenced a trainee's approach to and ability to perform the tasks of patient follow-up and documentation: (i) diligence; (ii) relationship to the team (dependent, independent, collaborative), and (iii) level of performance (Data Gatherer, Sensemaker, Manager). Diligence and relationship to the team appeared to influence the quality and focus of a trainee's approach at all levels of performance. Level of performance was felt, by focus group attending physicians, to reflect a developmental progression of knowledge and skills.

CONCLUSIONS

Our findings contribute to the existing literature in three ways. Firstly, they extend our understanding of how junior trainees approach the task of in-patient follow-up and clinical documentation and the value of those activities. Secondly, they provide new insights to support formative and summative assessment. Finally, they contribute to a growing body of literature exploring the factors that impact trainees' roles and interactions with the team. Future research should focus on validating our findings and exploring their utility in the development of novel assessment strategies.