Evaluation of Medical School Grading Variability in the United States: Are All Honors the Same?

R.I.M.E. and reason: multi-station OSCE enhancement to neutralize grade inflation

To offset grade inflation, many clerkships combine faculty evaluations with objective assessments including the Medical Examiners Subject Examination (NBME-SE) or Objective Structured Clinical Examination (OSCE), however, standardized methods are not established. Following a curriculum transition removing faculty clinical evaluations from summative grading, final clerkship designations of fail (F), pass (P), and pass-with-distinction (PD) were determined by combined NBME-SE and OSCE performance, with overall PD for the clerkship requiring meeting this threshold in both. At the time, 90% of students achieved PD on the Internal Medicine (IM) OSCE resulting in overall clerkship grades primarily determined by the NBME-SE. The clerkship sought to enhance the OSCE to provide a more thorough objective clinical skills assessment, offset grade inflation, and reduce the NBME-SE primary determination of the final clerkship grade. The single-station 43-point OSCE was enhanced to a three-station 75-point OSCE using the Reporter-Interpreter-Manager-Educator (RIME) framework to align patient encounters with targeted assessments of progressive skills and competencies related to the clerkship rotation. Student performances were evaluated pre- and post-OSCE enhancement. Student surveys provided feedback about the clinical realism of the OSCE and the difficulty. Pre-intervention OSCE scores were more tightly clustered (SD = 5.65%) around a high average performance with scores being highly negatively skewed. Post-intervention OSCE scores were more dispersed (SD = 6.88%) around a lower average with scores being far less skewed resulting in an approximately normal distribution. This lowered the total number of students achieving PD on the OSCE and PD in the clerkship, thus reducing the relative weight of the NMBE-SE in the overall clerkship grade. Student response was positive, indicating the examination was fair and reflective of their clinical experiences. Through structured development, OSCE assessment can provide a realistic and objective measurement of clinical performance as part of the summative evaluation of students.

Resident Recruitment in a New Era

ABSTRACT

This chapter focuses on resident recruitment and recent US National Resident Matching Program changes and the impact in the evaluation and ranking of applicants within the specialty of anesthesiology. Recruitment challenges are examined as well as program strategies and potential future directions. Also discussed are DEI initiatives within the recruitment process.

Analysis of variability and trends in medical school clerkship grades

Background

Medical school clerkship grades are used to evaluate orthopedic surgery residency applicants, however, high interinstitutional variability in grade distribution calls into question the utility of clerkship grades when evaluating applicants from different medical schools. This study aims to evaluate the variability in grade distribution among medical schools and look for trends in grade distribution over recent years.

Methods

Applications submitted to Rush University's orthopedic surgery residency program from 2015, 2019, and 2022 were collected from the Electronic Residency Application Service. Applications from the top 100 schools according to the 2023-2024 U.S. News and World Report Research Rankings were reviewed. The percentage of "honors" grades awarded by medical schools for the surgery and internal medicine clerkships were extracted from applicants' Medical Student Performance Evaluation letters.

Results

The median percentage of honors given in 2022 was 36.0 % (range 10.0-82.0) for the surgery clerkship and 33.0 % (range 6.7-80.0) for the internal medicine clerkship. Honors were given 6.6 % more in the surgery clerkship in 2022 compared to 2015. There was a negative correlation between a higher (worse) U.S. News and World Report research ranking and the percentage of honors awarded in 2022 for the surgery and internal medicine clerkships.

Conclusion

There is substantial interinstitutional variability in the rate that medical schools award an "honors" grade with evidence of grade inflation in the surgery clerkship. Residency programs using clerkship grades to compare applicants should do so cautiously provided the variability demonstrated in this study.

Predictors of Ophthalmology Resident Performance From Medical Student Application Materials

OBJECTIVE

To determine whether elements in ophthalmology residency applications are predictors of future resident performance.

DESIGN

This multi-institutional, cross-sectional, observational study retrospectively reviewed the residency application materials of ophthalmology residents who graduated from residency from 2006 through 2018. Resident performance was scored by 2 faculty reviewers in 4 domains (clinical, surgical, academic, and global performance). Correlation between specific elements of the residency application and resident performance was assessed by Spearman correlation coefficients (univariate) and linear regression (multivariate) for continuous variables and logistic regression (multivariate) for categorical variables.

SETTING

Seven ophthalmology residency programs in the US.

PARTICIPANTS

Ophthalmology residents who graduated from their residency program.

RESULTS

High-performing residents were a diverse group, in terms of sex, ethnicity, visa status, and educational background. Residents with United States Medical Licensing Examination Step 1 scores higher than the national average for that year had significantly higher scores in all 4 performance domains than those who scored at or below the mean (all domains P < 0.05). Residents who had honors in at least 4 core clerkships and who were members of Alpha Omega Alpha Medical Honor Society also had higher scores in all 4 performance domains (all domains P ≤ 0.04). Step 1 score (ρ=0.26, P < 0.001) and the difference between Step 1 score and the national average for that year (ρ=0.19, P = 0.009) positively correlated with total resident performance scores. Residents who passed the American Board of Ophthalmology Written Qualifying Examination or Oral Examination on their first attempt had significantly higher Step 1/2 scores (P ≤ 0.005), Ophthalmology Knowledge Assessment Program scores (P = 0.001), and resident performance scores (P ≤ 0.004).

CONCLUSIONS

In this new landscape of increasing numbers of applicants to residency programs and changing of the Step 1 score to pass/fail, our findings may help guide selection committees as they holistically review applicants to select exceptional future residents in ophthalmology.

All medical degrees are equal, but some are more equal than others: An analysis of medical degree classifications

BACKGROUND

Inequity in assessment can lead to differential attainment. Degree classifications, such as 'Honours', are an assessment outcome used to differentiate students after graduation. However, there are no standardised criteria used to determine what constitutes these awards.

METHODS

We contacted all medical schools in the UK and collected data relating to classifications awarded, criteria used and percentage of students receiving classifications across the 5-year period prior to the 2019/2020 academic year.

RESULTS

All 42 UK medical schools responded, and 36 universities provided usable data. Of these 36 universities, 30 (83%) awarded classifications above a 'Pass'. We identified four classifications above a 'Pass', and these were 'Commendation', 'Merit', 'Distinction' and 'Honours'. Sixteen (44%) universities awarded a single additional classification, and 14 (39%) universities awarded two or more. There was considerable variation in the criteria used by each university to award classifications. For example, 30 (67%) out of 45 classifications were dependent on all examined years, 9 (20%) for a combination of years and 6 (13%) for final year alone. Twenty-five of 30 universities that awarded classifications provided data on the percentage of students awarded a classification, and a median of 15% of students received any type of classification from their university (range 5.3% to 38%). There was a wide range in the percentage of students awarded each classification type across the universities (e.g. Honours, range = 3.1%-24%).

CONCLUSIONS

We demonstrate considerable variation in the way UK medical degree classifications are awarded-regarding terminology, criteria and percentage of students awarded classifications. We highlight that classifications are another form of inequity in medical education. There is a need to fully evaluate the value of hierarchical degree awards internationally as the consequential validity of these awards is understudied.

The Problematic Persistence of Tiered Grading in Medical School

Issue: The evaluation of medical students is a critical, complex, and controversial process. It is tightly woven into the medical school curriculum, beginning at the inception of the medical student's professional journey. In this respect, medical student evaluation is among the first in a series of ongoing, lifelong assessments that influence the interpersonal, ethical, and socioeconomic dimensions necessary for an effective physician workforce. Yet, tiered grading has a questionable historic pedagogic basis in American medical education, and evidence suggests that tiered grading itself is a source of student burnout, anxiety, depression, increased competitiveness, reduced group cohesion, and racial biases. Evidence: In its most basic form, medical student evaluation is an assessment of the initial cognitive and technical competencies ultimately needed for the safe and effective practice of contemporary medicine. At many American medical schools, such evaluation relies largely on norm-based comparisons, such as tiered grading. Yet, tiered grading can cause student distress, is considered unfair by most students, is associated with biases against under-represented minorities, and demonstrates inconsistent correlation with residency performance. While arguments that tiered grading motivates student performance have enjoyed historic precedence in academia, such arguments are not supported by robust data or theories of motivation. Implications: Given the evolving recognition of the deleterious effects on medical student mental health, cohesiveness, and diversity, the use of tiered grading in medical schools to measure or stimulate academic performance, or by residency program directors to distinguish residency applicants, remains questionable. Examination of tiered grading in its historical, psychometric, psychosocial, and moral dimensions and the various arguments used to maintain it reveals a need for investigation of, if not transition to, alternative and non-tiered assessments of our medical students.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

Examination Scores but not Clinical Performance Correlate With Duration of Preclinical Didactic Time: A Synchronous Comparison of Second- Versus Third-Year Medical Students on the Surgery Clerkship

OBJECTIVE

Numerous institutions have reduced preclinical didactic time to facilitate earlier clinical exposure during the second year of medical education. However, the effects that shortened preclinical education may have on performance in the surgery clerkship are unclear. This study aims to compare the clinical and examination performance of second- (MS2) and third-year (MS3) students synchronously completing an identical surgery clerkship.

DESIGN

All students completing the surgery clerkship (identical didactics, examinations, clinical rotations, etc.) were included. MS3s received 24 months of preclinical education, whereas MS2s received 14 months. Performance outcomes included weekly quizzes based on lectures, NBME Surgery Shelf Exam, numeric clinical evaluations, objective structured clinical examination (OSCE) scores, and overall clerkship grades.

SETTING

University of Miami Miller School of Medicine.

PARTICIPANTS

All second- (MS2) and third-year (MS3) medical students completing the Surgery Clerkship over 1 year (n = 395).

RESULTS

There were 199 MS3 (50%) and 196 MS2 (50%) students. MS3s demonstrated higher median shelf exams (77% vs 72% MS2s), weekly quiz score averages (87% vs 80% MS2s), clinical evaluations (96% vs 95% MS2s), and overall clerkship grades (89% vs 87% MS2s), all p < 0.020. There was no difference in median OSCE performance (both 92%; p = 0.499). A greater proportion of MS3 students performed in the highest 50% of weekly quiz scores (57% vs 43% MS2), NBME shelf exams (59% vs 39% MS2), and overall clerkship grades (45% vs 37% MS2), all p < 0.010. No significant difference in the proportion of students placing in the top 50% of clinical parameters including the OSCE (48% MS3 vs 46% MS2; p = 0.106) and clinical evaluations (45% MS3 vs 38%; p = 0.185) was observed.

CONCLUSIONS

Although the duration of preclerkship education may correspond to examination scores, MS2s and MS3s perform similarly on clinical metrics. Future strategies to enhance available preclinical didactic time and preparation for examinations are needed.

Identifying Meaningful Patterns of Internal Medicine Clerkship Grading Distributions: Application of Data Science Techniques Across 135 U.S. Medical Schools

PROBLEM

Residency program directors use clerkship grades for high-stakes selection decisions despite substantial variability in grading systems and distributions. The authors apply clustering techniques from data science to identify groups of schools for which grading distributions were statistically similar in the internal medicine clerkship.

APPROACH

Grading systems (e.g., honors/pass/fail) and distributions (i.e., percent of students in each grade tier) were tabulated for the internal medicine clerkship at U.S. MD-granting medical schools by manually reviewing Medical Student Performance Evaluations (MSPEs) in the 2019 and 2020 residency application cycles. Grading distributions were analyzed using k-means cluster analysis, with the optimal number of clusters selected using model fit indices.

OUTCOMES

Among the 145 medical schools with available MSPE data, 64 distinct grading systems were reported. Among the 135 schools reporting a grading distribution, the median percent of students receiving the highest and lowest tier grade was 32% (range: 2%-66%) and 2% (range: 0%-91%), respectively. Four clusters was the most optimal solution (η 2 = 0.8): cluster 1 (45% [highest grade tier]-45% [middle tier]-10% [lowest tier], n = 64 [47%] schools), cluster 2 (25%-30%-45%, n = 40 [30%] schools), cluster 3 (20%-75%-5%, n = 25 [19%] schools), and cluster 4 (15%-25%-25%-25%-10%, n = 6 [4%] schools). The findings suggest internal medicine clerkship grading systems may be more comparable across institutions than previously thought.

NEXT STEPS

The authors will prospectively review reported clerkship grading approaches across additional specialties and are conducting a mixed-methods analysis, incorporating a sequential explanatory model, to interview stakeholder groups on the use of the patterns identified.

Teaching about chronicity in medical schools - A review of the current situation

In recent decades, progressive population aging in developed countries has led to a significant increase in the number of people with at least one chronic medical condition. As a result, acquiring knowledge about chronicity in medical school is key for physicians to be able to provide proper management for these patients. However, the presence of chronicity in educational curricula is scarce and highly variable. On the one hand, this work consisted of a review of the educational programs of the main medical schools on each continent with the aim of identifying whether they included chronicity and, on the other, a literature review focused on identifying educational projects in the field of chronicity. The presence of chronicity in most medical schools' curricula is marginal and only a few universities include specific skills or competences linked to chronicity. In most cases, this topic appears as a global, cross-curricular competence that students are supposed to acquire over the course of their entire education. The literature review retrieved 21 articles on innovative teaching projects on chronicity. Direct contact with chronic patients, most times as "health mentors," the role of the student as a teacher, and continuous evaluation and feedback from all participants are the main characteristics of the projects analyzed. Some previously published experiences support the usefulness of innovative methodologies for better approaching this capital field in current medical practice. Despite the current situation in which chronic patients consume most healthcare resources, the presence of chronicity in medical schools is marginal. However, a literature review did identify some useful experiences for improving education on chronicity in medical schools. Medical schools should change their academic curricula and redirect them towards providing students all the necessary tools for improving their knowledge on chronicity.

An Elite Privilege: Top-Ranked Medical Schools Provide Fewer Comparative Performance Data on Their Students

PURPOSE

The objective of this study was to determine differences in the reporting of performance data on medical student performance evaluations (MSPEs) by medical school ranking.

METHODS

MSPEs from all US allopathic and osteopathic medical schools received by a single diagnostic radiology residency program during the 2021-2022 application cycle were retrospectively reviewed. Preclinical class and core clerkship grades were categorized as pass/fail or multitiered. Comparative summative assessments provided in the MSPEs were recorded. Medical schools were grouped by their US News & World Report rankings, and the proportion of reported performance metrics for each group was compared.

RESULTS

Information from 95% of US allopathic medical schools (148 of 155) and 73% of osteopathic medical schools (27 of 37) was collected, on the basis of 1,046 applications received. For preclinical classes, multitiered grading was reported by no schools ranked in the top 10, 17% of schools ranked 11th to 50th, 52% of schools ranked 51st to 100th, and 59% of unranked schools (P < .001). For core clinical clerkships, multitiered grades were reported by 70% of the top 10 ranked schools, 90% of schools ranked 11th to 50th, 94% of those ranked 51st to 100th, and 94% of unranked schools (P = .0463). Comparative summative assessments were reported by none of the top 10 ranked schools, 56% of schools ranked 11th to 50th, 80% of those ranked 51th to 100th, and 81% of unranked schools (P < .001).

CONCLUSIONS

Higher ranked medical schools are less likely to provide comparative assessment data on their MSPEs, which may disadvantage students from lower ranked medical schools.

The Effect of Pass/Fail Exam Grading on Exam Performance in a Pediatric Clerkship

OBJECTIVE

Clerkship grades are a component of determining a residency candidate's competitiveness. In 2017, the University of Minnesota Medical School's pediatric clerkship transitioned its standardized multiple-choice exam, the Aquifer Pediatrics Examination, to pass/fail with eligibility for honors being determined by clinical performance, not exam performance. We assessed the effect this change had on Aquifer exam performance and evaluated for correlation between Aquifer exam performance and clinical evaluation scores in order to gather insight into the validity of each type of assessment with respect to one another.

METHODS

We analyzed de-identified data from 750 medical students between the academic years of 2016 to 2017 and 2019 to 2020. Individual Aquifer exam scores were compared to individual clinical performance scores. Differences in exam performance before and after the transition to pass/fail were investigated with a two-sample t-test and Cohen's d for effect size.

RESULTS

No correlation was found between Aquifer exam scores and clinical performance scores. The mean Aquifer exam score prior to the transition to pass/fail was 80.02 ± 7.51 while the mean after the exam was made pass/fail was 77.8 ± 7.42. This difference was statistically significant (P < .001) with a Cohen's d (effect size) of 0.297.

CONCLUSIONS

A lack of correlation between the Aquifer exam scores and clinical performance scores was found. There was a small yet statistically significant decrease in Aquifer exam scores after the change to pass/fail; it is not clear if this represents a meaningful decrease in learning by students.

Exploring Potential Schedule-Related and Gender Biases in Ophthalmology Residency Interview Scores

Purpose Prior studies have revealed grading discrepancies in evaluation of personal statements and letters of recommendation based on candidate's race and gender. Fatigue and the end-of-day phenomenon can negatively impact task performance but have not been studied in the residency selection process. Our primary objective is to determine whether factors related to interview time and day as well as candidate's and interviewer's gender have a significant effect on residency interview scores.

Methods Seven years of ophthalmology residency candidate evaluation scores from 2013 to 2019 were collected at a single academic institution, standardized by interviewer into relative percentiles (0–100 point grading scale), and grouped into the following categories for comparisons: different interview days (Day 1 vs. Day 2), morning versus afternoon (AM vs. PM), interview session (Day 1 AM/PM vs. Day 2 AM/PM), before and after breaks (morning break, lunch break, and afternoon break), residency candidate's gender, and interviewer's gender.

Results Candidates in the morning sessions were found to have higher scores than afternoon sessions (52.75 vs. 49.28, p < 0.001). Interview scores in the early morning, late morning, and early afternoon were higher than late afternoon scores (54.47, 53.01, 52.15 vs. 46.74, p < 0.001). Across all interview years, there were no differences in scores received before and after morning breaks (51.71 vs. 52.83, p = 0.49), lunch breaks (53.01 vs. 52.15, p = 0.58), and afternoon breaks (50.35 vs. 48.30, p = 0.21). No differences were found in scores received by female versus male candidates (51.55 vs. 50.49, p = 0.21) or scores given by female versus male interviewers (51.31 vs. 50.84, p = 0.58).

Conclusion Afternoon residency candidate interview scores, especially late afternoon, were significantly lower than morning scores, suggesting the need to further study the effects of interviewer's fatigue in the residency interview process. The interview day, presence of break times, candidate's gender, and interviewer's gender had no significant effects on interview score.

Role of faculty characteristics in failing to fail in clinical clerkships

INTRODUCTION

In the context of competency-based medical education, poor student performance must be accurately documented to allow learners to improve and to protect the public. However, faculty may be reluctant to provide evaluations that could be perceived as negative, and clerkship directors report that some students pass who should have failed. Student perception of faculty may be considered in faculty promotion, teaching awards, and leadership positions. Therefore, faculty of lower academic rank may perceive themselves to be more vulnerable and, therefore, be less likely to document poor student performance. This study investigated faculty characteristics associated with low performance evaluations (LPEs).

METHOD

The authors analysed individual faculty evaluations of medical students who completed the third-year clerkships over 15 years using a generalised mixed regression model to assess the association of evaluator academic rank with likelihood of an LPE. Other available factors related to experience or academic vulnerability were incorporated including faculty age, race, ethnicity, and gender.

RESULTS

The authors identified 50 120 evaluations by 585 faculty on 3447 students between January 2007 and April 2021. Faculty were more likely to give LPEs at the midpoint (4.9%), compared with the final (1.6%), evaluation (odds ratio [OR] = 4.004, 95% confidence interval [CI] [3.59, 4.53]; p < 0.001). The likelihood of LPE decreased significantly during the 15-year study period (OR = 0.94 [0.90, 0.97]; p < 0.01). Full professors were significantly more likely to give an LPE than assistant professors (OR = 1.62 [1.08, 2.43]; p = 0.02). Women were more likely to give LPEs than men (OR = 1.88 [1.37, 2.58]; p 0.01). Other faculty characteristics including race and experience were not associated with LPE.

CONCLUSIONS

The number of LPEs decreased over time, and senior faculty were more likely to document poor medical student performance compared with assistant professors.

What Does Honors Mean? National Analysis of Medical School Clinical Clerkship Grading

OBJECTIVE

Medical Student Performance Evaluation (MSPE) letters provide critical comparative information about clerkship performance, and are a crucial part of the surgical residency application. The elimination of USMLE Step 1 numeric reporting increases the importance of transparency, standardization, and accessibility of comparative information reported on the MSPE. The objective of our study was to measure the variability in clerkship grade reporting on the MSPE from US medical schools, particularly focusing on the highest (honors) grades.

DESIGN, SETTING, AND PARTICIPANTS

We identified representative MSPE letters from US medical schools and recorded the percentage of honors for 5 core clerkships. We grouped medical schools according to medical school rankings, geographic region, and number of grading categories RESULTS: Of 122 medical schools, 106 schools (87%) reported their grading scheme and percent honors. The most commonly used grading scheme was a 4-tier system (51/122; 42%). The percentage of honors was highly variable (from 1-91%) and did not vary by region. However, schools in the top 20 research ranking were less likely to report grade comparisons (30% vs. 10%), and more likely to award more students honors in 4 of the 5 clerkships. Schools in the top 20 primary care ranking were more likely to award more honors in the medicine clerkship.

CONCLUSIONS

There is significant variability in the number of grading tiers used and the percentage of students awarded honors across US medical schools. Factors that correlated to higher grades included schools with higher rankings, and higher ranked schools were less likely to report comparative information at all.

Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum

BACKGROUND

With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need.

OBJECTIVES

To better understand the degree to which DOM LORs comply with published guidelines.

METHODS

We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines.

RESULTS

Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases.

CONCLUSIONS

Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.

Does Objective Scoring of Applications for Radiology Residency Affect Diversity?

RATIONALE AND OBJECTIVES

It is important to recognize if reliance on certain factors in applications affects selection of trainees from under-represented groups. Our purpose is to determine if objective scoring of radiology residency applications based on quantifiable data regarding academic performance, United States Medical Licensing Examination (USMLE) scores and research productivity affects selection of female and under-represented minority (URM) candidates.

MATERIALS AND METHODS

We reviewed 502 applications from three successive match cycles from United States allopathic medical students. Each application was scored for academic performance, USMLE results and research productivity determining an overall score. The scores of males were compared to females and URM were compared to non-URM candidates. USMLE cutoff scores were evaluated for disparate effects.

RESULTS

There were 348 male, 154 female, 73 URM and 429 non-URM candidates. For male versus female applicants, there was no significant difference in mean academic performance, USMLE Step 2 Clinical Knowledge and research productivity scores. Males had higher mean USMLE Step 1 (p = 0.005) and overall candidate scores (p = 0.02). Between URM and non-URM candidates there was no significant difference in academic performance. Non-URM applicants had higher mean USMLE Step 1 (p = 0.008), USMLE Step 2 (p = 0.002), research productivity (p = 0.001) and overall scores (p = 0.02). Use of USMLE cutoff scores demonstrated disparate effects on female and URM candidates.

CONCLUSION

Objective scoring of applications and use of USMLE cutoff scores can disadvantage candidates from underrepresented groups. Screening filters can affect the diversity of candidate pools for radiology residency.

Review of the Medical Student Performance Evaluation: analysis of the end-users' perspective across the specialties

The Medical Student Performance Evaluation (MSPE) is an important tool of communication used by program directors to make decisions in the residency application process. To understand the perspective and usage of the MSPE across multiple medical specialties now and in anticipation of the planned changes in USMLE Step 1 score-reporting. A survey instrument including quantitative and qualitative measures was developed and piloted. The final survey was distributed to residency programs across 28 specialties in 2020 via the main contact on the ACGME listserv. Of the 28 specialties surveyed, at least one response was received from 26 (93%). Eight percent of all programs (364/4675) responded to the survey, with most respondents being program directors. Usage of the MSPE varied among specialties. Approximately 1/3 of end-users stated that the MSPE is very or extremely influential in their initial screening process. Slightly less than half agreed or strongly agreed that they trust the information to be an accurate representation of applicants, though slightly more than half agree that the MSPE will become more influential once USMLE Step 1 becomes pass/fail. Professionalism was rated as the most important component and noteworthy characteristics among the least important in the decision-making process. Performance in the internal medicine clerkship was rated as the most influential while neurology and psychiatry performances were rated as less influential. Overwhelmingly, respondents suggested that including comparative performance and/or class rank would make the MSPE more useful once USMLE Step 1 becomes pass/fail. MSPE end-users across a variety of specialties utilize this complex document in different ways and value it differentially in their decision-making processes. Despite this, continued mistrust of the MSPE persists. A better understanding of end-users' perceptions of the MSPE offers the UME community an opportunity to transform the MSPE into a highly valued, trusted document of communication.

Underrepresented in medicine in graduate medical education: Historical trends, bias, and recruitment practices

Health inequities disproportionately impact children of color in the United States. Research demonstrates that a diverse physician workforce that reflects the demographic profile of the United States population has been associated with better health outcomes and physicians of color are more likely to work in communities that are medically underserved. Therefore, a key driver to reducing health inequities is recruiting a physician workforce that reflects the racial/ethnic composition of the population. Although efforts on increasing UIM applicants and matriculants into medical school and more recently, graduate medical education (GME) programs have shown some small successes, there has still not been significant change in decades. This narrative review examined the historical trends and current landscape of recruitment in GME to better understand the barriers and facilitators that continue to exist and identify successful interventions that might serve as a model to support recruitment of UIM applicants. There is evidence for racial disparities and bias in traditional residency selection metrics (USMLE, Clerkship Evaluations, MSPE, AOA status, etc.) and a need for more holistic recruitment strategies. Along with implementing new holistic recruitment practices, there is a need for ongoing research and collaboration to study inclusive and equitable learning environments and assessments that will eliminate racial disparities in educational outcomes. Additionally, there must be better longitudinal data utilization and transparency across Electronic Residency Application Services (ERAS) and National Resident Matching Program (NRMP) for accountability and tracking of interventions.

Surgical clerkship: Do examination scores correlate with clinical performance?

BACKGROUND

This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship.

METHODS

Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed.

RESULTS

There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R² = 0.027, p < 0.001), oral exam (R² = 0.021, p < 0.001), and weekly quizzes (R² = 0.053, p = 0.092).

CONCLUSIONS

Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.

Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

BACKGROUND

The Medical Student Performance Evaluation (MSPE) provides important information to residency programs. Despite recent recommendations for standardization, it is not clear how much variation exists in MSPE content among schools.

OBJECTIVES

We describe the current section content of the MSPE in US allopathic medical schools, with a particular focus on variations in the presentation of student performance.

METHODS

A representative MSPE was obtained from 95.3% (143 of 150) of allopathic US medical schools through residency applications to the Zucker School of Medicine at Hofstra/Northwell in select programs for the 2019-2020 academic year. A manual data abstraction tool was piloted in 2018-2019. After training, it was used to code all portions of the MSPE in this study. The results were analyzed, and descriptive statistics were reported.

RESULTS

In preclinical years, 30.8% of MSPEs reported data regarding performance of students beyond achieving "passes" in a pass/fail curriculum. Only half referenced performance in the fourth year including electives, acting internships, or both. About two-thirds of schools included an overall descriptor of comparative performance in the final paragraph. Among these schools, a majority provided adjectives such as "outstanding/excellent/very good/good," while one-quarter reported numerical data categories. Regarding clerkship grades, there were numerous nomenclature systems used.

CONCLUSIONS

This analysis demonstrates the existence of extreme variability in the content of MSPEs submitted by US allopathic medical schools in the 2019-2020 cycle, including the components and nomenclature of grades and descriptors of comparative performance, display of data, and inclusion of data across all years of the medical education program.

Systems-Level Reforms to the US Resident Selection Process: A Scoping Review

BACKGROUND

Calls to reform the US resident selection process are growing, given increasing competition and inefficiencies of the current system. Though numerous reforms have been proposed, they have not been comprehensively cataloged.

OBJECTIVE

This scoping review was conducted to characterize and categorize literature proposing systems-level reforms to the resident selection process.

METHODS

Following Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guidelines, searches of Embase, MEDLINE, Scopus, and Web of Science databases were performed for references published from January 2005 to February 2020. Articles were included if they proposed reforms that were applicable or generalizable to all applicants, medical schools, or residency programs. An inductive approach to qualitative content analysis was used to generate codes and higher-order categories.

RESULTS

Of 10 407 unique references screened, 116 met our inclusion criteria. Qualitative analysis generated 34 codes that were grouped into 14 categories according to the broad stages of resident selection: application submission, application review, interviews, and the Match. The most commonly proposed reforms were implementation of an application cap (n = 28), creation of a standardized program database (n = 21), utilization of standardized letters of evaluation (n = 20), and pre-interview screening (n = 13).

CONCLUSIONS

This scoping review collated and categorized proposed reforms to the resident selection process, developing a common language and framework to facilitate national conversations and change.

Medical School to Residency: How Can We Trust the Process?

Background

To say that the transition from undergraduate medical education (UME) to graduate medical education (GME) is under scrutiny would be an understatement. Findings from a panel discussion at the 2018 Association of American Medical Colleges Annual meeting entitled, “Pass-Fail in Medical School and the Residency Application Process and Graduate Medical Education Transition” addressed what and when information should be shared with residency programs, and how and when that information should be shared.

Materials and Methods

Over 250 participants representing UME and GME (e.g. leadership, faculty, medical students) completed worksheets addressing these questions. During report-back times, verbal comments were transcribed in real time, and written comments on worksheets were later transcribed. All comments were anonymous. Thematic analysis was conducted manually by the research team to analyze the worksheet responses and report back comments.

Results

Themes based on suggestions of what information should be shared included the following: 1) developmental/assessment benchmarks such as demonstrating the ability/competencies to do clinical work; 2) performance on examinations; 3) grades and class ranking; 4) 360 evaluations; 5) narrative evaluations; 6) failures/remediation/gaps in training; 7) professionalism lapses; 8) characteristics of students such as resiliency/reliability; and 9) service/leadership/participation. In terms of how this information should be shared, the participants suggested enhancements to the current process of transmitting documents rather than alternative methods (e.g., video, telephonic, face-to-face discussions) and information sharing at both the time of the match and again near/at graduation to include information about post-match rotations.

Discussion

Considerations to address concerns with the transition from medical school to residency include further enhancements to the Medical Student Performance Evaluation, viewing departmental letters as ones of evaluation and not recommendation, a more meaningful educational handoff, and limits on the number of residency applications allowed for each student. The current medical education environment is ready for meaningful change in the UME to GME transition.

Objective Resident Characteristics Associated with Performance on the Ophthalmic Knowledge Assessment Program Examination

Background To determine objective resident characteristics that correlate with Ophthalmic Knowledge Assessment Program (OKAP) performance, as well as to correlate OKAP performance with Accreditation Council for Graduate Medical Education (ACGME) milestone assessments, written qualifying examination (WQE) scores, and oral board pass rates.

Methods Review of administrative records at an ACGME-accredited ophthalmology residency training program at an urban, tertiary academic medical center.

Results The study included data from a total of 50 resident physicians who completed training from 2012 to 2018. Mean (standard deviation) OKAP percentile performance was 60.90 (27.51), 60.46 (28.12), and 60.55 (27.43) for Years 1, 2, and 3 examinations, respectively. There were no statistically significant differences based on sex, marital status, having children, MD/PhD degree, other additional degree, number of publications, number of first author publications, or grades on medical school medicine and surgery rotations. OKAP percentile scores were significantly associated with United States Medical Licensing Examination (USMLE) Step 1 scores (linear regression coefficient 0.88 [0.54–1.18], p = 0.008). Finally, continuous OKAP scores were significantly correlated with WQE (r s = 0.292, p = 0.049) and oral board (r s = 0.49, p = 0.001) scores.

Conclusion Higher OKAP performance is correlated with passage of both WQE and oral board examinations during the first attempt. USMLE Step 1 score is the preresidency academic factor with the strongest association with success on the OKAP examination. Programs can utilize this information to identify those who may benefit from additional OKAP, WQE, and oral board preparation assistance.

Determining Grades in the Internal Medicine Clerkship: Results of a National Survey of Clerkship Directors

PURPOSE

Trust in and comparability of assessments are essential in clerkships in undergraduate medical education for many reasons, including ensuring competency in clinical skills and application of knowledge important for the transition to residency and throughout students' careers. The authors examined how assessments are used to determine internal medicine (IM) core clerkship grades across U.S. medical schools.

METHODS

A multisection web-based survey of core IM clerkship directors at 134 U.S. medical schools with membership in the Clerkship Directors in Internal Medicine was conducted in October through November 2018. The survey included a section on assessment practices to characterize current grading scales used, who determines students' final clerkship grades, the nature/type of summative assessments, and how assessments are weighted. Respondents were asked about perceptions of the influence of the National Board of Medical Examiners (NBME) Medicine Subject Examination (MSE) on students' priorities during the clerkship.

RESULTS

The response rate was 82.1% (110/134). There was considerable variability in the summative assessments and their weighting in determining final grades. The NBME MSE (91.8%), clinical performance (90.9%), professionalism (70.9%), and written notes (60.0%) were the most commonly used assessments. Clinical performance assessments and the NBME MSE accounted for the largest percentage of the total grade (on average 52.8% and 23.5%, respectively). Eighty-seven percent of respondents were concerned that students' focus on the NBME MSE performance detracted from patient care learning.

CONCLUSIONS

There was considerable variability in what IM clerkships assessed and how those assessments were translated into grades. The NBME MSE was a major contributor to the final grade despite concerns about the impact on patient care learning. These findings underscore the difficulty in comparing learners across institutions and serve to advance discussions for how to improve accuracy and comparability of grading in the clinical environment.

Are We Measuring What Matters? How Student and Clerkship Characteristics Influence Clinical Grading

PURPOSE

Given the growing emphasis placed on clerkship performance for residency selection, clinical evaluation and its grading implications are critically important; therefore, the authors conducted this study to determine which evaluation components best predict a clinical honors recommendation across 3 core clerkships.

METHOD

Student evaluation data were collected during academic years 2015-2017 from the third-year internal medicine (IM), pediatrics, and surgery clerkships at the University of Alabama at Birmingham School of Medicine. The authors used factor analysis to examine 12 evaluation components (12 items), and they applied multilevel logistic regression to correlate evaluation components with a clinical honors recommendation.

RESULTS

Of 3,947 completed evaluations, 1,508 (38%) recommended clinical honors. The top item that predicted a clinical honors recommendation was clinical reasoning skills for IM (odds ratio [OR] 2.8; 95% confidence interval [CI], 1.9 to 4.2; P < .001), presentation skills for surgery (OR 2.6; 95% CI, 1.6 to 4.2; P < .001), and knowledge application for pediatrics (OR 4.8; 95% CI, 2.8 to 8.2; P < .001). Students who spent more time with their evaluators were more likely to receive clinical honors (P < .001), and residents were more likely than faculty to recommend clinical honors (P < .001). Of the top 5 evaluation items associated with clinical honors, 4 composed a single factor for all clerkships: clinical reasoning, knowledge application, record keeping, and presentation skills.

CONCLUSIONS

The 4 characteristics that best predicted a clinical honors recommendation in all disciplines (clinical reasoning, knowledge application, record keeping, and presentation skills) correspond with traditional definitions of clinical competence. Structural components, such as contact time with evaluators, also correlated with a clinical honors recommendation. These findings provide empiric insight into the determination of clinical honors and the need for heightened attention to structural components of clerkships and increased scrutiny of evaluation rubrics.

Medical student perceptions of assessment systems, subjectivity, and variability on introductory dermatology clerkships

Background

Elective introductory clerkships in dermatology serve a critical function in providing formative experiences to medical students interested in the field. Although dermatology clerkships play a pivotal role in students’ career choices and residency preparation, the assessment systems used to evaluate students on these clerkships are widely different and likely affect student experiences.

Objective

This study aimed to explore the relationship between dermatology clerkship assessment systems and student experiences through interviews with students about their clerkship reflections and perceptions of assessment.

Methods

The authors contacted clerkship directors via the Association of Professors of Dermatology mailing list and invited them to provide a description of the assessment system at their institution. The authors, via contacted clerkship directors, then invited students who had completed an introductory dermatology clerkship in between 2018 and 2019 to provide a description of the assessment system at their institution and to participate in a qualitative interview about their experiences with assessment systems. The authors then iteratively synthesized interview transcripts using phenomenological analysis, in which a templated approach was used to achieve comprehensive thematic categorization.

Results

Prior to clerkship onset, students expressed a limited understanding of their clinical role and the assessment system. During the clerkship, students endorsed variable expectations across preceptors, limited feedback experiences, and pressures to perform for evaluators. After their clerkship, students continued to perceive assessment systems as nontransparent, subjective, and preordained.

Conclusion

Medical students perceived assessment systems on introductory dermatology clerkships to be unclear and arbitrary. Encouragingly, students also viewed these challenges in assessment as malleable, identifying several opportunities for educational reform in dermatology clerkships.

Prioritizing the Interview in Selecting Resident Applicants: Behavioral Interviews to Determine Goodness of Fit

From our initial screening of applications, we assess that the 10% to 15% of applicants whom we will interview are all academically qualified to complete our residency training program. This initial screening to select applicants to interview includes a personality assessment provided by the personal statement, Dean's letter, and letters of recommendation that, taken together, begin our evaluation of the applicant's cultural fit for our program. While the numerical scoring ranks applicants preinterview, the final ranking into best fit categories is determined solely on the interview day at a consensus conference by faculty and residents. We analyzed data of 819 applicants from 2005 to 2017. Most candidates were US medical graduates (62.5%) with 23.7% international medical graduates, 11.7% Doctors of Osteopathic Medicine (DO), and 2.1% Caribbean medical graduates. Given that personality assessment began with application review, there was excellent correlation between the preinterview composite score and the final categorical ranking in all 4 categories. For most comparisons, higher scores and categorical rankings were associated with applicants subsequently working in academia versus private practice. We found no problem in using our 3-step process employing virtual interviews during the COVID pandemic.

Beyond Medical Knowledge and Patient Care: A Program Director's Perspective for the Role of General Surgery Subinternships

OBJECTIVE

Established primary goals of general surgery subinternships are improvement in patient care and medical knowledge. However, the secondary gains such as obtaining recommendation letters and forming relationships are apparent but poorly defined. We sought the opinion of general surgery program directors (PDs) on the secondary purposes of subinternships. Our aim is to aide mentors and students to optimize the subinternship experience relative to the residency application process.

DESIGN

A survey was administered in July 2019. This questionnaire consisted of 11 items and was broken down into 3 sections: demographics, PD perspective on the secondary goals of a general surgery subinternship, and the role of general surgery subinternships on the students' residency application.

SETTING

An online, national survey through the Association for Program Directors listserv.

PARTICIPANTS

United States general surgery PDs affiliated with the Association for Program Directors listserv.

RESULTS

Sixty-one PDs completed the survey from 42 (69%) academic and 14 (23%) community programs. The majority of PDs (n = 33, 54%) reported that assessment of a subintern's suitability for their residency was the most important secondary purpose. Furthermore, PDs (n = 24, 39%) valued a letter of recommendation from faculty the student worked with during a subinternship as the most important criteria in the interview selection process. Away rotations were perceived as of equal value to subinternships completed at the student's home institution. Overall, PD opinions were similar at academic and community programs.

CONCLUSIONS

Our study suggests subinternships significantly impact a student's application to general surgery residency, clarifying a secondary role for these rotations. Subinternships are important for PDs to assess a student for ranking at their program. All students should pursue a letter of recommendation from subinternship faculty, when possible, as they can heavily influence the interview selection process. Away rotations should only be recommended for those students who need to strengthen their application.

Best Practices in Letters of Recommendation for General Surgery Residency: Results of Expert Stakeholder Focus Groups

OBJECTIVE

Performance inflation is rampant in applications to general surgery residency. The medical student performance evaluation, transcript, and letters of recommendation (LOR) have all been shown significantly biased in the applicants' favor. This study sought to determine best practices for LOR to improve transparency and alignment of applicant and program characteristics.

DESIGN

Two 1-hour focus groups were conducted using semi-structured interviews. Participants were asked to discuss the value and role of LOR characteristics, including standardized LOR, and provide recommendations for best practices. The transcribed discussions were coded by two educators using grounded theory and an inductive approach utilizing NVivo 12. Codes were then reviewed and revised to achieve consensus and recommendations.

SETTING

Focus groups were held during the annual Surgical Education Week meeting in April 2019.

PARTICIPANTS

General Surgery Program Directors from 10 institutions and Surgery Clerkship Directors from 11 other medical schools participated, with each group meeting independently from the other.

RESULTS

Individually, 18 codes were identified by the authors, with consensus agreed on ten. These were grouped into 4 themes: author factors, letter content, bias, and standardized letters. Overall, a checkbox and short-answer standardized LOR was not recommended, favoring a template of items to include and exclude. Ideal letter writers were felt to be surgeons who best know the applicant, and the Chair's letter, when they have no working knowledge of the applicant, was perceived to add little value. Use of specific examples to demonstrate applicant characteristics were favored, and descriptors for coded language should be included to aid in interpretation.

CONCLUSION

The focus groups identified best practices to guide writing LOR in support of applicants to general surgery residency. A template of content is provided to improve the efficiency, transparency, and accuracy of these letters for the benefit of students, medical schools, and residency programs.

Resident Selection in the Wake of United States Medical Licensing Examination Step 1 Transition to Pass/Fail Scoring

INTRODUCTION

The numeric score for the United States Medical Licensing Examination Step 1 is one of the only universal, objective, scaled criteria for comparing the many students who apply to orthopaedic surgery residency. However, on February 12, 2020, it was announced that Step 1 would be transitioning to pass/fail scoring. The purpose of this study was to (1) determine the most important factors used for interview and resident selection after this change and (2) to assess how these factors have changed compared with a previous report on resident selection.

METHODS

A survey was distributed to the program directors (PDs) of all 179 orthopaedic surgery programs accredited by the Accreditation Council for Graduate Medical Education. Questions focused on current resident selection practices and the impact of the Step 1 score transition on expected future practices.

RESULTS

A total of 78 PDs (44%) responded to the survey. Over half of PDs (59%) responded that United States Medical Licensing Examination Step 2 clinical knowledge (CK) score is the factor that will increase most in importance after Step 1 transitions to pass/fail, and 90% will encourage applicants to include their Step 2 CK score on their applications. The factors rated most important in resident selection from zero to 10 were subinternship performance (9.05), various aspects of interview performance (7.49 to 9.01), rank in medical school (7.95), letters of recommendation (7.90), and Step 2 CK score (7.27). Compared with a 2002 report, performance on manual skills testing, subinternship performance, published research, letters of recommendations, and telephone call on applicants' behalf showed notable increases in importance.

DISCUSSION

As Step 2 CK is expected to become more important in the residency application process, current applicant stress on Step 1 scores may simply move to Step 2 CK scores. Performance on subinternships will remain a critical aspect of residency application, as it was viewed as the most important resident selection factor and has grown in importance compared with a previous report.

Evaluating Urology Residency Applications: What Matters Most and What Comes Next?

PURPOSE OF REVIEW

In light of the announcement that the United States Medical Licensing Examination Step 1 exam will transition to pass/fail reporting, we reviewed recent literature on evaluating residency applicants with a focus on identifying objective measurements of applicant potential.

RECENT FINDINGS

References from attending urologists, Step 1 scores, overall academic performance, and research publications are among the most important criteria used to assess applicants. There has been a substantial increase in the average number of applications submitted per applicant, with both applicants and residency directors indicating support for a cap on the number of applications that may be submitted. Additionally, there are increasing efforts to promote diversity with the goal of improving care and representation in urology. Despite progress in standardizing interview protocols, inappropriate questioning remains an issue. Opportunities to improve residency application include promoting diversity, enforcing prohibitions of illegal practices, limiting application numbers, and finding more transparent and equitable screening measures to replace Step 1.

The Current Status of Medical School Clerkship Grades in Residency Applicants

BACKGROUND

Two criteria that have been investigated for evaluating orthopedic surgery residency candidates are achieving an "honors" grade during a surgery clerkship and the total number of honors grades received in all clerkships. Unfortunately, the rate of honors grades given and the criteria for earning an honors grade differ between medical schools, making comparison of applicants from different medical schools difficult.

OBJECTIVE

We measured the rate of honors grades in clerkships at different medical schools in the United States to examine the utility of clerkship grades in evaluating orthopedic surgery residency applicants.

METHODS

Adequate data via the Electronic Residency Application Service were available for 86 of 142 Association of American Medical Colleges medical schools from the 2017 Match cycle. Descriptive statistics and Wilcoxon rank sum tests were performed to identify differences in grade distributions within each clerkship and in school ranking for research by U.S. News & World Report.

RESULTS

For the surgery clerkship, the median rate of honors grades given was 32.5% (range 5%-67%). There was a high rate of interinstitutional variability in all clerkships. We were unable to demonstrate a statistically significant relationship between research ranking and percentage honors grades given for individual clerkships.

CONCLUSIONS

A standardized method for grading medical students during clinical clerkships does not exist, resulting in a high degree of interinstitutional variability. Surgery clerkship grades are an unreliable measure for comparing orthopedic surgery residency applicants from different medical schools. Standardized measures of applicant evaluation might be helpful in the future.