Racial bias in using USMLE step 1 scores to grant internal medicine residency interviews

Medical student non-modifiable risk factors and USMLE Step 1 exam score

For diversity to exist in the medical graduate workforce, students from all backgrounds should have equitable opportunities of employment. Specialties have utilized a minimal threshold for USMLE Step 1 score when screening applicants for residency interviews. The OHSU SOM class of 2021 completed a 14-question voluntary survey on their Step 1 score and the following non-modifiable risk factors: Adverse Childhood Experience score (ACEs), sex, gender, Underrepresented in Medicine status (URiM), family income during adolescence, highest degree held by a guardian, discrimination experience during medical school, federal/state assistance use, and rural versus urban primary home. Descriptive statistics and unadjusted risk ratios were applied to study the relation between Step 1 score and non-modifiable risk factors as well as certain non-modifiable risk factors and ACEs ≥ 3. The mean Step 1 score was 230 (213, 247). Of the students, 28.2% identified ACEs ≥ 3, 13.6% were considered URiM, and 65.4% were female. URiM were 2.34 (1.30, 4.23),females were 2.77 (1.06-7.29), and those who experienced discrimination in medical school were 4.25 (1.85, 9.77) times more likely to have ACEs ≥ 3. Students who had ACEs ≥ 3 were 3.58 (1.75, 7.29) times less likely to meet a minimal threshold for residency interviews of 220. These are the first results to demonstrate a relationship between Step 1 score and ACEs. Those who identified as URiM, females, and those who experienced discrimination in medical school were at a higher risk of ACEs of ≥ 3. Step 1 transitioned to pass/fail in January 2022. However, the first application cycle that residencies will see pass/fail scoring is 2023-2024, and fellowships will continue to see scored Step 1 until, at the earliest, the 2026-2027 application cycle. These data contribute to a foundation of research that could apply to Step 2CK testing scores, and help to inform decisions about the diversity and equity of the residency interview process.

Does Masked Interviewing Encourage Holistic Review in Residency Selection? A Mixed-Methods Study

Problem: Medical educators increasingly champion holistic review. However, in U.S. residency selection, holistic review has been difficult to implement, hindered by a reliance on standardized academic criteria such as board scores. Masking faculty interviewers to applicants' academic files is a potential means of promoting holistic residency selection by increasing the interview's ability to make a discrete contribution to evaluation. However, little research has directly analyzed the effects of masking on how residency selection committees evaluate applicants. This mixed-methods study examined how masking interviews altered residency selection in an anesthesiology program at a large U.S. academic medical center. Intervention: During the 2019-2020 residency selection season in the University of Pennsylvania's Department of Anesthesiology & Critical Care, we masked interviewers to the major academic components of candidates' application files (board scores, transcripts, letters) on approximately half of interview days. The intent of the masking intervention was to mitigate the tendency of interviewers to form predispositions about candidates based on standardized academic criteria and thereby allow the interview to make a more independent contribution to candidate evaluation. Context: Our examination of the masking intervention used a concurrent, partially mixed, equal-status mixed-methods design guided by a pragmatist approach. We audio-recorded selection committee meetings and qualitatively analyzed them to explore how masking affected the process of candidate evaluation. We also collected independent candidate ratings from interviewers and consensus committee ratings and statistically compared ratings of candidates interviewed on masked days to ratings from conventional days. Impact: In conventional committee meetings, interviewers focused on how to reconcile academic metrics and interviews, and their evaluations of interviews were framed according to predispositions about candidates formed through perusal of application files. In masked meetings, members instead spent considerable effort evaluating candidates' "fit" and whether they came off as tactful. Masked interviewers gave halting opinions of candidates and sometimes pushed for committee leaders to reveal academic information, leading to masking breaches. Higher USMLE Step 1 score and higher medical school ranking were statistically associated with more favorable consensus rating. We found no significant differences in rating outcomes between masked and conventional interview days. Lessons learned: Elimination of academic metrics during the residency interview phase does not straightforwardly promote holistic review. While critical reflection among medical educators about the fairness and utility of such metrics has been productive, research and intervention should focus on the more proximate topic of how programs apply academic and other criteria to evaluate applicants.

Effect of United States Medical Licensing Examination Score Cutoffs on Recruitment of Underrepresented Applicants in the Urology Match

OBJECTIVE

To determine how the use of United States Medical Licensing Examination (USMLE) score cutoffs during the screening process of the Urology Residency Match Program may affect recruitment of applicants who are underrepresented in medicine (URM).

MATERIALS AND METHODS

Deidentified data from the Association of American Medical Colleges' (AAMC) Electronic Residency Application Service (ERAS) system was reviewed, representing all applicants to our institution's urology residency program from 2018 to 2022. We analyzed self-reported demographic variables including race/ethnicity, age, sex/gender, as well as USMLE Step 1 and Step 2 scores. Chi-square tests and ANOVA were used to determine the association between race/ethnicity and other sociodemographic factors and academic metrics. Applicants were stratified according to USMLE Step 1 cutoff scores and the distribution of applicants by race/ethnicity was assessed using a Gaussian nonlinear regression fit.

RESULTS

A total of 1258 applicants submitted applications to our program during the 5-year period, including 872 males (69.3%) and 386 females (30.7%). Most applicants were White (43.5%), followed by Asian (28.3%), Hispanic/Latino (11.7%), and Black (7.0%). There was an association between race/ethnicity and USMLE scores. Median USMLE Step 1 scores for White, Asian, Hispanic/Latino, and Black applicants were 242, 242, 237, and 232, respectively (P < .001). As cutoff score increases, percentage of URM applicants decreases.

CONCLUSION

The use of cutoffs based on USMLE scores disproportionately affects URM applicants. Transitioning from numeric scores to pass/fail may enhance holistic review processes and increase the representation of URM applicants offered interviews at urology residency programs.

Selection and Recruitment Strategies among Competitive Pediatric Training Programs and the Impact of Diversity

OBJECTIVE

Aspects of the written application, interview and ranking may negatively impact recruitment of underrepresented in medicine (URiM) applicants. Our objectives were to explore knowledge, attitudes, and perceptions of pediatric faculty who assess potential trainees and how diversity impacts these assessments.

METHODS

We performed qualitative interviews of 20 geographically diverse faculty at large pediatric residencies and fellowships. We analyzed data using the constant comparative method to develop themes.

RESULTS

Four main themes emerged.

CONCLUSIONS

We describe ways in which bias infiltrates recruitment and strategies to promote diversity. Many strategies are variably implemented and the impact on workforce diversity in pediatric training programs remains unknown.

Trends in the Main Residency Match From 2007 to 2020

Background The United States Medical Licensing Exam (USMLE) Step 1 was recently changed from a numerically scored grading system to a pass/fail grading system. Until late 2024, there will be no formal studies about the impact that the grading change will have on the match process. To thoroughly assess the impact that this change will have on the overall match process, it is important to look at what the trends in applicants' objective measures have been in the years before the change. We aim to systematically evaluate the rates of change and mean trend of objective metrics found in residency applications in the main residency match.  Methods Objective medical student data of matched and unmatched applicants were queried from the National Matching Program's Charting Outcomes in the Match Reports for the 2007 to 2020 application cycles. Data were used to create linear regression analyses and statistical tests were performed to evaluate trends over time. Results For matched applicants, there were statistically significant positive trends for the mean number of contiguous ranks (m=0.33, p<0.01), having another non-doctoral graduate degree (m=0.67, p<0.01), membership to Alpha Omega Alpha (AOA) honor society (m=0.22, p<0.01), mean USMLE Step 1 score (m=1.01, p<0.01), mean USMLE Step 2 score (m=1.68, p<0.01), mean number of research experiences (m=0.12, p<0.01), and mean number of abstracts, presentations, and publications (m=0.34, p<0.01). Additionally, there was a statistically significant negative trend for the percentage who graduated from a top 40 National Institutes of Health-funded medical school (m=-0.41, p<0.01). For unmatched applicants, there were statistically significant positive trends for having another non-doctoral graduate degree (m=0.83, p<0.01), mean USMLE Step 1 score (m=1.26, p<0.01), mean USMLE Step 2 score (m=2.27, p<0.01), mean number of research experiences (m=0.13, p<0.01), and mean number of abstracts, presentations, and publications (m=0.33, p<0.01). Conclusion Our study shows that there have been statistically significant increases in almost all objective measures in the residency application. Recent changes to the abstracts, presentations, and publications on the Step 1 scoring system will force almost all residency programs to overhaul their application process and potentially increase reliance on Step 2, research, and other nonobjective factors. For students early in their medical education, emphasis on Step 2 and research will yield increased chances of matching into residency in the future.

Interrogating Consequential Validity Evidence in NAPLEX Studies Involving the Use of Demographic Variables

OBJECTIVE

to review the purpose, methods, and discussion of student demographics related to North American Pharmacist Licensure Examination (NAPLEX) performance; demographic characteristics, significant findings, and related text were extracted from each reviewed article as evidence of consequential validity.

FINDINGS

Nine articles met the inclusion criteria. Prior degree attainment (n = 8, 88.9%), age (n = 6, 66.7%), race/ethnicity (n = 6, 66.7%), and sex (n = 5, 55.6%) were the most common demographic variables included. One study found that prior degree attainment was negatively related to NAPLEX performance and no studies using this variable discussed their findings. Three studies found significant relationships between age and NAPLEX performance, attributing differences to older students for "unique psychosocial challenges and competing responsibilities" "greater maturity" and being more "professional." Measures of race/ethnicity differed in every study, with 3 reporting significant findings and 2 discussing their findings. Studies referenced literature suggesting that "minority groups" tended to score lower on standardized examinations while others referenced literature that suggests "standardized testing may not be predictive of the performance of minority students." Sex was not related to NAPLEX performance nor was it discussed in any studies.

SUMMARY

Consequential validity evidence is a critical yet underreported aspect of NAPLEX evaluation in pharmacy education. How demographic variables are selected, utilized, and discussed warrants further exploration and consideration by educators, scholars, and practitioners as these decisions can have important sociocultural and political implications.

Implementation of a Holistic Review Process of US Allopathic Medical Students Eliminates Non-Comparable Metrics and Bias in General Surgery Residency Interview Invitations

OBJECTIVE

Use of traditional scoring metrics for residency recruitment creates racial and gender bias. In addition, widespread use of pass/fail grading has led to noncomparable data. To adjust to these challenges, we developed a holistic review (HR) rubric for scoring residency applicants for interview selection.

DESIGN

Single-center observational study comparing the proportion of underrepresented in medicine (URM) students and their United States Medical Licensing Exam (USMLE) scores who were invited for interview before (2015-2020) and after (2022) implementation of a holistic review process.

SETTING

General surgery residency program at a tertiary academic center.

PARTICIPANTS

US allopathic medical students applying for general surgery residency.

RESULTS

After initial screening, a total of 1514 allopathic applicants were narrowed down to 586 (38.7%) for HR. A total of 52% were female and 17% identified as URM. Based on HR score, 20% (118/586) of applicants were invited for an interview. The median HR score was 11 (range 4-19). There was a fourfold higher coefficient of variation of HR scores (22.3; 95% CI 21.0-23.7) compared to USMLE scores (5.1; 95% Cl 4.8-5.3), resulting in greater spread and distinction among applicants. There were no significant differences in HR scores between genders (p = 0.60) or URM vs non-URM (p = 0.08). There were no significant differences in Step 1 (p = 0.60) and 2CK (p = 0.30) scores between those who were invited to interview or not. On multivariable analysis, USMLE scores (OR 1.01; 95% CI 0.98-1.03), URM status (OR 1.71 95% CI 0.98-2.92), and gender (OR 0.94, 95% CI 0.60-1.45) did not predict interview selection (all p > 0.05). There was a meaningful increase in the percentage of URM interviewed after HR implementation (12.9% vs 23.1%, p = 0.016).

CONCLUSION

The holistic review process is feasible and eliminates the use of noncomparable metrics for surgical applicant interview invitations and increases the percentage of URM applicants invited to interview.

Strategies to Counteract Impact of Harmful Bias in Selection of Medical Residents

Human biases impact medical care in ways that perpetuate health disparities. Research has demonstrated that biases negatively affect patient outcomes and stifle diversity across the physician workforce, further compounding health disparities by worsening patient-physician concordance. Taken as one, the application, interview, recruitment, and selection processes employed by residency programs has been one of the critical junctures where bias has exacerbated inequities among future physicians. In this article, the authors define diversity and bias, review the history of bias in residency programs' processes for selecting residents, explore the impact of this history on workforce demographics, and discuss ways to optimize and work toward equity in the practices used by residency programs to select residents.

Using a Consensus-Driven Approach to Incorporate Holistic Review Into an Internal Medicine Residency Program

Background

Despite growing interest in incorporating holistic review within residency admissions, implementation by residency programs remains challenging.

Objective

To incorporate holistic review into the internal medicine residency program at the University of Wisconsin and to report initial feasibility and acceptability data.

Methods

During the 2020-2021 application cycle, residency stakeholders performed a consensus-driven process to identify highly valued applicant attributes. We used a holistic review process to identify the presence of these attributes among applicants and updated our rank list algorithm to incorporate these attributes. We modified our interview screening criteria and rank list algorithm to de-emphasize other metrics. We surveyed stakeholders to assess time required for this process and compared our final rank list to what it would have been using our prior system.

Results

The final list of 10 prioritized applicant attributes included extraordinary leadership, community service, and grit, among others. Among 25 matched residents, 8 (32%) were recognized to have exceptional achievement within one of these 10 attributes. Four members of the incoming intern class (16%) would have been in a rank position lower than our historical matched resident cutoff had they not received additional points for these attributes. Faculty reported that holistic review of applications took an additional 3.8 minutes on average. It was felt that current application materials limit the ability to implement a fully holistic review.

Conclusions

The addition of holistic review to our residency admissions process was achieved using a consensus-driven approach and showed favorable feasibility and acceptability data.

Multivariate Modeling of Student Performance on NBME Subject Exams

Aim This study sought to determine whether it was possible to develop statistical models which could be used to accurately correlate student performance on clinical subject exams based on their National Board of Medical Examiner (NBME) self-assessment performance and other variables, described below, as such tools are not currently available.  Methods Students at a large public medical school were provided fee vouchers for NBME self-assessments before clinical subject exams. Multivariate regression models were then developed based on how self-assessment performance correlated to student success on the subsequent subject exam (Medicine, Surgery, Family Medicine, Obstetrics-Gynecology, Pediatrics, and Psychiatry) while controlling for the proximity of the self-assessment to the exam, USMLE Step 1 score, and the academic quarter. Results The variables analyzed satisfied the requirements of linear regression. The correlation strength of individual variables and overall models varied by discipline and outcome (equated percent correct or percentile, Model R² Range: 0.1799-0.4915). All models showed statistical significance on the Omnibus F-test (p<0.001). Conclusion The correlation coefficients demonstrate that these models have weak to moderate predictive value, dependent on the clinical subject, in predicting student performance; however, this varies widely based on the subject exam in question. The next step is to utilize these models to identify struggling students to determine if their use reduces failure rates and to further improve model accuracy by controlling for additional variables.

Objective Residency Applicant Assessment Using a Linear Rank Model

OBJECTIVES

Residency applicant assessment is imperfect, with little objectivity built into the process, which, unfortunately, impacts recruitment diversity. Linear rank modeling (LRM) is an algorithm that standardizes applicant assessment to model expert judgment. Over the last 5 years, we have used LRM to assist with screening and ranking integrated plastic surgery (PRS) residency applicants. This study's primary objective was to determine if LRM scores are predictive of match success and, secondarily, to compare LRM scores between gender and self-identified race categories.

DESIGN

Data was collected on applicant demographics, traditional application metrics, global intuition rank, and match success. LRM scores were calculated for screened and interviewed applicants, and scores were compared by demographic groups. Univariate logistic regression was used to evaluate the association of LRM scores and traditional application metrics with match success.

SETTING

University of Wisconsin, Division of Plastic and Reconstructive Surgery. Academic institution.

PARTICIPANTS

Six hundred seventeen candidates who applied to a single institution over 4 application cycles (2019-2022).

RESULTS

Using area under the curve modeling, LRM score was the most predictive indicator for match success. With every one-point increase in LRM score, there was an 11% and 8.3% increase in the likelihood of screened and interviewed applicant match success (p < 0.001). An algorithm was developed to estimate the probability of match success based on LRM score. No significant differences in LRM scores were appreciated for interviewed applicant gender or self-identified race groups.

CONCLUSIONS

LRM score is the most predictive indicator of match success for PRS applicants and can be used to estimate an applicant's probability of successfully matching into an integrated PRS residency. Furthermore, it provides a holistic evaluation of the applicant that can streamline the application process and improve recruitment diversity. In the future, this model could be applied to assist in the match process for other specialties.

The Upcoming Pass/Fail USMLE Step 1 Score Reporting: An Impact Assessment From Medical School Deans

Background: The US Medical Licensing Examination (USMLE) Step 1 change to pass/fail has been met with mixed reviews, and the impact on medical student education and residency match is unknown. We surveyed medical school student affairs deans regarding their thoughts on the upcoming transition of Step 1 to pass/fail. Methods: A questionnaire was emailed to medical school deans. Deans were asked to rank the importance of the following after the Step 1 reporting change: Step 2 Clinical Knowledge (Step 2 CK), clerkship grades, letters of recommendation, personal statement, medical school reputation, class rank, Medical Student Performance Evaluation, and research. They were asked how the score change will affect curriculum, learning, diversity, and student mental health. Deans were asked to select 5 specialties they thought would be most affected. Results: Regarding perceived importance of residency applications following the scoring change, the most frequent number 1 choice was Step 2 CK. The majority of deans (93.5%, n = 43) felt that the change to pass/fail would benefit medical student education/learning environment; however, most (68.2%, n = 30) did not believe their school curriculum would change. Students applying to dermatology, neurosurgery, orthopedic surgery, ENT, and plastic surgery were felt to be most affected by the scoring change; 58.7% (n = 27) felt it would not adequately address future diversity. Conclusion: The majority of deans feel the USMLE Step 1 change to pass/fail would benefit medical student education. Deans feel that students applying to traditionally more competitive specialties (ie, programs with fewer overall residency positions available) will be most affected.

Perceptions of the Emergency Medicine Resident Selection Process by Program Directors Following the Transition to a Pass/Fail USMLE Step 1

Background

Beginning January 26th, 2022, the National Board of Medical Examiners transitioned scoring of the United States Medical Licensing Examination (USMLE) Step 1 from a 3-digit score to pass/fail. In the past, the Step 1 score has been weighted heavily by program directors (PDs) as one of the most important metrics when assessing medical student's competitiveness.

Objective

The objective of this study was to evaluate the perceptions of emergency medicine (EM) PDs on the transition to a pass/fail USMLE Step 1 exam, and to elicit the opinions of EM PDs on the USMLE examinations' ability to predict resident performance.

Methods

A survey consisting of ranking and multiple-choice questions was sent to EM PDs. The multiple-choice questions were asked to determine EM PDs level of confidence in the ability of Step 1 and Step 2 Clinical Knowledge (CK) to predict a student's ability to succeed in residency. The ranking questions focused on assessing each program's current resident selection practices in comparison to expected selection criteria changes following a transition to pass/fail Step 1. R studio and MATLAB were used for statistical analysis, and a P value <0.05 was considered significant.

Results

The survey was completed by 57 (20.21%) EM PDs. When asked if Step 1 and Step 2 CK are accurate predictors of a resident's ability to perform clinically within EM, only 10.5% of PDs answered 'yes' to Step 1 being predictive, compared to 31.6% for Step 2 CK. Regarding selection criteria, the top quartile of attributes (standardized letters of evaluation [1st], away rotations [2nd], clerkship grades [3rd] and Step 2 CK score [4th]) remained the same following the transition.

Conclusion

Our results indicate that the top quartile of attributes might remain the same, despite most PDs agreeing that Step 2 CK is a better predictor of a resident's performance.

The Effect of Pass/Fail Exam Grading on Exam Performance in a Pediatric Clerkship

OBJECTIVE

Clerkship grades are a component of determining a residency candidate's competitiveness. In 2017, the University of Minnesota Medical School's pediatric clerkship transitioned its standardized multiple-choice exam, the Aquifer Pediatrics Examination, to pass/fail with eligibility for honors being determined by clinical performance, not exam performance. We assessed the effect this change had on Aquifer exam performance and evaluated for correlation between Aquifer exam performance and clinical evaluation scores in order to gather insight into the validity of each type of assessment with respect to one another.

METHODS

We analyzed de-identified data from 750 medical students between the academic years of 2016 to 2017 and 2019 to 2020. Individual Aquifer exam scores were compared to individual clinical performance scores. Differences in exam performance before and after the transition to pass/fail were investigated with a two-sample t-test and Cohen's d for effect size.

RESULTS

No correlation was found between Aquifer exam scores and clinical performance scores. The mean Aquifer exam score prior to the transition to pass/fail was 80.02 ± 7.51 while the mean after the exam was made pass/fail was 77.8 ± 7.42. This difference was statistically significant (P < .001) with a Cohen's d (effect size) of 0.297.

CONCLUSIONS

A lack of correlation between the Aquifer exam scores and clinical performance scores was found. There was a small yet statistically significant decrease in Aquifer exam scores after the change to pass/fail; it is not clear if this represents a meaningful decrease in learning by students.

Eligible Applicants and Diversity Across Settings in a Regional Family Medicine Residency Network

OBJECTIVES

Residency programs must gather and track data on the diversity of their applicants, interviewees, and matched residents as part of the process of checking for bias in the interview and rank process. As such, the aims of this study were (1) to provide data from a large, regional network of family medicine residencies on who is applying, interviewing, and matching into our programs as a baseline for the family medicine residency community; and (2) to assess potential differences in the gender and racial diversity of the eligible applicants to programs across settings, including in rural and underserved communities.

METHODS

Survey of programs in Alaska, Idaho, Montana, Washington and Wyoming about their applicant pool in the 2020-2021 interview season.

RESULTS

Programs received a median of 100 applications per position, 57 of which were considered eligible. Programs offered 17 interviews per position and 15 of these were completed. Programs in rural and underserved communities did not have fewer eligible applicants per position, nor was there less diversity within that pool of applicants. Most programs are working to increase their program's diversity.

CONCLUSION

On average, the racial and gender diversity of eligible applicants to programs in rural and underserved settings is no different than other programs. What is important in terms of diversity varies across programs, based on community needs and program mission, but having a mechanism to extract and review data and to then be able to assess progress is a place to start.

Black Physicians' Experiences with Anti-Black Racism in Healthcare Systems Explored Through An Attraction-Selection-Attrition Lens

Anti-Black racism is a specific form of racism directed at Black people. In healthcare, there are poignant examples of anti-Black racism in the recruitment, selection, and retention stages of the job cycle. Research shows that anti-Black racism is associated with inequitable work outcomes and the under-representation of Black physicians. However, empirical findings are scattered with no organizing framework to consolidate these findings. To add to the literature, in this paper we present the attraction-selection-attrition (ASA) model (Schneider, 1987) as an organizing framework to discuss Black physicians' experiences with anti-Black racism and discrimination throughout their careers. We draw from previous literature to highlight specific experiences of Black physicians at each stage of the job cycle (i.e., attraction, selection, retention), and we offer considerations on how practitioners can mitigate anti-Black racism throughout the job cycle. In the wake of COVID-19 and highly publicized social justice movements, healthcare systems are seeking ways to increase the recruitment, selection, and retention of Black physicians to ensure health equity. We believe this guide will be valuable to practitioners, leaders, researchers, and program directions seeking to advance diversity, equity, and inclusion of Black physicians in their healthcare systems. We conclude by providing practical implications and directions for future research.

The time is now: addressing implicit bias in obstetrics and gynecology education

Obstetrician-gynecologists can improve the learning environment and patient care by addressing implicit bias. Accumulating evidence demonstrates that racial and gender-based discrimination is woven into medical education, formal curricula, patient-provider-trainee interactions in the clinical workspace, and all aspects of learner assessment. Implicit bias negatively affects learners in every space. Strategies to address implicit bias at the individual, interpersonal, institutional, and structural level to improve the well-being of learners and patients are needed. The authors review an approach to addressing implicit bias in obstetrics and gynecology education, which includes: (1) curricular design using an educational framework of antiracism and social justice theories, (2) bias awareness and management pedagogy throughout the curriculum, (3) elimination of stereotypical patient descriptions from syllabi and examination questions, and (4) critical review of epidemiology and evidence-based medicine for underlying assumptions based on discriminatory practices or structural racism that unintentionally reinforce stereotypes and bias. The movement toward competency-based medical education and holistic evaluations may result in decreased bias in learner assessment. Educators may wish to monitor grades and narratives for bias as a form of continuous educational equity improvement. Given that practicing physicians may have little training in this area, faculty development efforts in bias awareness and mitigation strategies may have significant impact on learner well-being.

Racial and Ethnic Differences in Internal Medicine Residency Assessments

Importance

Previous studies have demonstrated racial and ethnic inequities in medical student assessments, awards, and faculty promotions at academic medical centers. Few data exist about similar racial and ethnic disparities at the level of graduate medical education.

Objective

To examine the association between race and ethnicity and performance assessments among a national cohort of internal medicine residents.

Design, Setting, and Participants

This retrospective cohort study evaluated assessments of performance for 9026 internal medicine residents from the graduating classes of 2016 and 2017 at Accreditation Council of Graduate Medical Education (ACGME)-accredited internal medicine residency programs in the US. Analyses were conducted between July 1, 2020, and June 31, 2022.

Main Outcomes and Measures

The primary outcome was midyear and year-end total ACGME Milestone scores for underrepresented in medicine (URiM [Hispanic only; non-Hispanic American Indian, Alaska Native, or Native Hawaiian/Pacific Islander only; or non-Hispanic Black/African American]) and Asian residents compared with White residents as determined by their Clinical Competency Committees and residency program directors. Differences in scores between Asian and URiM residents compared with White residents were also compared for each of the 6 competency domains as supportive outcomes.

Results

The study cohort included 9026 residents from 305 internal medicine residency programs. Of these residents, 3994 (44.2%) were female, 3258 (36.1%) were Asian, 1216 (13.5%) were URiM, and 4552 (50.4%) were White. In the fully adjusted model, no difference was found in the initial midyear total Milestone scores between URiM and White residents, but there was a difference between Asian and White residents, which favored White residents (mean [SD] difference in scores for Asian residents: -1.27 [0.38]; P < .001). In the second year of training, White residents received increasingly higher scores relative to URiM and Asian residents. These racial disparities peaked in postgraduate year (PGY) 2 (mean [SD] difference in scores for URiM residents, -2.54 [0.38]; P < .001; mean [SD] difference in scores for Asian residents, -1.9 [0.27]; P < .001). By the final year 3 assessment, the gap between White and Asian and URiM residents' scores narrowed, and no racial or ethnic differences were found. Trends in racial and ethnic differences among the 6 competency domains mirrored total Milestone scores, with differences peaking in PGY2 and then decreasing in PGY3 such that parity in assessment was reached in all competency domains by the end of training.

Conclusions and Relevance

In this cohort study, URiM and Asian internal medicine residents received lower ratings on performance assessments than their White peers during the first and second years of training, which may reflect racial bias in assessment. This disparity in assessment may limit opportunities for physicians from minoritized racial and ethnic groups and hinder physician workforce diversity.

Recommendations to improve recruitment and retention of underrepresented in medicine trainees in neonatal-perinatal medicine

Physicians who identify as Black, Latinx, American Indian, Pacific Islander, and certain Asian subgroups represent racial and ethnic populations that are underrepresented in medicine (URM). While the proportion of URM pediatric trainees has remained unchanged, that of Neonatal-Perinatal Medicine (NPM) fellows has decreased. Informed by the medical literature and our lived experiences, we compiled and developed a list of recommendations to support NPM fellowship programs in the recruitment, retention, and promotion of URM trainees. We describe ten recommendations that address 1) creating a culture of inclusivity and psychological safety, 2) the critical appraisal of recruitment practices and climate, and 3) an inclusive and holistic fellowship application process. The first two themes lay the foundation, while the final theme spotlights our recommendations for URM recruitment. Each recommendation is a step towards improvement in recruitment and inclusion at a program.

Exploring Characteristics of Academic General Surgery Residency Applicants: A Group Concept-Mapping Approach

OBJECTIVE

Holistic review, which emphasizes qualitative attributes over objective measures, has been proposed as a method for selecting candidates for surgical residency in order to improve diversity in graduate medical education, and, ultimately, the field of surgery. This study seeks to articulate desirable traits of applicants as a first-step in standardizing the holistic review process.

DESIGN

Using Group Concept Mapping, a web-based mixed-methods participatory research methodology, residency selection committee members were asked to 1) list desirable characteristics of applicants, 2) group these into categories, 3) rate their importance to academic/clinical success on a 5-point Likert scale (1 = not at all important, 5 = extremely important), and 4) rate the degree to which each characteristic is feasible to assess on a 3-point Likert scale (1 = not at all feasible, 3 = very feasible). Grouped characteristics submitted to hierarchical cluster analysis depicted committee's consensus about desirable qualities/criteria for applicants. Bivariate scatter-plots and pattern-matching graphics demonstrated which of these criteria were most important and reliably assessed.

SETTING

A single academic general surgery residency training program in Western Pennsylvania.

PARTICIPANTS

Members of the selection committee for the UPMC General Surgery Residency program who had participated in at least 1 prior cycle of applicant selection.

RESULTS

Desirable characteristics of highly qualified applicants into an academic general surgery residency were clustered into domains of 1) scholarly work and research, 2) grades/formal assessments, 3) program fit, 4) behavioral assets, and 5) aspiration. Behavioral assets, which was felt to be the most important to clinical and academic success were considered to be the least feasible to reliably assess. Within this domain, initiative, being self-motivated, intellectual curiosity, work ethic, communication skills, maturity and self-awareness, and thoughtfulness were viewed as most frequently reliably assessed from the application and interview process.

CONCLUSIONS

High quality applicants possess several behavioral assets that faculty deem are important to academic and clinical success. Adapting validated metrics for assessing these assets, may provide a solution for addressing subjectivity and other challenges scrutinized by critics of holistic review.

Identifying Potential Attrition during the Residency Applicant Screening Process Using a Situational Judgment Test

OBJECTIVE

As USMLE Step 1 moves to pass/fail, residency programs are seeking alternate interview selection processes. Attrition in general surgery is reported as high as 26%. Thus, it is important to ensure that programs are selecting and matching applicants with shared values. Situational judgment tests (SJTs) measure educational and cultural values by posing ambiguous situations and individuals rate the effectiveness of possible reactions. SJTs have previously been shown to identify job applicants with shared values while promoting diversity. Scoring categories are high, moderate, or low values congruence. We sought to explore predictive validity of the SJT relative to program attrition.

DESIGN

Residents who matched into our program between 2018 and 2021 completed the SJT. We tracked attrition.

SETTING

UT Health San Antonio, Texas PARTICIPANTS: Fifty-six categorical general surgery residents RESULTS: Per SJT ratings, the numbers of residents who had high, moderate, and low values congruence were 27, 16, and 13, respectively. Attrition numbers for residents who scored high and moderate congruence were similar, indicating that these ratings were indistinguishable. As such, we combined those 2 categories to create a 2 × 2 matrix and used signal detection theory as a framework for analysis. Overall attrition was 16.1% (9/56). Of the 43 residents who scored high or moderate congruence, 90.7% remained in the program. There was a 9.3% chance of attrition for these residents. Of the 13 residents who scored low congruence, 38.5% attrited. While scoring as low congruence on the SJT does not definitively indicate attrition, it does indicate that attrition is 4.14 times more likely for these residents (chi-square, p = 0.0121).

CONCLUSIONS

One of the most important aspects of residency applicant selection and interviewing is mitigating risk by identifying applicants who carry a high risk of attrition. The SJT significantly identifies at-risk applicants.

Effect of teleconferencing variables on faculty impression of mock residency applicants

Purpose

The objective of this study was to assess how teleconferencing variables influence faculty impressions of mock residency applicants.

Methods

In October 2020, we conducted an online experiment studying five teleconferencing variables: background, lighting, eye contact, internet connectivity, and audio quality. We created interview videos of three mock residency applicants and systematically modified variables in control and intervention conditions. Faculty viewed the videos and rated their immediate impression on a 1-10 scale. The effect of each variable was measured as the mean difference between the intervention and control impression ratings. One-way analysis of variance (ANOVA) was performed to assess whether ratings varied across applicants. Paired-samples Wilcoxon signed-rank tests were conducted to assess the significance of the effect of each variable.

Results

Of 711 faculty members who were emailed a link to the experiment, 97 participated (13.6%). The mean ratings for control videos were 8.1, 7.2, and 7.6 (P < .01). Videos with backlighting, off-center eye contact, choppy internet connectivity, or muffled audio quality had lower ratings when compared with control videos (P < .01). There was no rating difference between home and conference room backgrounds (P = .77). Many faculty participants reported that their immediate impressions were very much or extremely influenced by audio quality (60%), eye contact (57%), and internet connectivity (49%).

Conclusions

Teleconferencing variables may serve as a source of assessment bias during residency interviews. Mock residency applicants received significantly lower ratings when they had off-center eye contact, muffled audio, or choppy internet connectivity, compared to optimal teleconferencing conditions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00053-w.

Racial Bias on the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

Studies on components of residency applications have shown evidence of racial bias. The Standardized Letter of Evaluation (SLOE) is an assessment measure for emergency medicine (EM) residency applications and, as more specialties opt to use SLOEs in place of narrative letters of recommendation, understanding bias on standardized assessments is essential.

OBJECTIVE

To determine whether there is a difference in rankings on the EM SLOE between underrepresented in medicine (UIM) and non-UIM applicants, White and non-White applicants, and to examine whether differences persist after controlling for other characteristics.

METHODS

The sample was drawn from medical students who applied to EM residency at the study institution in 2019. We compared rankings between UIM and non-UIM students and between students of each individual race/ethnicity and White students, after controlling for United States Medical Licensing Examination Step scores, Alpha Omega Alpha status, type of school (US MD, US DO, internation medical graduate), Medical Student Performance Evaluation class percentile, affiliated program vs visiting clerkship SLOE, gender and the interaction of race/ethnicity and gender, and adjusted for students submitting multiple SLOEs, using ordinal regression.

RESULTS

There were 1555 applicants to the study institution in 2019; 1418 (91.2%) had a SLOE and self-identified race/ethnicity. After controlling for applicant characteristics, non-UIM students were significantly more likely to be ranked higher than UIM students on "Rank Against Peers," (OR 1.46, 95% CI 1.03-2.07) and Grade (OR 1.46, 95% CI 1.05-2.04).

CONCLUSIONS

Analysis of EM SLOEs submitted to our institution demonstrates racial bias on this standardized assessment tool, which persists after controlling for other performance predictors.

A Survey of Vascular Surgery Program Directors: Perspectives Following USMLE Step 1 Conversion to Pass/Fail and Virtual Only Interviews

INTRODUCTION

Step 1 of the United States Medical Licensing Examination (USMLE), a historically important factor in the selection of vascular surgery residents and fellows, transitioned to completely pass/fail on January 26, 2022. Due to the COVID-19 pandemic, residency and fellowship interviews were conducted virtually during the 2020-21 and 2021-22 application cycles. Given these significant changes in the evaluation of candidates for residency training, we sought to understand vascular surgery program directors' perspectives regarding the change in Step 1 scoring and use of virtual interviews as well as determine which factors will assume importance when applying to vascular surgery training programs in the future.

METHODS

A 26-item survey questionnaire was created using Qualtrics survey tools with questions regarding attitudes towards the change in Step 1 scoring and virtual interviews, the importance of additional factors utilized by programs in selecting candidates for interviews and ranking applicants for residency/fellowship selection, and programmatic demographics. This was distributed anonymously to all vascular surgery training programs over a two-week period using a comprehensive list of 249 unique email addresses created by cross-referencing the Accreditation Council for Graduate Medical Education (ACGME) list of certified vascular training programs with email addresses from individual program websites identifying current program leadership. Responses were analyzed with descriptive statistics with values listed as average Likert scale weight ± standard deviation (SD) or percentages.

RESULTS

68 of 249 (27.3%) program and associate program directors responded to the survey. 33.9% of respondents strongly disagreed with Step 1 going completely pass/fail. In the absence of a scored USMLE Step 1, letters of recommendation (average Likert scale weight ± SD, 4.43 ± 0.92), dedication to specialty (4.14 ± 1.03), and USMLE Step 2 CK (4.06 ± 0.92) had the highest average scores for deciding which applicants to interview for integrated vascular surgery residency. For determining which candidates to interview for vascular surgery fellowship, letters of recommendation (4.51 ± 0.84), dedication to specialty (4.12 ± 0.90), and research (4.10 ± 0.80) had the highest average scores. For ranking residency candidates, the interviewee's perceived "fit" (4.61 ± 0.55), letters of recommendation (4.53 ± 0.76), and overall interview experience (4.47 ± 0.62) had the highest average scores. Similarly, the factors with the highest average Likert scores for ranking fellowship candidates included the interviewee's perceived "fit" (4.69 ± 0.51), letters of recommendation (4.65 ± 0.52), and overall interview experience (4.51 ± 0.59). The majority (72.2%) of program directors preferred in-person interviews; however, 50% of respondents were at least "somewhat satisfied" with virtual interviews during the 2021-22 application cycle as they could judge applicants' interview skills at least "moderately well." The minority (18.8%) who preferred virtual interviews most commonly noted a "reduction of the financial burden for applicants" as the reason for this preference.

CONCLUSION

Most vascular surgery program and associate program directors were dissatisfied with USMLE Step 1 going pass/fail with most indicating pre-screening applicants using both Step 1 and Step 2 CK during the residency and fellowship selection processes. In the absence of a scored Step 1, the top factors for interviewing and ranking integrated vascular surgery residency and fellowship candidates included letters of recommendation, dedication to specialty, research, USMLE Step 2 CK, the interviewee's perceived "fit," and overall interview experience. Though most program directors preferred face-to-face interviews, they were overall at least "somewhat satisfied" with the virtual format that took place during the 2021-22 cycle.

Diversity and Inclusion in Orthopaedic Surgery from Medical School to Practice: AOA Critical Issues

Orthopaedic surgery has historically been one of the least diverse fields in medicine in the U.S. Despite having been declared a critical issue by the American Orthopaedic Association, a lack of diversity and inclusion across all metrics remains a persistent problem. In order to balance representation within orthopaedic surgery, a multiyear and multilevel approach should be considered across the life cycle of a surgeon. Talented individuals from diverse backgrounds must be identified and given early exposure to orthopaedic surgery. They must be nurtured, mentored, and retained. Representation at the medical student, resident, and faculty level is vital to ensuring diversity across the next generation of surgeons.We cannot alter representation within orthopaedics unless we broaden the candidate selection pool. Medical school classes provide the selection pool for residency, residency provides the candidates for fellowship, fellowships provide the candidates for employment, and employment provides the pool for promotion to leadership positions. Through each progression, there is a loss of underrepresented applicants, which leads to a lack of balanced representation in orthopaedic surgery. With stronger efforts to identify and retain individuals at each phase of a surgeon's career, we hope to mitigate the loss of talented and diverse individuals from this field.We challenge the paradigm of increasing diversity that focuses only on the resident selection level. Instead, efforts must begin at the medical student level. Efforts for early and meaningful exposure to the field through a musculoskeletal curriculum and rotations as well as connection through mentorship and sponsorship are vital for retention. At each ascending level of education, reinvestment in each individual is critical. Exposure, mentorship, retention, and promotion should lead to a more diverse and rich future. To achieve this, deliberate and longitudinal action should be instituted to increase diversity within orthopaedics.

Race, But Not Gender, Is Associated With Admissions Into Orthopaedic Residency Programs

BACKGROUND

Orthopaedic surgery is one of the most competitive but least diverse surgical specialties, with ever-increasing academic achievements (such as test scores) shown by applicants. Prior research shows that white applicants had higher United States Medical Licensing Exam (USMLE) Step 1 and Step 2 Clinical Knowledge scores as well as higher odds of Alpha Omega Alpha status compared with Black, Hispanic, and other applicant groups. Yet, it still remains unknown whether differences in application metrics by race/ethnicity sufficiently explain the underrepresentation of certain racial or ethnic minority groups in orthopaedic residency programs.

QUESTIONS/PURPOSES

In this study, we sought to determine (1) the relative weight of academic variables for admission into orthopaedic residency, and (2) whether race and gender are independently associated with admission into an orthopedic residency.

METHODS

The Electronic Residency Application System (ERAS) data from the Association of American Medical Colleges (AAMC) and the National Board of Medical Examiners (NBME) of first-time MD applicants (n = 8966) for orthopaedic surgery residency positions in the United States and of admitted orthopaedic residents (n = 6218) from 2005 to 2014 were reviewed. This dataset is the first and most comprehensive of its kind to date in orthopaedic surgery. Academic metrics, such as USMLE Step 1 and Step 2 Clinical Knowledge scores, number of publications, Alpha Omega Alpha status, volunteer experiences, work experience, as well as race and gender, were analyzed using hierarchical logistic regression models. The first model analyzed the association of academic metrics with admission into orthopaedic residency. In the second model, we added race and gender and controlled for metrics of academic performance. To determine how well the models simulated the actual admissions data, we computed the receiver operating characteristics (ROC) including the area under curve (AUC), which measures the model's ability to simulate which applicants were admitted or not admitted, with an AUC = 1.0 representing a perfect simulation. The odds ratio and confidence interval of each variable were computed.

RESULTS

When only academic variables were analyzed in the first model, Alpha Omega Alpha status (odds ratio 2.12 [95% CI 1.80 to 2.50]; p < 0.001), the USMLE Step 1 score (OR 1.04 [95% CI 1.03 to 1.04]; p < 0.001), the USMLE Step 2 Clinical Knowledge score (OR 1.01 [95% CI 1.01 to 1.02]; p < 0.001), publication count (OR 1.04 [95% CI 1.03 to 1.05]; p < 0.001), and volunteer experience (OR 1.03 [95% CI 1.01 to 1.04]; p < 0.001) were associated with admissions into orthopaedics while work and research experience were not. This model yielded a good prediction of the results with an AUC of 0.755. The second model, in which the variables of race and gender were added to the academic variables, also had a good prediction of the results with an AUC of 0.759. This model indicates that applicant race, but not gender, is associated with admissions into orthopaedic residency. Applicants from Asian (OR 0.78 [95% CI 0.67 to 0.92]), Black (OR 0.63 [95% CI 0.51 to 0.77], Hispanic (OR 0.48 [95% CI 0.36 to 0.65]), or other race groups (OR 0.65 [95% CI 0.55 to 0.77]) had lower odds of admission into residency compared with white applicants.

CONCLUSION

Minority applicants, but not women, have lower odds of admission into orthopaedic surgery residency, even when accounting for academic performance metrics. Changes in the residency selection processes are needed to eliminate the lower admission probability of qualified minority applicants in orthopaedic residency and to improve the diversity and inclusion of orthopaedic surgery. Changes including increasing the diversity of the selection committee, bias training, blinding applications before review, removal of metrics with history of racial disparities from an interviewer's candidate profile before an interview, and use of holistic application review (where an applicants' experiences, attributes, and academic metrics are all considered) can improve the diversity landscape in training. In addition, cultivating an environment of inclusion will be necessary to address these long-standing trends in orthopaedic surgery.

CLINICAL RELEVANCE

Race, but not gender, is associated with the odds of acceptance into orthopaedic surgery residency despite equivalent academic metrics. Changes in residency selection processes are suggested to eliminate the lower admission probability of qualified minority applicants into orthopaedic residency and to improve the diversity and inclusion of orthopaedic surgery.

Analysis of Current Orthopedic Surgery Residents and Their Prior Medical Education: Does Medical School Ranking Matter in Orthopedic Surgery Match?

OBJECTIVE

The purpose of this study was to determine the strength of the association between medical school ranking and orthopedic surgery residency ranking using the current cohort of orthopedic surgery residents.

DESIGN

We obtained a list of accredited programs from Doximity for orthopedic surgery residency programs and U.S. News & World Report for medical schools. Each orthopedic surgery residency program webpage was evaluated for the presence of an orthopedic surgery residency roster. For each resident, the medical school attended, allopathic or osteopathic degree, and year of post-graduate training was recorded. Orthopedic surgery residency programs and medical schools were assigned to one of four tiers for each based on their respective ranking. Descriptive statistics, Chi squared tests and Pearson residuals were used to analyze the association of orthopedic surgery residency tier and medical school tier. Post-hoc pairwise comparisons were performed utilizing the Bonferroni correction to account for 16 tests, correcting the significance level to p = 0.003.

SETTING

187 orthopedic surgery residency program webpages.

PARTICIPANTS

4123 orthopedic surgery residents.

RESULTS

There was a significant association between medical school tier and orthopedic surgery residency tier (X² [9] = 1214.78, p < 0.001). The post-hoc residual values were statistically significant for 75% (12/16) of tests performed. The majority of Tier 1 orthopedic surgery residents 50.5% (800/1585) attended a Tier 1 medical school. The strongest positive association exists between Tier 1 medical students attending Tier 1 residencies (residual = 23.978, p < 0.001). The strongest negative association with Tier 4 residencies was with Tier 1 medical schools (residual= -15.656, p< 0.001).

CONCLUSIONS

Medical school ranking is an important consideration for prospective orthopedic surgery applicants and may become more important with less objective measures of academic performance such as United States Medical Licensing Examination Step 1.

LEVEL OF EVIDENCE

Observational.

Emergency Medicine Residency Does Not Reduce a Racial Minority-Based Test Performance Gap

BACKGROUND

Health care inequity is corrected more readily when safe, high-quality care is provided by physicians who reflect the gender, race, and ethnicity of patient communities. It is important to train and evaluate racially diverse physicians involved in residency training.

OBJECTIVE

This study sought to determine any test-taking differences for black Emergency Medicine (EM) residents and whether any such differences would narrow as residency progressed.

METHODS

This was an observational, cross-sectional study that reviewed performance (scaled scores) on the American Board of Emergency Medicine (ABEM) In-Training Examination (ITE) for 2018, 2019, and 2020. The study included EM residents in 3-year programs who took the ITE. A linear regression model was used for the variables of race, which included black physicians and white physicians (reference group), and level of training (EM resident year 1 [EM1] as the reference group).

RESULTS

There were 9591 residents included; 539 were black and 9052 were white. Mean scaled scores were higher as a function of training level. Regression showed a scaled score intercept of 73.51. The ITE score increased for all groups as a function of training level (EM2 β = +5.45, p < 0.0001; EM3 β = +8.09, p < 0.0001). The regression coefficient for black residents was -5.87 (p < 0.0001). There was relative improvement by training level compared with improvement in the reference group, but this difference was not materially or statistically significant.

CONCLUSION

In this study of the ABEM ITE, a test-taking performance gap identified early in residency for black physicians persisted into late residency.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

OBJECTIVE

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences.

DESIGN

PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded.

RESULTS

Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation.

CONCLUSIONS

As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

Responding to Recommended Changes to the 2020-2021 Residency Recruitment Process From a Diversity, Equity, and Inclusion Perspective

In May 2020, the Coalition for Physician Accountability's Work Group on Medical Students in the Class of 2021 Moving Across Institutions for Post Graduate Training (WG) released its final report and recommendations. These recommendations pertain to away rotations, virtual interviews, Electronic Residency Application Service opening for programs and the overall residency timeline, and general communications and attempt to provide clarity and level the playing field during the 2020-2021 residency application cycle. The WG's aims include promoting professional accountability by improving the quality, efficiency, and continuity of the education, training, and assessment of physicians. The authors argue the first 3 WG recommendations may disproportionately impact candidates from historically excluded and underrepresented groups in medicine (HEURGMs) and may affect an institution's ability to ensure equity in the selection of residency applicants and, thus, warrant further consideration. The authors examine these recommendations from a diversity, equity, and inclusion (DEI) perspective. For each of the first 3 WG recommendations, the authors highlight new opportunities created by the recommendations and detail challenges that programs must carefully navigate to ensure equity for all candidates. The authors also recommend solutions to guide programs as they address these challenges, meet new common program requirements, and attempt to promote equity for HEURGMs. Finally, the authors recommend that after the 2020-2021 recruitment cycle, the medical education community evaluate DEI-related outcomes of both the WG's and the authors' recommendations and incorporate the findings into future application cycles.

Freedom from discrimination or freedom to discriminate? Discursive tensions within discrimination policies in medical education

The importance of advancing equity, diversity, and inclusion for all members of the academic medical community has gained recent attention. Academic medical organizations have attempted to increase broader representation while seeking structural reforms consistent with the goal of enhancing equity and reducing disproportionality. However, efforts remain constrained while minority groups continue to experience discrimination. In this study, the authors sought to identify and understand the discursive effects of discrimination policies within medical education. The authors assembled an archive of 22 texts consisting of publicly available discrimination and harassment policy documents in 13 Canadian medical schools that were active as of November 2019. Each text was analysed to identify themes, rhetorical strategies, problematization, and power relations. Policies described truth statements that appear to idealize equity, yet there were discourses related to professionalism and neutrality that were in tension with these ideals. There was also tension between organizations' framing of a shared responsibility for addressing discrimination and individual responsibility on complainants. Lastly, there were also competing discourses on promoting freedom from discrimination and the concept of academic freedom. Overall, findings reveal several areas of tension that shape how discrimination is addressed in policy versus practice. Existing discourses regarding self-protection and academic freedom suggest equity cannot be advanced through policy discourse alone and more substantive structural transformation may be necessary. Existing approaches may be inadequate to address discrimination unless academic medical organizations interrogate the source of these discursive tensions and consider asymmetries of power.

A disturbing trend: An analysis of the decline in surgical critical care (SCC) fellowship training of Black and Hispanic surgeons

BACKGROUND

Underrepresented minorities in medicine (URiM) are disproportionally represented in surgery training programs. Rates of URiM applying to and completing General Surgery residency remain low. We hypothesized that the patterns of URiM disparities would persist into Surgical Critical Care (SCC) fellowship applicants, matriculants and graduates.

METHODS

We performed a retrospective analysis of SCC applicants, matriculants and graduates from 2005-2020 using the Graduate Medical Education (GME) resident survey and analyzed applicant characteristics using the Surgical critical care and Acute care surgery Fellowship Application Service (SAFAS) from 2018-2020. The data were stratified by race/ethnicity and gender. Indicator variables were created for Asian, Hispanic, White and Black trainees. Yearly proportions for each race/ethnicity and gender categories completing or enrolling in a program were calculated and plotted over time with Loess smoothing lines and overlying 95% confidence bands. The yearly rate and statistical significance of change over time were tested with linear regression models with race/ethnicity and gender proportion as the dependent variables and year as the explanatory variable.

RESULTS

From 2005-2020, there were a total of 2,481 graduates. Black men accounted for 4.7% of male graduates with a significant decline of 0.3% per year for the study period of those completing the fellowship (p = 0.02). Black women comprised 6.4% of female graduates and had a 0.6% decline each year (p < 0.01). A similar trend was seen with Hispanic men, who comprised 3.2% of male graduates and had a 0.3% annual decline (p = 0.02). White men had a significant increase in both matriculation to and graduation from SCC fellowships during the same interval. Similarly, Black and Hispanic applicants declined from 2019 to 2020, while the percentage of White applicants increased.

CONCLUSIONS

Disparities in URiM representation remain omnipresent in surgery and extend from residency training to SCC fellowship. Efforts to enhance the recruitment and retention of URiM in SCC training are warranted.

LEVEL OF EVIDENCE

Level IV - Therapeutic/Care Management.

Effect of Application Screening Methods on Racial and Ethnic Diversity in Otolaryngology

Otolaryngology-head and neck surgery (OHNS) lags behind other surgical subspecialties in the representation of underrepresented minorities in medicine (URMs). Given the recently announced changes to Step 1 scoring, we aimed to assess the effect of alternative application screening methods-Step 2 Clinical Knowledge scores and Alpha Omega Alpha membership-on the racial/ethnic diversity of the OHNS applicant pool. After reviewing OHNS residency applications submitted to our institution for the 2015-2020 matches (N = 2177), we determined that a significantly greater proportion of URM vs non-URM applicants would be screened out from interview consideration if any the following were used as an initial screening method: Step 2 cutoff score of 240, Step 2 cutoff score of 253 or non-Alpha Omega Alpha membership (P < .01 for each). Given that using these metrics to screen applications disproportionately affects URMs, programs should consider implementing alternative application review methods, such as holistic evaluation, which may promote more equitable distribution of interviews.

Perceptions of USMLE Step 1 Pass/Fail Score Reporting Among Orthopedic Surgery Residency Program Directors

The United States Medical Licensing Examination (USMLE) Step 1 examination will transition from graded to pass/fail scoring starting no earlier than January 2022. Orthopedic surgery residency programs will need to adapt to these changes. The goal of this study was to investigate the perceptions of orthopedic surgery residency program directors on the change of Step 1 from a graded to a pass/fail examination. We also investigated how the change would affect the other factors that are typically considered in the selection of orthopedic surgery residents. A survey was distributed to 161 directors of allopathic orthopedic surgery programs. Contact information was obtained from a national database. Of those contacted, 75 (46.6%) program directors responded. Most (85.3%) did not support the pass/fail change. Most believe that greater importance will be placed on the Step 2 Clinical Knowledge examination (96.0%), audition elective with their department (84.0%), personal knowledge of the applicant (78.7%), grades (74.7%), letters of recommendation from recognizable orthopedic surgeons (74.7%), and Alpha Omega Alpha status (69.3%). Most also believe that this change will advantage allopathic students who attend highly regarded schools (58.7%). Most of the program directors support a graded preclinical curriculum (69.3%) and caps on the number of orthopedic surgery residency applications (70.7%). Although most orthopedic surgery program directors disagree with the change to a pass/fail Step 1 examination, residency programs will need to reevaluate how they screen applicants for an interview once the scored Step 1 is no longer available. With this change, other factors, such as Step 2 score, audition rotations, and grades in clerkships, will be emphasized more heavily. [Orthopedics. 2022;45(1):e30-e34.].

Evaluating the Whole Applicant: Use of Situational Judgment Testing and Personality Testing to Address Disparities in Resident Selection

PURPOSE OF REVIEW

Urology program directors are faced with increasing numbers of applications annually, making holistic review of each candidate progressively more difficult. Efforts to streamline evaluation using traditional cognitive metrics have fallen short as these do not predict overall resident performance. Situational judgment tests (SJTs) and personality assessment tools (PATs) have been used in business and industry for decades to evaluate candidates and measure non-cognitive attributes that better predict subsequent performance. The purpose of this review is to describe what these assessments are and the current literature on the use of these metrics in medical education.

RECENT FINDINGS

SJTs relative to PATs have more original research. Data suggests that SJTs decrease bias, increase diversity, and may be predictive of performance in residency. PATs are also emerging with data to support use with ability to assess fit to program and certain traits identified more consistently among high-performing residents and correlation to performance on ACGME milestones. PATs may be more coachable than SJTs. SJTs and PATs are emerging as techniques to supplement the current resident application review process. Early evidence supports their use in undergraduate medical education as does some early preliminary results in graduate medical education.

Strategies for Recruiting and Retaining Women and Minorities in Orthopaedics: AOA Critical Issues Symposium

Despite considerable attention being paid to the lack of diversity in orthopaedic surgery over the last decade, there has been very little actual change in the racial and gender demographics. This article discusses mechanisms for improving the diversity of interested programs, including reviewing potential barriers to racial and gender-based diversity programs.

While You Don’t See Color, I See Bias: Identifying Barriers in Access to Graduate Medical Education Training

There is a need to acknowledge and address issues of implicit and explicit bias within medical education. These biases can impact standardized test questions and scores, evaluations of clinical performance, and subsequent letters of recommendation, all of which can affect the selection of diverse candidates advancing through medical training. Biased behavior toward trainees can negatively impact their learning environment and career trajectory. This article outlines key definitions related to bias and discusses the ways in which bias potentially impacts selection and entry into Pulmonary and Critical Care Medicine fellowship training. Finally, we will describe some ways to mitigate bias within the fellowship selection process and training programs.

What is the Trend in Representation of Women and Under-represented Minorities in Orthopaedic Surgery Residency?

BACKGROUND

Orthopaedic surgery training programs have lagged behind other surgical specialties in increasing their representation of women and people from under-represented minority (URM) groups. Comparative data between orthopaedic surgery and other specialties are needed to help identify solutions to closing the diversity gap.

QUESTIONS/PURPOSES

(1) Which surgical specialties have the greatest representation of women residents and residents from URM groups? (2) How have the proportions of women residents and residents from URM groups changed across the surgical specialties during the past decade?

METHODS

This was a retrospective evaluation of a large, longitudinally maintained survey database. Resident data by gender and ethnicity were retrieved from the Accreditation Council for Graduate Medical Education Data Resource Books for the 2011 to 2012 through 2019 to 2020 academic years. The Accreditation Council for Graduate Medical Education database is updated annually; thus, it is the most up-to-date and complete database available for gender and ethnicity data for all surgical residents. Data were obtained and analyzed for seven different surgical specialties: orthopaedic surgery, neurosurgery, ophthalmology, otolaryngology, plastic surgery, general surgery, and urology. No sampling was necessary, and thus descriptive statistics of the data were completed. Because the entire population of residents was included for the period of time in question, no statistical comparisons were made, and the reported differences represent absolute differences between the groups for these periods. Linear regression analyses were performed to estimate the annual growth rates of women residents and residents from URM groups in each specialty.

RESULTS

Among the seven surgical specialties, representation of women residents increased from 28% (4640 of 16,854) of residents in 2012 to 33% (6879 of 20,788) in 2020. Orthopaedic surgery had the lowest representation of women residents every year, with women residents comprising 16% of residents (700 of 4342) in 2020. Among the seven surgical specialties, representation of residents from URM groups increased from 8.1% (1362 of 16,854) in 2012 to 9.7% (2013 of 20,788) in 2020. In 2020, the representation of residents from URM groups in orthopaedic surgery was 7.7% (333 of 4342). In 2020, general surgery had the highest representation of women residents (42%; 3696 of 8809) as well as residents from URM groups (12%; 1065 of 8809). Plastic surgery (1.46% per year) and general surgery (0.95% per year) had larger annual growth rates of women residents than the other specialties did. In each surgical specialty, the annual growth rate of residents from URM groups was insignificant.

CONCLUSION

During the past decade, there was only a small increase in the representation of women in orthopaedic surgery, while the representation of people from URM groups did not change. In contrast, by 2020, general surgery had become the most diverse among the seven surgical specialties. To increase diversity in our field, we need to evaluate and implement some of the effective interventions that have helped general surgery become the diverse surgical specialty that it is today.

CLINICAL RELEVANCE

General surgery has substantially reduced gender and ethnic disparities that existed in the past, while those in orthopaedic surgery still persist. General surgery residencies have implemented a holistic review of resident applications and longitudinal mentoring programs to successfully address these disparities. Orthopaedic surgery programs should consider placing less emphasis on United States Medical Licensing Examination score thresholds and more weight on applicants' non-academic attributes, and put more efforts into targeted longitudinal mentorship programs, some of which should be led by non-minority faculty.

Implicit Bias and the Association of Redaction of Identifiers With Residency Application Screening Scores

Importance

Diversity in the ophthalmology profession is important when providing care for an increasingly diverse patient population. However, implicit bias may inadvertently disadvantage underrepresented applicants during resident recruitment and selection.

Objective

To evaluate the association of the redaction of applicant identifiers with the review scores on ophthalmology residency applications as an intervention to address implicit bias.

Design, Setting, and Participants

In this quality improvement study, 46 faculty members reviewed randomized sets of 462 redacted and unredacted applications from a single academic institution during the 2019-2020 ophthalmology residency application cycle.

Interventions

Applications electronically redacted for applicant identifiers, including name, sex or gender, race and ethnicity, and related terms.

Main Outcomes and Measures

The main outcome was the distribution of scores on redacted and unredacted applications, stratified by applicant's sex, underrepresentation in medicine (URiM; traditionally comprising American Indian or Alaskan Native, Black, and Hispanic individuals) status, and international medical graduate (IMG) status; the application score β coefficients for redaction and the applicant and reviewer characteristics were calculated. Applications were scored on a scale of 1 to 9, where 1 was the best score and 9 was the worst score. Scores were evaluated for a significant difference based on redaction among female, URiM, and IMG applicants. Linear regression was used to evaluate the adjusted association of redaction, self-reported applicant characteristics, and reviewer characteristics with scores on ophthalmology residency applications.

Results

In this study, 277 applicants (60.0%) were male and 71 (15.4%) had URiM status; 32 faculty reviewers (69.6%) were male and 2 (0.4%) had URiM status. The distribution of scores was similar for redacted vs unredacted applications, with no difference based on sex, URiM status, or IMG status. Applicant's sex, URiM status, and IMG status had no association with scores in multivariable analysis (sex, β = -0.08; 95% CI, -0.32 to 0.15; P = .26; URiM status, β = -0.03; (95% CI, -0.36 to 0.30; P = .94; and IMG status, β = 0.39; 95% CI, -0.24 to 1.02; P = .35). In adjusted regression, redaction was not associated with differences in scores (β = -0.06 points on a 1-9 scale; 95% CI, -0.22 to 0.10 points; P = .48). Factors most associated with better scores were attending a top 20 medical school (β = -1.06; 95% CI, -1.37 to -0.76; P < .001), holding an additional advanced degree (β = -0.86; 95% CI, -1.22 to -0.50; P < .001), and having a higher United States Medical Licensing Examination Step 1 score (β = -0.35 per 10-point increase; 95% CI, -0.45 to -0.26; P < .001).

Conclusions and Relevance

This quality improvement study did not detect an association between the redaction of applicant characteristics on ophthalmology residency applications and the application review scores among underrepresented candidates at this institution. Although the study may not have been powered adequately to find a difference, these findings suggest that the association of redaction with application review scores may be preempted by additional approaches to enhance diversity, including pipeline programs, implicit bias training, diversity-centered culture and priorities, and targeted applicant outreach. Programs may adapt this study design to probe their own application screening biases and track over time before-and-after bias-related interventions.

Inclusion and equity: Experiences of underrepresented in medicine physicians throughout the medical education continuum

The mission of pediatric medical education is to train the workforce who will provide and innovate child healthcare, including addressing healthcare inequities for children and adolescents of color. A diverse physician workforce is associated with more equitable outcomes. Therefore, it is imperative that our workforce continue to reflect the ever-growing diverse nature of our society. While diversity has been at the forefront of conversations for increasing the number of underrepresented communities in medicine, inclusion and equity are oftentimes overlooked. It is crucial to recognize that while diversity is an important aspect to the UIM experience, the overall impact of diversity is diluted when it is not paired with inclusion and equity. Therefore, the purpose of this paper is to illustrate the experiences that shape inclusion and equity for UIM physicians along the training continuum and how negative factors may impact the wellness and career longevity of UIM physicians. Our study explored the experiences of UIM physicians throughout their medical education through the lens of inclusion and equity by exploring three common factors: social support, racism and discrimination, and well-being and burnout. There may be a perception that stressors, including racial discrimination, diminish with progression through one's educational career; however, this review suggests that these stressors manifest differently relative to the trials inherent to each career stage. By exploring the interplay between these factors and the UIM experience as a continuum, we are better able to discuss effective solutions to diversity, equity and inclusion for UIM physicians.

Underrepresented in medicine in graduate medical education: Historical trends, bias, and recruitment practices

Health inequities disproportionately impact children of color in the United States. Research demonstrates that a diverse physician workforce that reflects the demographic profile of the United States population has been associated with better health outcomes and physicians of color are more likely to work in communities that are medically underserved. Therefore, a key driver to reducing health inequities is recruiting a physician workforce that reflects the racial/ethnic composition of the population. Although efforts on increasing UIM applicants and matriculants into medical school and more recently, graduate medical education (GME) programs have shown some small successes, there has still not been significant change in decades. This narrative review examined the historical trends and current landscape of recruitment in GME to better understand the barriers and facilitators that continue to exist and identify successful interventions that might serve as a model to support recruitment of UIM applicants. There is evidence for racial disparities and bias in traditional residency selection metrics (USMLE, Clerkship Evaluations, MSPE, AOA status, etc.) and a need for more holistic recruitment strategies. Along with implementing new holistic recruitment practices, there is a need for ongoing research and collaboration to study inclusive and equitable learning environments and assessments that will eliminate racial disparities in educational outcomes. Additionally, there must be better longitudinal data utilization and transparency across Electronic Residency Application Services (ERAS) and National Resident Matching Program (NRMP) for accountability and tracking of interventions.