The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Gender differences in cardiothoracic surgery letters of recommendation

OBJECTIVE

To investigate whether or not gender influences letters of recommendation for cardiothoracic surgery fellowship.

METHODS

From applications to an Accreditation Council Graduate Medical Education cardiothoracic surgery fellowship program between 2016 and 2021, applicant and author characteristics were examined with descriptive statistics, analysis of variance, and Pearson χ2 tests. Linguistic software was used to assess communication differences in letters of recommendation, stratified by author and applicant gender. An additional higher-level analysis was then performed using a generalized estimating equations model to examine linguistic differences among author-applicant gender pairs.

RESULTS

Seven hundred thirty-nine recommendation letters extracted from 196 individual applications were analyzed; 90% (n = 665) of authors were men and 55.8% (n = 412) of authors were cardiothoracic surgeons. Compared with women authors, authors who are men wrote more authentic (P = .01) and informal (P = .03) recommendation letters. When writing for women applicants, authors who are men were more likely to display their own leadership and status (P = .03) and discuss women applicants' social affiliations (P = .01), like occupation of applicant's father or husband. Women authors wrote longer letters (P = .03) and discussed applicants' work (P = .01) more often than authors who are men. They also mentioned leisure activities (P = .03) more often when writing for women applicants.

CONCLUSIONS

Our work identifies gender-specific differences in letters of recommendation. Women applicants may be disadvantaged because their recommendation letters are significantly more likely to focus on their social ties, leisure activities, and the status of the letter writer. Author and reviewer awareness of gender-biased use of language will aid in improvements to the candidate selection process.

Perception of Quiet Students in Emergency Medicine: An Exploration of Narratives in the Standardized Letter of Evaluation

INTRODUCTION

The Standardized Letter of Evaluation (SLOE) is designed to assist emergency medicine (EM) residency programs in differentiating applicants and in selecting those to interview. The SLOE narrative component summarizes the student's clinical skills as well as their non-cognitive attributes. The purpose of this qualitative investigation was to explore how students described in the SLOE as quiet are perceived by faculty and to better understand how this may impact their residency candidacy.

METHODS

This retrospective cohort study included all SLOEs submitted to one EM residency program during one application cycle. We analyzed sentences in the SLOE narrative describing students as "quiet," "shy," and/or "reserved." Using grounded theory, thematic content analysis with a constructivist approach, we identified five mutually exclusive themes that best characterized the usage of these target words.

RESULTS

We identified five themes: 1) quiet traits portrayed as implied-negative attributes (62.4%); 2) quiet students portrayed as overshadowed by more extraverted peers (10.3%); 3) quiet students portrayed as unfit for fast-paced clinical settings (3.4%); 4) "quiet" portrayed as a positive attribute (10.3%); and 5) "quiet" comments deemed difficult to assess due to lack of context (15.6%).

CONCLUSION

We found that quiet personality traits were often portrayed as negative attributes. Further, comments often lacked clinical context, leaving them vulnerable to misunderstanding or bias. More research is needed to determine how quiet students perform compared to their non-quiet peers and to determine what changes to instructional practices may support the quiet student and help create a more inclusive learning environment.

The Presence of Implicit Gender Bias in Colon and Rectal Surgery Residency Letters of Recommendation

BACKGROUND

Selection for colorectal surgery residency relies on letters of recommendation for assessment of candidates' strengths and weaknesses. It is unclear whether this process incorporates implicit gender bias.

OBJECTIVE

This study aimed to assess the presence of gender bias in letters of recommendation for colorectal surgery residency.

DESIGN

Mixed methods assessment of the characteristics described within the blinded letters of the 2019 application cycle to a single academic residency.

SETTINGS

Academic medical center.

PATIENTS

Blinded letters from the 2019 colorectal surgery residency application cycle.

INTERVENTIONS

Characteristics of the letters were qualitatively and quantitatively analyzed.

MAIN OUTCOME MEASURES

Association of gender with the presence of descriptors within the letters.

RESULTS

A total of 111 applicants, 409 letter writers, and 658 letters were analyzed. Forty-three percent of applicants were female. Female and male applicants had an equal mean number of positive (5.4 vs 5.8; p = 0.10) and negative (0.5 vs 0.4; p = 0.07) attributes represented. Female applicants were more likely to be described as having poor academic skills (6.0 vs 3.4%; p = 0.04) and possessing negative leadership qualities (5.2% vs 1.4%; p < 0.01) than male applicants. Male applicants were more likely to be described as kind (36.6% vs 28.3%; p = 0.03), curious (16.4% vs 9.2%; p = 0.01), possessing positive academic skills (33.7% vs 20.0%; p < 0.01), and possessing positive teaching skills (23.5% vs 17.0%; p = 0.04).

LIMITATIONS

This study analyzed a single year of applications to an academic center and may not be generalizable.

CONCLUSIONS

There are differences in the qualities used to describe female versus male applicants in colorectal surgery residency application letters of recommendation. Female applicants were more often described in negative academic terms and possessing negative leadership qualities. Males were more likely to be described as kind, curious, academically impressive, and possessing good teaching skills. The field may benefit from educational initiatives to reduce implicit gender bias in letters of recommendation. See Video Abstract at http://links.lww.com/DCR/C191 .

LA PRESENCIA DE SESGO DE GNERO IMPLCITO EN LAS CARTAS DE RECOMENDACIN DE RESIDENCIA EN CIRUGA DE COLON Y RECTO

ANTECEDENTES:La selección para la residencia en cirugía colorrectal se basa en cartas de recomendación para la evaluación subjetiva de las fortalezas y debilidades de los candidatos. No está claro si este proceso incorpora un sesgo de género implícito.OBJETIVO:Evaluar la presencia de sesgo de género en las cartas de recomendación para la residencia en cirugía colorrectal.DISEÑO:Evaluación de métodos mixtos de las características descritas dentro de las cartas selladas del ciclo de solicitud de 2019 a una sola residencia académica.ENTORNO CLÍNICO:Centro médico académico.PACIENTES:Cartas selladas del ciclo de solicitud de residencia en cirugía colorrectal de 2019.INTERVENCIONES:Las características de las cartas se determinaron utilizando medidas cualitativas y cuantitativas.PRINCIPALES MEDIDAS DE VALORACIÓN:Asociación del género con la presencia de descriptores dentro de las cartas.RESULTADOS:Hubo 111 solicitantes, 409 escritores de cartas y se analizaron 658 cartas. El 43% de los solicitantes eran mujeres. Los solicitantes masculinos y femeninos tenían el mismo promedio de atributos positivos (5,4 frente a 5,8; p = 0,10) y negativos (0,5 frente a 0,4; p = 0,07) representados. Las solicitantes femeninas tenían más probabilidades de ser descritas como con deficientes habilidades académicas (6,0 frente a 3,4%, p = 0,04) y poseían cualidades de liderazgo negativas (5,2% frente a 1,4%; p < 0,01) en comparacion con los solicitantes masculinos. Los solicitantes masculinos tenían más probabilidades de ser descritos como amables (36,6 % frente a 28,3%; p = 0,03), curiosos (16,4% frente a 9,2%; p = 0,01), que poseían habilidades académicas positivas (33,7 % frente a 20,0%; p < 0,01), y habilidades docentes positivas (23,5% vs 17,0%; p = 0,04).LIMITACIONES:Este estudio analizó un solo año de solicitudes a un centro académico y puede no ser generalizable.CONCLUSIÓN:Existen diferencias en las cualidades utilizadas para describir a los solicitantes femeninos versus masculinos en las cartas de recomendación de solicitud de residencia en cirugía colorrectal. Las candidatas femeninas se describieron con mayor frecuencia en términos académicos negativos y poseían cualidades de liderazgo negativas. Los hombres eran más propensos a ser descritos como amables, curiosos, académicamente impresionantes y con buenas habilidades docentes. El campo puede beneficiarse de iniciativas educativas para reducir el sesgo de género implícito en las cartas de recomendación. Consulte Video Resumen en http://links.lww.com/DCR/C191 . (Traducción-Dr. Ingrid Melo ).

Standardized Letter of Recommendation: Can Everyone Be Awesome?

OBJECTIVE

Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed.

DESIGN

Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good.

SETTING

University of Hospitals Cleveland Medical Center.

PARTICIPANTS

SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009).

CONCLUSIONS

The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.

Characterizing Standardized Letters of Recommendation in Urology Residency Applications

OBJECTIVE

To identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants' ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases.

METHODS

We evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings.

RESULTS

Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in "urology resident potential," "academic urologist potential," and "performance as a sub-intern" domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for "communication".

CONCLUSION

While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.

Gender Differences in Language of Standardized Letter of Evaluation Narratives in Osteopathic Emergency Medicine Residency Applicants

Background

The standardized letter of evaluation (SLOE) is used by emergency medicine (EM) faculty during the interview and match process. Data has shown that female allopathic applicants score higher in communal characteristics and have a greater number of ability words in the narrative portion of the SLOE as compared to their male counterparts. 

Objective

To determine if there is a difference in the language used to describe male and female osteopathic applicants within the SLOE. 

Methods

All applicants to a three-year EM residency within a single application cycle were eligible for inclusion. Exclusion criteria included allopathic applicants, applicants without a SLOE, or applicants with a SLOE only from the interviewing program. Data collected included applicant demographics and SLOE narratives. The previously validated Linguistic Inquiry and Word Count (LIWC; Pennebaker Conglomerates, Inc., Austin, TX) product was used to analyze word counts from the narrative portion of each SLOE. Descriptive statistics and t-tests for continuous data were used. 

Results

Of the 577 applicants to the residency program, 318 met inclusion criteria and 33% were female. Females had a higher COMLEX-2 (590 vs 559; p=0.05) as compared to males but no difference was found for the remainder of the baseline demographics. No difference was found for the number of words in the narrative portion of the SLOE between males and females (males = 122 words; females = 127 words; p=0.53). Words within the social (p=0.006), achievement (p=0.007), and standout (p<0.001) categories were more frequent in osteopathic female applicants as compared to males. No statistical differences were detected for the other 13 categories analyzed. 

Conclusion

In this sample of osteopathic applicants, little linguistic difference was noted for the narrative portion of the SLOE. SLOE authors did, however, use more social, achievement, and standout words to describe females as compared to male applicants. 

Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

BACKGROUND

The Medical Student Performance Evaluation (MSPE) provides important information to residency programs. Despite recent recommendations for standardization, it is not clear how much variation exists in MSPE content among schools.

OBJECTIVES

We describe the current section content of the MSPE in US allopathic medical schools, with a particular focus on variations in the presentation of student performance.

METHODS

A representative MSPE was obtained from 95.3% (143 of 150) of allopathic US medical schools through residency applications to the Zucker School of Medicine at Hofstra/Northwell in select programs for the 2019-2020 academic year. A manual data abstraction tool was piloted in 2018-2019. After training, it was used to code all portions of the MSPE in this study. The results were analyzed, and descriptive statistics were reported.

RESULTS

In preclinical years, 30.8% of MSPEs reported data regarding performance of students beyond achieving "passes" in a pass/fail curriculum. Only half referenced performance in the fourth year including electives, acting internships, or both. About two-thirds of schools included an overall descriptor of comparative performance in the final paragraph. Among these schools, a majority provided adjectives such as "outstanding/excellent/very good/good," while one-quarter reported numerical data categories. Regarding clerkship grades, there were numerous nomenclature systems used.

CONCLUSIONS

This analysis demonstrates the existence of extreme variability in the content of MSPEs submitted by US allopathic medical schools in the 2019-2020 cycle, including the components and nomenclature of grades and descriptors of comparative performance, display of data, and inclusion of data across all years of the medical education program.

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

OBJECTIVES

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.

METHODS

A multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

RESULTS

In total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: "ability to develop and justify an appropriate differential and a cohesive treatment plan" (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), "how much guidance do you predict this applicant will need during residency?" (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and "what is your prediction of success for the applicant?" (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

CONCLUSIONS

There was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.

Making a match: trends in the application, interview, and ranking process for the neurological surgery residency programs

OBJECTIVE

Neurosurgery is a highly competitive residency field with a match rate lower than that of other specialties. The aim of this study was to analyze trends associated with the residency match process from the applicants' and program directors' perspectives.

METHODS

Between 2010 and 2020, the National Residency Matching Program (NRMP) Applicant and Program Director Surveys, the NRMP Charting Outcomes reports, and the Accreditation Council for Graduate Medical Education (ACGME) Data Resource Books were analyzed to identify the number of applicants interviewed and ranked in US programs, the applicants' ranking preferences, the program directors' preferential factors in offering interviews, and rank list order. Applicants were divided between US senior medical students and independent applicants. Each cohort was dichotomized for matched and unmatched applicants.

RESULTS

Over the study period, 2935 applicants applied to neurosurgery residency, including 2135 US senior medical students and 800 independent applicants, with an overall match rate of 65%. Overall, matched applicants had a significantly higher number of publications (p < 0.05). Among US senior medical student applicants, the application-to-interview ratio more than doubled over the study period, yet the number of interview invitations received, interviews accepted, and programs ranked remained unchanged. In the US senior medical student cohort, the number of submitted applications, interview invitations, accepted interviews, and programs ranked did not significantly differ between matched and unmatched applicants. In both cohorts, applicants shifted ranking factors from a more academic focus in early years to more well-being in later years. Letters of recommendation and board scores were key factors for program directors while screening applicants for interviews and ranking.

CONCLUSIONS

Neurosurgery residency continues to be a highly competitive field in medicine, with match rates of 65%. Recently, applicants have placed greater importance on ranking programs that value residents' well-being, as opposed to strictly academic factors. A data-driven understanding of factors important to applicants and program directors during the match process has the potential to improve resident candidate recruitment and overall resident-program fit, thereby improving well-being during residency, reducing the attrition rate, and overall enhancing the diversity of the neurosurgery resident workforce.

A Model for Exploring Compatibility Between Applicants and Residency Programs: Right Resident, Right Program

Holistic review of residency applications is touted as the gold standard for selection, yet vast application numbers leave programs reliant on screening using filters such as United States Medical Licensing Examination scores that do not reliably predict resident performance and may threaten diversity. Applicants struggle to identify which programs to apply to, and devote attention to these processes throughout most of the fourth year, distracting from their clinical education. In this perspective, educators across the undergraduate and graduate medical education continuum propose new models for student-program compatibility based on design thinking sessions with stakeholders in obstetrics and gynecology education from a broad range of training environments. First, we describe a framework for applicant-program compatibility based on applicant priorities and program offerings, including clinical training, academic training, practice setting, residency culture, personal life, and professional goals. Second, a conceptual model for applicant screening based on metrics, experiences, attributes, and alignment with program priorities is presented that might facilitate holistic review. We call for design and validation of novel metrics, such as situational judgment tests for professionalism. Together, these steps could improve the transparency, efficiency and fidelity of the residency application process. The models presented can be adapted to the priorities and values of other specialties.

The Residency Selection Process in the New World; Time for a Rethink

The transition of the USMLE to a pass fail system provides a unique opportunity to re-evaluate the residency selection process.

Everyone Is Awesome: Analyzing Letters of Reference in a General Surgery Residency Selection Process

BACKGROUND

The resident selection process involves the analysis of multiple data points, including letters of reference (LORs), which are inherently subjective in nature.

OBJECTIVE

We assessed the frequency with which LORs use quantitative terms to describe applicants and to assess whether the use of these terms reflects the ranking of trainees in the final selection process.

METHODS

A descriptive study analyzing LORs submitted by Canadian medical graduate applicants to the University of Ottawa General Surgery Program in 2019 was completed. We collected demographic information about applicants and referees and recorded the use of preidentified quantitative descriptors (eg, best, above average). A 10% audit of the data was performed. Descriptive statistics were used to analyze the demographics of our letters as well as the frequency of use of the quantitative descriptors.

RESULTS

Three hundred forty-three LORs for 114 applicants were analyzed. Eighty-five percent (291 of 343) of LORs used quantitative descriptors. Eighty-four percent (95 of 113) of applicants were described as above average, and 45% (51 of 113) were described as the "best" by at least 1 letter. The candidates described as the "best" ranked anywhere from second to 108th in our ranking system.

CONCLUSIONS

Most LORs use quantitative descriptors. These terms are generally positive, and while the use does discriminate between different applicants, it was not helpful in the context of ranking applicants in our file review process.