Council of emergency medicine residency directors standardized letter of recommendation writers' questionnaire

Measuring and Predicting Faculty Consensus Rankings of Standardized Letters of Evaluation

Background Standardized letters of evaluation (SLOE) are becoming more widely incorporated into the residency application process to make the letter of recommendation, an already critical component in a residency application packet, more objective. However, it is not currently known if the reviewers of these letters share consensus regarding the strength of an applicant determined by their SLOE. Objective We measured the level of faculty agreement regarding applicant competitiveness as determined by SLOEs and the ability of 2 algorithms to predict faculty consensus rankings. Methods Using data from the 2021-2022 Match cycle from the Council of Residency Directors in Emergency Medicine SLOE Database as a blueprint, authors created 50 fictional SLOEs representative of the national data. Seven faculty then rated these SLOEs in order of applicant competitiveness, defined as suggested rank position. Consensus was evaluated using cutoffs established a priori, and 2 prediction models, a point-based system and a linear regression model, were tested to determine their ability to predict consensus rankings. Results There was strong faculty consensus regarding the interpretation of SLOEs. Within narrow windows of agreement, faculty demonstrated similar ranking patterns with 83% and 93% agreement for "close" and "loose" agreement, respectively. Predictive models yielded a strong correlation with the consensus ranking (point-based system r=0.97, linear regression r=0.97). Conclusions Faculty displayed strong consensus regarding the competitiveness of applicants via SLOEs, adding further support to the use of SLOEs for selection and advising. Two models predicted consensus competitiveness rankings with a high degree of accuracy.

Standardized Letter of Recommendation: Can Everyone Be Awesome?

OBJECTIVE

Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed.

DESIGN

Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good.

SETTING

University of Hospitals Cleveland Medical Center.

PARTICIPANTS

SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009).

CONCLUSIONS

The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.

Trends in utilization of standardized letters of recommendation in the 2021-2022 neurosurgery application process

OBJECTIVE

Standardized letters of recommendation (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor interrater reliability and grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. The authors examined the second year of SLOR utilization. They hypothesized that grade inflation and interrater reliability would be improved from the first iteration. They also hypothesized that increased numbers of letters by single writers would correlate with broader rating distributions.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of the letter writer. Interrater reliability was evaluated using Krippendorff's alpha. The frequency of letters written was evaluated using the Kruskal-Wallis H test.

RESULTS

Ninety percent of SLORs rated applicants among the top 25%, but there was a significant decrease in the usage of the top 1% and top 2%-5% ratings. Interrater reliability was poor across all competencies. Writers who completed 1 SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs.

CONCLUSIONS

Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, interrater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.

An Examination of Standardized Letters of Recommendation Rating Scales Among Neurosurgical Residency Candidates During the 2020-2021 Application Cycle

BACKGROUND

Letters of recommendation (LORs) are historically an important, though subjective, component of the neurosurgery residency application process. Standardized LORs (SLORs) were introduced during the 2020 to 2021 application cycle. The intent of SLORs is to allow objective comparison of applicants and to reduce bias.

OBJECTIVE

To examine the utility of SLORs during this application cycle. We hypothesized that "grade inflation" and poor inter-rater reliability, as described by other specialties using SLORs, would limit the utility of SLORs in their current form.

METHODS

This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2020 to 2021 cycle. Data from 7 competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff's alpha.

RESULTS

One or more SLORs was submitted as part of 298 of 393 applications (76%). Approximately 58.3% of letters written by neurosurgery chairpersons rated a given applicant as being within the top 5% across all competencies. Approximately 44.4% of program director letters similarly rated applicants as amongst the top 5%, while 73.2% and 81.4% of letters by other neurosurgeons and general surgery evaluators, respectively, rated applicants in the top 5%. Inter-rater reliability was poor (<0.33) in all rating categories, including overall (α = 0.18).

CONCLUSION

The utility of the first iteration of SLORs in neurosurgery applications is undermined by significant "grade inflation" and poor inter-rater reliability. Improvements are necessary for SLORs if they are to provide meaningful information in future application cycles.

Validity Evidence for the Emergency Medicine Standardized Letter of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) is the application component that program directors value most when evaluating candidates to interview and rank for emergency medicine (EM) residency. Given its successful implementation, other specialties, including otolaryngology, dermatology, and orthopedics, have adopted similar SLOEs of their own, and more specialties are considering creating one. Unfortunately, for such a significant assessment tool, no study to date has comprehensively examined the validity evidence for the EM SLOE.

OBJECTIVE

We summarized the published evidence for validity for the EM SLOE using Messick's framework for validity evidence.

METHODS

A scoping review of the validity evidence of the EM SLOE was performed in 2020. A scoping review was chosen to identify gaps and future directions, and because the heterogeneity of the literature makes a systematic review difficult. Included articles were assigned to an aspect of Messick's framework and determined to provide evidence for or against validity.

RESULTS

There have been 22 articles published relating to validity evidence for the EM SLOE. There is evidence for content validity; however, there is a lack of evidence for internal structure, relation to other variables, and consequences. Additionally, the literature regarding response process demonstrates evidence against validity.

CONCLUSIONS

Overall, there is little published evidence in support of validity for the EM SLOE. Stakeholders need to consider changing the ranking system, improving standardization of clerkships, and further studying relation to other variables to improve validity. This will be important across GME as more specialties adopt a standardized letter.

The standardized letter of evaluation in emergency medicine: Are the qualifications useful?

OBJECTIVES

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is a widely used metric for determining interview invitations and ranking of candidates. Previous research has questioned the validity of certain sections of the SLOE. However, there remains a paucity of literature on the qualifications for EM section, which evaluates seven attributes of applicants. The aim of this study was to determine the correlation between the qualifications questions and grades, global assessment, and anticipated rank list position for EM applicants.

METHODS

A multi-institutional cross-sectional study was performed using SLOEs from applicants to three geographically distinct U.S. EM residency programs during the 2019-2020 application cycle. We abstracted EM rotation grade, qualifications scores, global assessment, and anticipated rank list position from the SLOEs. A Spearman correlation was calculated between each of the qualifications scores and the applicant's grades, global assessment, and anticipated rank list position in a pairwise fashion.

RESULTS

In total, 2,106 unique applicants (4,939 SLOEs) were included. Of the seven qualifications for EM questions, three were moderately to strongly correlated with global assessment and anticipated rank list position: "ability to develop and justify an appropriate differential and a cohesive treatment plan" (ρ = 0.65 and ρ = 0.63, respectively; p < 0.001), "how much guidance do you predict this applicant will need during residency?" (ρ = 0.68 and ρ = 0.68, respectively; p < 0.001), and "what is your prediction of success for the applicant?" (ρ = 0.69 and ρ = 0.69, respectively; p < 0.001). There was no strong correlation between the seven qualifications and grades.

CONCLUSIONS

There was a moderate to strong correlation between three of seven qualifications for EM questions (ability to develop and justify a differential and develop a cohesive plan, anticipated need for the amount of guidance, and prediction of success) with both global assessment and anticipated rank list position, suggesting that these qualifications may provide the most useful data to residency selection while some of the other factors may not be needed.

A novel faculty development tool for writing a letter of recommendation

Objective

Based on a national survey of program directors we developed a letter of recommendation (LOR) scoring rubric (SR) to assess LORs submitted to a pediatric residency program. The objective was to use the SR to analyze: the consistency of LOR ratings across raters and LOR components that contributed to impression of the LOR and candidate.

Methods

We graded 30 LORs submitted to a pediatric residency program that were evenly distributed based on final rank by our program. The SR contained 3 sections (letter features, phrases, and applicant abilities) and 2 questions about the quality of the LOR (LORQ) and impression of the candidate (IC) after reading the LOR on a 5-point Likert scale. Inter-rater reliability was calculated with intraclass correlation coefficients (ICC(2,1)). Pearson (r) correlations and stepwise multivariate linear regression modeling predicted LORQ and IC. Mean scores of phrases, features, and applicant abilities were analyzed with ANOVA and Bonferroni correction.

Results

Phrases (ICC(2,1) = 0.82, p<0.001)) and features (ICC(2,1) = 0.60, p<0.001)) were rated consistently, while applicant abilities were not (ICC(2,1) = 0.28, p<0.001)). For features, LORQ (R² = 0.75, p<0.001) and IC (R² = 0.58, p<0.001) were best predicated by: writing about candidates’ abilities, strength of recommendation, and depth of interaction with the applicant. For abilities, LORQ (R² = 0.47, p<0.001) and IC (R² = 0.51, p<0.001) were best predicted by: clinical reasoning, leadership, and communication skills (0.2). There were significant differences for phrases and features (p<0.05).

Conclusions

The SR was consistent across raters and correlates with impression of LORQ and IC. This rubric has potential as a faculty development tool for writing LORS.

Best Practices in Letters of Recommendation for General Surgery Residency: Results of Expert Stakeholder Focus Groups

OBJECTIVE

Performance inflation is rampant in applications to general surgery residency. The medical student performance evaluation, transcript, and letters of recommendation (LOR) have all been shown significantly biased in the applicants' favor. This study sought to determine best practices for LOR to improve transparency and alignment of applicant and program characteristics.

DESIGN

Two 1-hour focus groups were conducted using semi-structured interviews. Participants were asked to discuss the value and role of LOR characteristics, including standardized LOR, and provide recommendations for best practices. The transcribed discussions were coded by two educators using grounded theory and an inductive approach utilizing NVivo 12. Codes were then reviewed and revised to achieve consensus and recommendations.

SETTING

Focus groups were held during the annual Surgical Education Week meeting in April 2019.

PARTICIPANTS

General Surgery Program Directors from 10 institutions and Surgery Clerkship Directors from 11 other medical schools participated, with each group meeting independently from the other.

RESULTS

Individually, 18 codes were identified by the authors, with consensus agreed on ten. These were grouped into 4 themes: author factors, letter content, bias, and standardized letters. Overall, a checkbox and short-answer standardized LOR was not recommended, favoring a template of items to include and exclude. Ideal letter writers were felt to be surgeons who best know the applicant, and the Chair's letter, when they have no working knowledge of the applicant, was perceived to add little value. Use of specific examples to demonstrate applicant characteristics were favored, and descriptors for coded language should be included to aid in interpretation.

CONCLUSION

The focus groups identified best practices to guide writing LOR in support of applicants to general surgery residency. A template of content is provided to improve the efficiency, transparency, and accuracy of these letters for the benefit of students, medical schools, and residency programs.

Evaluating the Standardized Letter of Recommendation Form in Applicants to Orthopaedic Surgery Residency

INTRODUCTION

A standardized letter of recommendation (SLOR) form for orthopaedic surgery residency programs has recently been adopted for use, but it has not been scientifically evaluated. The purpose of this study is to investigate the usefulness of the SLOR form in the selection process.

METHODS

All SLOR forms submitted to our institution over a single application cycle were extracted and analyzed. The United States Medical Licensing Examination Step 1 scores, grades in clinical rotations, Alpha Omega Alpha status, and the number of publications were recorded for each applicant. Correlations were calculated with Spearman rho, and inter-rater reliability was evaluated by calculating intraclass correlation coefficients.

RESULTS

One thousand one hundred thirty-seven SLOR forms were analyzed for 513 applicants. There was substantial rank inflation with the SLOR form; the majority (92%) of applicants were rated as either ranked to match or in the top one-third of their rank list. Objective applicant factors such as grades and Step 1 scores demonstrated a very weak to nonexistent correlation with the summative rank (rho 0.07 to 0.13, P ≤ 0.012). Poor inter-rater reliability was observed with the intraclass correlation coefficient ranging from 0.22 to 0.33 (P < 0.001).

CONCLUSIONS

The usefulness of the SLOR form is limited by the very high ratings observed for all questions, and in particular, the final summative rank. Measures to reduce rank inflation must be implemented to improve the discriminant ability of the SLOR form, and if this cannot be accomplished, perhaps the form should be abandoned.

LEVEL OF EVIDENCE

Level III Retrospective.

Impact of PhORCAS references on overall application score for postgraduate year 1 pharmacy residency candidates

Purpose

The disparity between the number of applicants for postgraduate year 1 (PGY1) pharmacy residency positions and the number of available residency positions increases the need to optimize how applicants are evaluated. The purpose of the study described here was to evaluate the correlation of ratings of residency candidate characteristics by academic and professional references listed on residency applications with overall application score, applicant ranking, and the likelihood of candidates receiving an invitation to interview.

Methods

A multicenter, retrospective study was conducted to evaluate the correlation of reference writers’ ratings of 13 candidate characteristics and their overall recommendations with program-determined outcomes (eg, final application score, applicant ranking, and invitation to interview) through analysis of PGY1 applications submitted through the Pharmacy Online Residency Centralized Application System (PhORCAS) from 2015 through 2018. Keywords and themes within the open-ended section of letters of reference were also analyzed for correlation with overall application score.

Results

A total of 5,923 references listed on 1,867 applications to 4 PGY1 pharmacy residency programs processed by PhORCAS were included in the analysis. For the majority of applicant characteristic ratings (ie, 74% of 56,872 ratings overall), reference writers rated candidates as exceeding expectations, and applicants were “highly recommended” by these evaluators in 91% of cases. References’ average characteristic ratings and overall recommendations were poorly correlated with final application score (R2 = 0.12 [P &lt; 0.0001] and R2 = 0.08 [P &lt; 0.0001], respectively), final ranking (R2 = 0.02 [P &lt; 0.0001] and R2 = 0.03 [P &lt; 0.0001], respectively), and invitation to interview (R2 = 0.07 [P &lt; 0.0001] and R2 = 0.04 [P &lt; 0.0001], respectively). For the themes evaluated, references’ use of teaching words best correlated with normalized final application score, although the correlation was poor (R2 = 0.007, P = 0.0001).

Conclusion

Reference writers’ ratings of PGY1 residency candidate characteristics in PhORCAS are poorly correlated with application score, applicant ranking, and invitation to interview. The results of this study suggest that the existing PhORCAS standardized form for submitting references is of limited utility in its current state.

The Emergency Medicine Group Standardized Letter of Evaluation as a Workplace-based Assessment: The Validity Is in the Detail

Introduction

Interest is growing in specialty-specific assessments of student candidates based on clinical clerkship performance to assist in the selection process for postgraduate training. The most established and extensively used is the emergency medicine (EM) Standardized Letter of Evaluation (SLOE), serving as a substitute for the letter of recommendation. Typically developed by a program’s leadership, the group SLOE strives to provide a unified institutional perspective on performance. The group SLOE lacks guidelines to direct its development raising questions regarding the assessments, processes, and standardization programs employ. This study surveys EM programs to gather validity evidence regarding the inputs and processes involved in developing group SLOEs.

Methods

A structured telephone interview was administered to assess the input data and processes employed by United States EM programs when generating group SLOEs.

Results

With 156/178 (87.6%) of Accreditation Council of Graduate Medical Education-approved programs responding, 146 (93.6%) reported developing group SLOEs. Issues identified in development include the following: (1) 84.9% (124/146) of programs limit the consensus process by not employing rigorous methodology; (2) several stakeholder groups (nurses, patients) do not participate in candidate assessment placing final decisions at risk for construct under-representation; and (3) clinical shift assessments don’t reflect the task-specific expertise of each stakeholder group nor has the validity of each been assessed.

Conclusion

Success of the group SLOE in its role as a summative workplace-based assessment is dependent upon valid input data and appropriate processes. This study of current program practices provides specific recommendations that would strengthen the validity arguments for the group SLOE.

Selection of Neurological Surgery Applicants and the Value of Standardized Letters of Evaluation: A Survey of United States Program Directors

BACKGROUND

The letter of recommendation (LOR) represents a nonstandardized way to evaluate residency candidates. The goal of this project was to assess the current components of the Electronic Residency Application Service application and to determine and develop support for a standardized letter of evaluation (SLOE) in the resident selection process.

METHODS

A 16-question survey was sent to US neurosurgery program directors. In addition to demographic information, respondents were asked to rank 7 aspects of the current application (1-7), evaluate the inclusion of specific standardized questions about applicants (yes or no), note their agreement with statements about LORs (on a 5-point Likert scale), and provide any additional comments.

RESULTS

Fifty-three of 113 program directors (47%) completed the survey. The interview (average rank, 2.0 ± 1.4), United States Medical Licensing Exam step 1 score (2.86 ± 1.4), and LOR (2.96 ± 1.5) were ranked as the most important aspects of the application. Agreement was high for items regarding the utility of the current LOR (51%-78% agreement). Almost two-thirds (65%) of program directors agreed that implementing a standardized LOR would improve the resident selection process. Inclusion of questions regarding applicants' work ethic, teamwork, communication, professionalism, and initiative were strongly supported (>80% in favor), whereas including a question on theoretical rank position was mixed (54%).

CONCLUSIONS

Most neurosurgical program directors agree that increasing the objectivity of the application would be beneficial, including the addition of standardized questions. However, there is only moderate interest in implementing an SLOE.

Lower-Third SLOE Rankings Impede, But Do Not Prevent, A Match in Emergency Medicine Residency Training

OBJECTIVE

Emergency medicine program directors (PD) value the standardized letter of evaluation (SLOE) as the most important aspect of a residency application when making both invitation and ranking decisions. This study aims to determine whether the presence of any lower-third in either SLOE global assessment (GA) question impacted the ability of an applicant to match into EM. We hypothesized that any lower-third ranking would be associated with increased odds of not matching into EM.

METHODS

We conducted a retrospective cohort study evaluating allopathic applicants from medical schools in the United States (US allopathic applicants) to a single EM residency program during the 2018/2019 match cycles. GA SLOE rankings from all applications were tabulated and compared to the applicant's National Resident Matching Program (NRMP) match outcome. Comparative analyses were conducted between SLOE groupings and odds ratios (OR) were calculated.

RESULTS

A total of 2,017 SLOEs from 781 US allopathic applicants were analyzed during the study period. Of the total, 277 (35%) applicants in our sample had any lower-third GA ranking, which significantly decreased an applicant's odds of matching in EM by 79% (OR 0.21, 95% CI, 0.12-0.34). Having more than one lower-third GA ranking did not further statistically decrease the odds of a successful EM match (OR 0.60, 95% CI 0.31-1.17). As a secondary finding of the study, results demonstrate that those applicants having no lower-third GA rankings had a nearly 5 times increased odds of an EM match (OR 4.84, 95% CI, 2.91-8.03).

CONCLUSION

Having any lower-third GA ranking significantly reduced an applicant's chances of matching into an EM program. Faculty advisors should be aware of the increased risk of not matching for any applicant with any lower-third GA ranking and advise students appropriately, while maintaining the integrity of the SLOE and not divulging the confidential information contained within.

Does Applicant Gender Have an Effect on Standardized Letters of Evaluation Obtained During Medical Student Emergency Medicine Rotations?

BACKGROUND

The standardized letter of evaluation (SLOE) in emergency medicine (EM) is one of the most important items in a student's application to EM residency and replaces narrative letters of recommendation. The SLOE ranks students into quantile categories in comparison to their peers for overall performance during an EM clerkship and for their expected rank list position. Gender differences exist in several assessment methods in undergraduate and graduate medical education. No authors have recently studied whether there are differences in the global assessment of men and women on the SLOE.

OBJECTIVES

The objective of this study was to determine if there is an effect of student gender on the outcome of a SLOE.

METHODS

This was a retrospective observational study examining SLOEs from applications to a large urban, academic EM residency program from 2015 to 2016. Composite scores (CSs), comparative rank scores (CRSs), and rank list position scores (RLPSs) on the SLOE were compared for female and male applicants using Mann-Whitney U-test.

RESULTS

From a total 1,408 applications, 1,038 applicants met inclusion criteria (74%). We analyzed 2,092 SLOEs from these applications. Female applicants were found to have slightly lower and thus better CRSs, RLPSs, and CSs than men. The mean CRS for women was 2.27 and 2.45 for men (p < 0.001); RLPS for women was 2.32 and 2.52 for men (p < 0.001) and CS was 4.59 for women and 4.97 for men (p < 0.001).

CONCLUSIONS

Female applicants have somewhat better performance on the EM SLOE than their male counterparts.

A Retrospective Cohort Study of the Effect of Home Institution on Emergency Medicine Standardized Letters of Evaluation

BACKGROUND

The standardized letter of evaluation (SLOE) was developed to make letters of recommendation in emergency medicine (EM) more objective and discerning. Typically, students obtain one SLOE from a home EM rotation and at least one more SLOE from an away clerkship. It is unclear if students perform better on their home or away EM rotations.

OBJECTIVE

The purpose of this study was to determine if students perform better on the group SLOE at their home institution compared to an away institution.

METHODS

The authors performed a retrospective application review of all allopathic applicants to an urban, academic EM residency program. The authors calculated a composite score (CS) for each group SLOE, using the global assessment scores for comparative rank and rank list position. A lower CS indicates better performance. The authors compared mean CS for students' first home rotations with first away rotations. For students in the study who had a third (second away or second home site) SLOE available, the authors compared mean CS on the students' first SLOEs with mean CS on the students' third SLOEs.

RESULTS

A total of 624 records were included in the primary analysis. There was a small, but significant difference between mean CS for students' home rotations when compared to away rotations (4.67 vs. 4.85, p = 0.024). Students performed better on their home rotations. Students who had three SLOEs available performed worse on their third rotation (first = 4.40, second = 4.63, third = 4.77, p = 0.012 for first vs. third). For all available SLOEs, more than 50% of students fell into the top 10% or top one-third categories.

CONCLUSION

Students perform slightly better on their home EM rotations. Students' mean SLOE CS is slightly worse for a third rotation when compared to a first rotation.

The AAMC Standardized Video Interview and the Electronic Standardized Letter of Evaluation in Emergency Medicine: A Comparison of Performance Characteristics

PURPOSE

To compare the performance characteristics of the electronic Standardized Letter of Evaluation (eSLOE), a widely used structured assessment of emergency medicine (EM) residency applicants, and the Association of American Medical Colleges (AAMC) Standardized Video Interview (SVI), a new tool designed by the AAMC to assess interpersonal and communication skills and professionalism knowledge.

METHOD

The authors matched EM residency applicants with valid SVI total scores and completed eSLOEs in the 2018 Match application cycle. They examined correlations and group differences for both tools, United States Medical Licensing Examination (USMLE) Step exam scores, and honor society memberships.

RESULTS

The matched sample included 2,884 applicants. SVI score and eSLOE global assessment ratings demonstrated small positive correlations approaching r = 0.20. eSLOE ratings had higher correlations with measures of academic ability (USMLE scores, academic honor society membership) than did SVI scores. Group differences were minimal for the SVI, with scores slightly favoring women (d = -0.21) and US-MD applicants (d = 0.23-0.42). Group differences in eSLOE ratings were small, favoring women over men (approaching d = -0.20) and white applicants over black applicants (approaching d = 0.40).

CONCLUSIONS

Small positive correlations between SVI score and eSLOE global assessment ratings, alongside varying correlations with academic ability indicators, suggest these are complementary tools. Findings suggest the eSLOE is subject to similar sources and degrees of bias as other common assessments; these group differences were not observed with the SVI. Further examination of both tools is necessary to understand their ability to predict clinical performance.

How Well Does the Standardized Video Interview Score Correlate with Traditional Interview Performance?

Introduction

In 2017, all medical students applying for residency in emergency medicine (EM) were required to participate in the Standardized Video Interview (SVI). The SVI is a video-recorded, uni-directional interview consisting of six questions designed to assess interpersonal and communication skills and professionalism. It is unclear whether this simulated interview is an accurate representation of an applicant’s competencies that are often evaluated during the in-person interview.

Objective

The goal of this study was to determine whether the SVI score correlates with a traditional in-person interview score.

Methods

Six geographically and demographically diverse EM residency programs accredited by the Accreditation Council for Graduate Medical Education participated in this prospective observational study. Common demographic data for each applicant were obtained through an Electronic Residency Application Service export function prior to the start of any scheduled traditional interviews (TI). On each TI day, one interviewer blinded to all applicant data, including SVI score, rated the applicant on a five-point scale. A convenience sample of applicants was enrolled based on random assignment to the blinded interviewer. We studied the correlation between SVI score and TI score.

Results

We included 321 unique applicants in the final analysis. Linear regression analysis of the SVI score against the TI score demonstrated a small positive linear correlation with an r coefficient of +0.13 (p=0.02). This correlation remained across all SVI score subgroups (p = 0.03).

Conclusion

Our study suggests that there is a small positive linear correlation between the SVI score and performance during the TI.

Emergency Medicine Standardized Letter of Evaluation (SLOE): Findings From the New Electronic SLOE Format

BACKGROUND

Emergency medicine (EM) uses a standardized template for residency application letters of recommendations. In 1997, the Standardized Letter of Recommendation was developed with categories for applicant comparison. Now named the Standardized Letter of Evaluation (SLOE), it is a universal requirement in the EM residency application process. In 2016, a website called "eSLOE" was launched for credentialed academic EM faculty to enter applicant data, which generates a SLOE.

OBJECTIVE

This article outlines website creation for the eSLOE and its successful national implementation in the 2016-2017 EM residency application cycle. We analyzed current trends in applicant assessments from the eSLOE data and compared them to prior data.

METHODS

Data from 2016-2017 were sorted and analyzed for each question on the eSLOE. An analysis of Global Assessment and Qualifications for EM rankings, clerkship grade, and comparison with prior SLOE data was performed.

RESULTS

Analysis of 6715 eSLOEs for 3138 unique applicants revealed the following Global Assessment rankings: top 10%, top one-third, middle one-third, and lower one-third. There was less spread with the distribution for clerkship grade and Qualifications for EM. The 2011-2012 standard letter of recommendation global assessment data, with top 10%, top one-third, middle one-third, and lower one-third, also revealed top-clustered results with less spread compared with the ranking usage in 2016-2017.

CONCLUSIONS

Results indicate an improved spread of all rank categories for Global Assessment, enhancing the eSLOE's applicant discrimination. There has been an overall improvement in rank designation when compared with previously published data.

Standardized Video Interviews Do Not Correlate to United States Medical Licensing Examination Step 1 and Step 2 Scores

Introduction

In 2017, the Standardized Video Interview (SVI) was required for applicants to emergency medicine (EM). The SVI contains six questions highlighting professionalism and interpersonal communication skills. The responses were scored (6–30). As it is a new metric, no information is available on correlation between SVI scores and other application data. This study was to determine if a correlation exists between applicants’ United States Medical Licensing Examination (USMLE) and SVI scores. We hypothesized that numeric USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores would not correlate with the SVI score, but that performance on the Step 2 Clinical Skills (CS) portion may correlate with the SVI since both test communication skills.

Methods

Nine EM residency sites participated in the study with data exported from an Electronic Residency Application Service (ERAS®) report. All applicants with both SVI and USMLE scores were included. We studied the correlation between SVI scores and USMLE scores. Predetermined subgroup analysis was performed based on applicants’ USMLE Step 1 and Step 2 CK scores as follows: (≥ 200, 201–220, 221–240, 241–260, >260). We used linear regression, the Kruskal-Wallis test and Mann-Whitney U test for statistical analyses.

Results

1,325 applicants had both Step 1 and SVI scores available, with no correlation between the overall scores (p=0.58) and no correlation between the scores across all Step 1 score ranges, (p=0.29). Both Step 2 CK and SVI scores were available for 1,275 applicants, with no correlation between the overall scores (p=0.56) and no correlation across all ranges, (p=0.10). The USMLE Step 2 CS and SVI scores were available for 1,000 applicants. Four applicants failed the CS test without any correlation to the SVI score (p=0.08).

Conclusion

We found no correlation between the scores on any portion of the USMLE and the SVI; therefore, the SVI provides new information to application screeners.

Dear Program Director: Deciphering Letters of Recommendation

BACKGROUND

Letters of recommendation (LORs) are an important part of applications for residency and fellowship programs. Despite anecdotal use of a "code" in LORs, research on program director (PD) perceptions of the value of these documents is sparse.

OBJECTIVE

We analyzed PD interpretations of LOR components and discriminated between perceived levels of applicant recommendations.

METHODS

We conducted a cross-sectional, descriptive study of pediatrics residency and fellowship PDs. We developed a survey asking PDs to rate 3 aspects of LORs: 13 letter features, 10 applicant abilities, and 11 commonly used phrases, using a 5-point Likert scale. The 11 phrases were grouped using principal component analysis. Mean scores of components were analyzed with repeated-measures analysis of variance. Median Likert score differences between groups were analyzed with Mann-Whitney U tests.

RESULTS

Our survey had a 43% response rate (468 of 1079). "I give my highest recommendation" was rated the most positive phrase, while "showed improvement" was rated the most negative. Principal component analysis generated 3 groups of phrases with moderate to strong correlation with each other. The mean Likert score for each group from the PD rating was calculated. Positive phrases had a mean (SD) of 4.4 (0.4), neutral phrases 3.4 (0.5), and negative phrases 2.6 (0.6). There was a significant difference among all 3 pairs of mean scores (all P < .001).

CONCLUSIONS

Commonly used phrases in LORs were interpreted consistently by PDs and influenced their impressions of candidates. Key elements of LORs include distinct phrases depicting different degrees of endorsement.

Relationship Between Institutional Standardized Letter of Evaluation Global Assessment Ranking Practices, Interviewing Practices, and Medical Student Outcomes

BACKGROUND

Emergency medicine (EM) program directors rely largely on the standardized letter of evaluation (SLOE) to help determine which applicants to interview in the face of an increasing number of applications. To further characterize the SLOE's role in the EM application process, particularly the global assessment (GA) ranking and its effect on interviewing practices and medical student outcomes, the leaders of EM programs were surveyed regarding their experiences in both generating and utilizing the SLOE.

METHODS

Individuals on the Council of Emergency Medicine Residency Directors (CORD) and Clerkship Directors in Emergency Medicine (CDEM) Academy listservs were anonymously surveyed from March 21-30, 2015, with 18 questions in multiple-choice and fill-in-the-blank formats.

RESULTS

There were 99 respondents. Only 39 respondents (39%) reported adhering strictly to SLOE guidelines by evenly placing their students into thirds (top, middle, lower) on the SLOE GA. Most respondents interviewed individuals ranked in the lower third. Programs adhering strictly to ranking guidelines were more likely to interview students in the lower third than those adhering loosely or not at all. There was no relationship between a program's self-reported adherence to the SLOE ranking guidelines and the number of unmatched students in EM during the 2014 and 2015 academic years.

CONCLUSION

Many SLOE writers do not strictly adhere to CORD's SLOE writing guidelines when using the GA ranking, due to the fear of adversely impacting an applicant's ability to successfully match into EM. This calls into question the validity of the SLOE as it is currently used. However, this study suggests that adhering to recommended SLOE ranking guidelines is unlikely to substantially increase the risk that students will fail to match. If more evaluators were to adhere to the guidelines, the SLOE could become the valid evaluation instrument that graduate medical education has long been pursuing.

Gender Differences in Language of Standardized Letter of Evaluation Narratives for Emergency Medicine Residency Applicants

OBJECTIVE

While gender differences in language for letters of recommendation have been identified in other fields, no prior studies have evaluated the narrative portion of the emergency medicine (EM) standardized letter of evaluation (SLOE). We aim to examine the differences in language used to describe male and female applicants within the SLOE narrative.

METHODS

Invited applicants to a 4-year academic EM residency program within a single application year with a SLOE were included in the sample. Exclusion criteria were SLOE of applicants from non-Liaison Committee on Medical Education (LCME) schools or first rotation SLOE not available for download. Data were collected on applicant gender, age, rotation grade, Alpha Omega Alpha designation, and medical school rank. The previously validated Linguistic Inquiry and Word Count (LIWC) program was used to analyze frequency of words within categories relevant to letters of recommendation. Descriptive statistics, t-tests, and chi-square tests were employed in analysis.

RESULTS

Of 1,025 applicants within a single application year, 265 were invited to interview; 237 applicants had a first rotation SLOE available for analysis. There were no differences between male and female applicants for baseline characteristics. The median word count per SLOE narrative was 199; within the LIWC dictionary and user-defined categories, words within the categories of affiliation and ability appeared more frequently for female applicants.

CONCLUSIONS

Our results with respect to the SLOE narrative reinforce prior research that letters of recommendation for female applicants highlight communal characteristics of teamwork, helpfulness, and compassion. Contrary to prior research, ability words highlighting intelligence and skill appeared with greater frequency for female applicants. No pervasive differences were found in other word categories. In this sample, the standardized format of the SLOE resulted in letters that were relatively free of gender bias.

Letters of Recommendation: Association with Interviewers' Perceptions and Preferences

Objective Letters of recommendation (LORs) are an inescapable aspect of the application process. Standardized LORs (SLORs) have been developed and compared with traditional narrative LORs (NLORs). This study investigated whether there was a difference in degree of association between LOR types and face-to-face interviews. Interviewer preference for LOR was anonymously surveyed. Study Design Survey. Setting Single-institution otolaryngology residency program, 1 year before and 4 years after introduction of SLORs. Subjects and Methods Residency interviewers indicated on a visual analog scale how well their impression of an applicant compared between the LOR and the face-to-face interview. Interviewers assessed each applicant, each year, based on LOR type. Results Of 2573 assessments, 964 were collected (37.5% response rate), including 927 responses related to NLORs, 561 to SLORs, and 316 to medical student performance evaluations (ie, dean's letters). The average association of VAS scores between LORs and interviews ranged from 72 to 81 across years and LOR types. Sixty-one percent of interviewers preferred NLORs, and 13% preferred SLORs. Reasons for these preferences included more information provided in NLORs versus faster read time with SLORs. Conclusions Interviewers' perceptions of applicants based on LORs and face-to-face interviews were comparable across LOR types over the last 5 years. For many reasons, the general utility of the LOR remains questionable, and the continued need for it should be critically assessed.

The Standardized Letter of Evaluation for Postgraduate Training: A Concept Whose Time Has Come?

A medical student's letter of recommendation for postgraduate training applications should provide a fair and accurate assessment of academic and clinical performance, as well as define character attributes pertinent to the practice of medicine. Since its inception in 1997, the emergency medicine (EM) standardized letter of evaluation (SLOE) has evolved into an instrument that provides just such an assessment. Concise, standardized, and discriminating in its assessment of performance relevant to the practice of EM, the SLOE is judged by program directors in EM as the most valuable component of a potential resident's application. Other specialties would benefit from such a specialty-specific perspective, which is currently lacking in most Electronic Residency Application Service application materials. Creation of specialty-specific SLOEs which define performance metrics or competencies and noncognitive personality traits critical to each unique specialty would add substantially to the holistic review of our graduating students. As a result, specialty-specific SLOEs would increase the likelihood that programs could effectively identify applicants who would not only be a "good fit" for their programs but also graduate to become successful physicians.

Variability of Ratings in the Otolaryngology Standardized Letter of Recommendation

Objective

To determine the variability of ratings given to students on the otolaryngology standardized letter of recommendation (SLOR).

Study Design

Retrospective review.

Setting

Academic otolaryngology training program.

Subjects and Methods

496 SLORs to the Cleveland Clinic Otolaryngology Training Program were reviewed. The SLORs were extracted from the applications and analyzed. The distributions of ratings across the 10 categories were statistically analyzed including distribution and standard deviation. Rankings were grouped into deciles for this analysis.

Results

Ratings across the 10 domains revealed clustering of results across the top 2 deciles. The distribution of the bell-shaped curve was shifted significantly to the left, representing the upper deciles. No evidence of gender or geographic bias was found. Longer length of time of association between the applicant and the letter writer correlated to a higher ranking.

Conclusion

The explosion of applications being sent out by candidates for otolaryngology residency programs has prompted the implementation of the SLOR. The lack of variation in the ratings across the 10 domains does not allow for differentiation among student applicants. Reliance on the narrative letter of recommendation attached to the SLOR still remains the most significant way to differentiate among applicants. Refinements will need to be made in either the structure or use of the SLOR for it to be a more useful tool.