Program Directors' Perspectives on Residency Applications in the Post‒United States Medical Licensing Examination Step 1 Era: A Case for Standardized Letters of Recommendation?

INTRODUCTION

The United States Medical Licensing Examination (USMLE) Step 1 test evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements toward pass/fail systems for clerkship grading and trends toward abandonment of class ranking, assessing residency applications has become increasingly challenging.

METHODS

The Society of Academic Urologists convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology PDs in the US. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PDs.

RESULTS

There was a 60.5% response rate (89/147). Letters of recommendation (LORs) were ranked as the most important predictor, with a mean rank of 2.39, median of 2 (IQR 1-3). Clerkship grades and USMLE Step 1 were comparable and ranked second. Medical school reputation ranked the lowest. There was significant subjective heterogeneity among categories; however, this was less so for LORs, which predominated as the most important factor among application elements (P < .001).

CONCLUSIONS

To our knowledge, this is the largest sample size assessing PD perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving toward binary pass/fail systems create an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LORs to be the most important factor of residency applications, making a compelling argument for moving toward a standardized LOR to maximize this tool, mitigate bias, and improve interreviewer reliability.

Machine Learning for Predictive Analysis of Otolaryngology Residency Letters of Recommendation

INTRODUCTION

Letters of recommendation (LORs) are a highly influential yet subjective and often enigmatic aspect of the residency application process. This study hypothesizes that LORs do contain valuable insights into applicants and can be used to predict outcomes. This pilot study utilizes natural language processing and machine learning (ML) models using LOR text to predict interview invitations for otolaryngology residency applicants.

METHODS

A total of 1642 LORs from the 2022-2023 application cycle were retrospectively retrieved from a single institution. LORs were preprocessed and vectorized using three different techniques to represent the text in a way that an ML model can understand written prose: CountVectorizer (CV), Term Frequency-Inverse Document Frequency (TF-IDF), and Word2Vec (WV). Then, the LORs were trained and tested on five ML models: Logistic Regression (LR), Naive Bayes (NB), Decision Tree (DT), Random Forest (RF), and Support Vector Machine (SVM).

RESULTS

Of the 337 applicants, 67 were interviewed and 270 were not interviewed. In total, 1642 LORs (26.7% interviewed) were analyzed. The two best-performing ML models in predicting interview invitations were the TF-IDF vectorized DT and CV vectorized DT models.

CONCLUSION

This preliminary study revealed that ML models and vectorization combinations can provide better-than-chance predictions for interview invitations for otolaryngology residency applicants. The high-performing ML models were able to classify meaningful information from the LORs to predict applicant interview invitation. The potential of an automated process to help predict an applicant's likelihood of obtaining an interview invitation could be a valuable tool for training programs in the future.

LEVEL OF EVIDENCE

N/A Laryngoscope, 2024.

Writing effective letters of recommendation in physiology

A letter of recommendation is a statement of support for a person that has been requested by some individual or organization. In physiology, the purpose of the letter may be to support admission to an academic program, funding of a fellowship grant proposal, consideration for a trainee position in a research laboratory, an award from a professional society, or an application for a job. The goal of the letter should be to provide personalized insight into the suitability of the candidate for the position or award that cannot be easily obtained from other materials in an application or nomination process. Despite the importance of writing effective letters of recommendation, most physiologists receive no formal training in this requisite professional skill. In this Perspective, I first discuss the responsibilities and challenges of writing letters of recommendation, for whom you should consider writing a letter, the pros and cons of asking the candidate to create an initial draft, and the information required for you to write a letter. I then describe a helpful structure to follow when writing a letter of recommendation, including the opening paragraph (introduction), main body, and summary sections. Next, I share 10 insider tips for writing effective letters of recommendation. I complete the commentary by discussing special circumstances, including writing letters for solid but not highly ranking candidates and acting as a "substitute" for a primary mentor. It is hoped that this perspective will provide guidance for early career physiologists in this essential skill of the profession.NEW & NOTEWORTHY This Perspective provides essential background, a step-by-step guide, and key insider tips for writing an effective letter of recommendation aimed at helping early career physiologists with this important professional task.

What's in a Name? Experimental Evidence of Gender Bias in Recommendation Letters Generated by ChatGPT

BACKGROUND

Artificial intelligence chatbots such as ChatGPT (OpenAI) have garnered excitement about their potential for delegating writing tasks ordinarily performed by humans. Many of these tasks (eg, writing recommendation letters) have social and professional ramifications, making the potential social biases in ChatGPT's underlying language model a serious concern.

OBJECTIVE

Three preregistered studies used the text analysis program Linguistic Inquiry and Word Count to investigate gender bias in recommendation letters written by ChatGPT in human-use sessions (N=1400 total letters).

METHODS

We conducted analyses using 22 existing Linguistic Inquiry and Word Count dictionaries, as well as 6 newly created dictionaries based on systematic reviews of gender bias in recommendation letters, to compare recommendation letters generated for the 200 most historically popular "male" and "female" names in the United States. Study 1 used 3 different letter-writing prompts intended to accentuate professional accomplishments associated with male stereotypes, female stereotypes, or neither. Study 2 examined whether lengthening each of the 3 prompts while holding the between-prompt word count constant modified the extent of bias. Study 3 examined the variability within letters generated for the same name and prompts. We hypothesized that when prompted with gender-stereotyped professional accomplishments, ChatGPT would evidence gender-based language differences replicating those found in systematic reviews of human-written recommendation letters (eg, more affiliative, social, and communal language for female names; more agentic and skill-based language for male names).

RESULTS

Significant differences in language between letters generated for female versus male names were observed across all prompts, including the prompt hypothesized to be neutral, and across nearly all language categories tested. Historically female names received significantly more social referents (5/6, 83% of prompts), communal or doubt-raising language (4/6, 67% of prompts), personal pronouns (4/6, 67% of prompts), and clout language (5/6, 83% of prompts). Contradicting the study hypotheses, some gender differences (eg, achievement language and agentic language) were significant in both the hypothesized and nonhypothesized directions, depending on the prompt. Heteroscedasticity between male and female names was observed in multiple linguistic categories, with greater variance for historically female names than for historically male names.

CONCLUSIONS

ChatGPT reproduces many gender-based language biases that have been reliably identified in investigations of human-written reference letters, although these differences vary across prompts and language categories. Caution should be taken when using ChatGPT for tasks that have social consequences, such as reference letter writing. The methods developed in this study may be useful for ongoing bias testing among progressive generations of chatbots across a range of real-world scenarios.

TRIAL REGISTRATION

OSF Registries osf.io/ztv96; https://osf.io/ztv96.

Do certain phrases in letters of recommendation predict success in fellowship performance?

BACKGROUND

Letters of recommendation (LORs) are a valued, yet imperfect tool. Program directors (PDs) score phrases such as give my highest recommendation and top 5 to 10% of students as positive. Although positive phrases are valued by PDs, there is no evidence that these phrases predict performance. We attempt to identify whether 12 specific phrases found in letters of recommendation predict future performance of fellows.

METHODS

LORs were evaluated for 12 select phrases and statements. Alpha Omega Alpha (AOA) status, Step 2 Clinical Knowledge (CK) score, and whether the letter writer was personally known to our admission's committee were also categorized. Logistic regressions were performed to evaluate the relationship of the independent variables with fellow performance.

RESULTS

Using multivariate logistic regression, one of the best residents (OR = 4.02, 95% CI (1.0, 15.9), p < 0.05), exceeds expectations (OR = 4.74, 95% CI (1.4, 16.3), p = 0.01), and give my highest recommendation (OR = 3.87, 95% CI (1.3, 11.7), p = 0.02) predicted positive performance. Highly recommend (OR = 0.31, 95% CI (0.1, 1.0), p < 0.05) and top 5 to 10% (OR = 0.05, 95% CI (0.0, 0.6), p = 0.02) predicted negative performance. The remaining phrases did not correlate to fellowship performance.

CONCLUSION

The current LOR evaluation process may place undo importance on phrases that have limited bearing on a candidate's success in training. Training both letter readers and writers to avoid using coded language or avoid assigning improper importance to select phrases may help improve the candidate selection process.

Do gender differences exist in letters of recommendation for reproductive endocrinology and infertility fellowship?

OBJECTIVE(S)

To determine if gender differences exist in letters of recommendation (LORs) for reproductive endocrinology and infertility fellowship.

DESIGN

Cohort study.

SETTING

Academic medical center.

PATIENTS

A cohort of applicants to a reproductive endocrinology and infertility fellowship program at a single Midwestern academic institution in a single year.

INTERVENTIONS

None.

MAIN OUTCOME MEASURES

Quantitative measures included linguistics inquiry and word count analysis, with 4 summary variables and 25 word categories, to quantify tone and themes present by gender. Performing qualitative analysis in parallel to linguistic analysis allowed for the exploration of themes not conveyed in quantitative methods alone. Qualitative measures included the frequency of code themes in LOR by applicant gender.

RESULTS

There were 272 letters from 72 applicants, 54 (76%) of which were women and 17 (24%) were men. One applicant was excluded because gender information was not specified; 269 letters were included in the quantitative linguistics and qualitative coding analysis. One hundred 10 letters (41%) were written by women and 159 (59%) by men. LOR written for men had a higher mean word count than those written for women (537 vs. 474, P=.04). Linguistics Inquiry and Word Count analysis exhibited more risk words used to describe men applicants (P=.01). When comparing word categories by letter writer gender, women letter writers more frequently used communal, affect, and home word categories whereas men writers more frequently used affiliation-related words. Qualitative analysis revealed that letters for men applicants described leadership, altruism, rapport with patients, and strong endorsements more frequently, whereas women applicants' letters more often mentioned doubt raisers and disclosures of personal life. In addition, letters for women applicants more often described candidates as drama free, not easy to "fluster," and not complainers. Furthermore, letters for women more frequently disclosed parenthood status.

CONCLUSIONS

Gender differences exist among both letter writers and applicants. Qualitative analysis revealed that women applicants were more likely to be described with doubt raisers and personal life disclosures, whereas men were more likely to be described by their altruism and with a strong endorsement. Increasing bias awareness and implementing mitigation strategies would improve gender equity in LOR.

Words Used in Letters of Recommendation for Pediatric Residency Applicants: Demographic Differences and Impact on Interviews

OBJECTIVE

To describe differences in agentic (achievement) and communal (relationship) terms in letters of recommendation (LORs) for pediatric residency candidates by applicant and letter writer demographics and to examine if LOR language is associated with interview status.

METHODS

A random sample of applicant profiles and LORs submitted to one institution were analyzed from the 2020-21 Match. Letters of recommendation text was inputted into a customized natural language processing application which determined the frequency of agentic and communal words in each LOR. Neutral LORs were defined as having< 5% surplus of agentic or communal terms.

RESULTS

We analyzed 2094 LORs from 573 applicants: 78% were women, 24% were under-represented in medicine (URiM), and 39% were invited to interview. Most letter writers were women (55%) and of senior academic rank (49%). Overall, 53% of LORs were agency biased, 25% communal biased, and 23% neutral. There was no difference in agency and communally biased LORs by an applicant's gender (men 53% agentic vs women 53% agentic, P = .424), race or ethnicity (non-URiM 53% agentic vs URiM 51% agentic, P = .631). Male letter writers used significantly more agentic terms (8.5%) compared to women (6.7% agentic) or writers of both genders (3.1% communal) (P = .008). Applicants invited to interview were more likely to have a neutral LOR; however, no significant association existed between language and interview status.

CONCLUSIONS

No significant differences in language were found by applicant gender or race among pediatric residency candidates. Identifying potential biases within pediatric residency selection processes is important in creating an equitable approach to application review.

Gender differences in cardiothoracic surgery letters of recommendation

OBJECTIVE

To investigate whether or not gender influences letters of recommendation for cardiothoracic surgery fellowship.

METHODS

From applications to an Accreditation Council Graduate Medical Education cardiothoracic surgery fellowship program between 2016 and 2021, applicant and author characteristics were examined with descriptive statistics, analysis of variance, and Pearson χ2 tests. Linguistic software was used to assess communication differences in letters of recommendation, stratified by author and applicant gender. An additional higher-level analysis was then performed using a generalized estimating equations model to examine linguistic differences among author-applicant gender pairs.

RESULTS

Seven hundred thirty-nine recommendation letters extracted from 196 individual applications were analyzed; 90% (n = 665) of authors were men and 55.8% (n = 412) of authors were cardiothoracic surgeons. Compared with women authors, authors who are men wrote more authentic (P = .01) and informal (P = .03) recommendation letters. When writing for women applicants, authors who are men were more likely to display their own leadership and status (P = .03) and discuss women applicants' social affiliations (P = .01), like occupation of applicant's father or husband. Women authors wrote longer letters (P = .03) and discussed applicants' work (P = .01) more often than authors who are men. They also mentioned leisure activities (P = .03) more often when writing for women applicants.

CONCLUSIONS

Our work identifies gender-specific differences in letters of recommendation. Women applicants may be disadvantaged because their recommendation letters are significantly more likely to focus on their social ties, leisure activities, and the status of the letter writer. Author and reviewer awareness of gender-biased use of language will aid in improvements to the candidate selection process.

Confidence of Faculty in Writing Letters of Recommendation for Pediatric Fellowship Applicants

BACKGROUND

The assessment of pediatric residents applying to subspecialty fellowship programs relies on faculty letters of recommendation (LOR). However, it is unclear if pediatric faculty are confident that their LOR are effective.

OBJECTIVE

This study aims to assess the confidence of pediatric faculty in writing an effective LOR for pediatric residents applying to subspecialty fellowship programs.

METHODS

Survey development was conducted using evidence-based best practices. Surveys were distributed via email in 2021 to all full-time pediatric faculty members who taught pediatric residents in a large academic medical center. Categorical values were compared by chi-square test.

RESULTS

Eighty-five out of 150 (57%) faculty members completed the survey. Forty-one percent of participants were very confident that their LOR provided adequate content to assess residents during the application process. Confidence was associated with higher academic rank (p=0.02), frequent contact with residents (p=0.01), and writing >2 LOR in the last five years (p=0.0002). Confident LOR writers were more likely to describe their own background, details about the resident's scholarly activity, and the resident's ability to work as part of a team. Thirty-five percent of respondents reported never considering gender bias when writing LOR, whereas 28% reported always considering gender bias. Eighty-seven percent of respondents reported an interest in receiving LOR writing guidelines.

CONCLUSION

Half of the faculty respondents were not very confident in their ability to write an effective LOR for pediatric residents applying for a fellowship. Faculty development and standardized instructions on writing effective LOR may be helpful both at the institutional and national levels, including the importance of considering gender bias when writing LOR.

Can AI Mitigate Bias in Writing Letters of Recommendation?

Letters of recommendation play a significant role in higher education and career progression, particularly for women and underrepresented groups in medicine and science. Already, there is evidence to suggest that written letters of recommendation contain language that expresses implicit biases, or unconscious biases, and that these biases occur for all recommenders regardless of the recommender's sex. Given that all individuals have implicit biases that may influence language use, there may be opportunities to apply contemporary technologies, such as large language models or other forms of generative artificial intelligence (AI), to augment and potentially reduce implicit biases in the written language of letters of recommendation. In this editorial, we provide a brief overview of existing literature on the manifestations of implicit bias in letters of recommendation, with a focus on academia and medical education. We then highlight potential opportunities and drawbacks of applying this emerging technology in augmenting the focused, professional task of writing letters of recommendation. We also offer best practices for integrating their use into the routine writing of letters of recommendation and conclude with our outlook for the future of generative AI applications in supporting this task.

Barriers in Ophthalmology Residency Applications for Students Identifying as Underrepresented in Medicine: A San Francisco Match Analysis

OBJECTIVE

There is a significant lack of ophthalmologists who self-identify as underrepresented in medicine (URiM) in the physician workforce. Prior literature has revealed bias in traditional metrics for selection relied on by resident programs such as United States Medical Licensing Examination (USMLE) scores, letters of recommendation (LOR), and induction into medical honors societies such as Alpha Omega Alpha (AOA). The purpose of this study was to elucidate race-based differences in word usage within ophthalmology residency letters of recommendation that may disproportionately affect URiM applicants.

DESIGN

This was a retrospective, cohort study.

SETTING

This was a multicenter study across the Wilmer Eye Institute at Johns Hopkins, the University of California San Francisco, and the University of North Carolina at Chapel Hill.

PARTICIPANTS

San Francisco (SF) Match applications submitted to three ophthalmology residency programs between 2018 and 2020 were reviewed. URiM status, USMLE Step 1 score, and AOA membership were recorded. Letters of recommendation were analyzed using text analysis software. T-tests and chi-squared or Fisher's exact tests were used to compare continuous and categorical variables, respectively. Frequency of word/summary term usage in letters of recommendation were the main outcome measures.

RESULTS

Relative to non-URiM applicants, URiM applicants had lower USMLE Step 1 scores (mean difference=7.0; p<0.001). Non-URiM letters of recommendation were more likely to describe applicants as "dependable" (p=0.009) and highlight "research" (p=0.046). URiM letters were more likely to describe applicants as "warm" (p=0.02) and "caring" (p=0.02).

CONCLUSIONS

This study identified potential barriers for URiM ophthalmology residency applicants which can help guide future interventions to increase workforce diversity.

Examining Implicit Bias Differences in Pediatric Surgical Fellowship Letters of Recommendation Using Natural Language Processing

OBJECTIVE

We analyzed the prevalence and type of bias in letters of recommendation (LOR) for pediatric surgical fellowship applications from 2016-2021 using natural language processing (NLP) at a quaternary care academic hospital.

DESIGN

Demographics were extracted from submitted applications. The Valence Aware Dictionary for sEntiment Reasoning (VADER) model was used to calculate polarity scores. The National Research Council dataset was used for emotion and intensity analysis.  The Kruskal-Wallis H-test was used to determine statistical significance.  SETTING: This study took place at a single, academic, free standing quaternary care children's hospital with an ACGME accredited pediatric surgery fellowship.

PARTICIPANTS

Applicants to a single pediatric surgery fellowship were selected for this study from 2016 to 2021. A total of 182 individual applicants were included and 701 letters of recommendation were analyzed.

RESULTS

Black applicants had the highest mean polarity (most positive), while Hispanic applicants had the lowest.  Overall differences between polarity distributions were not statistically significant.   The intensity of emotions showed that differences in "anger" were statistically significant (p=0.03).  Mean polarity was higher for applicants that successfully matched in pediatric surgery.

DISCUSSION

This study identified differences in LORs based on racial and gender demographics submitted as part of pediatric surgical fellowship applications to a single training program. The presence of bias in letters of recommendation can lead to inequities in demographics to a given program. While difficult to detect for humans, natural language processing is able to detect bias as well as differences in polarity and emotional intensity. While the types of emotions identified in this study are highly similar among race and gender groups, the intensity of these emotions revealed differences, with "anger" being most significant.

CONCLUSION

From this work, it can be concluded that bias in LORs, as reflected as differences in polarity, which is likely a result of the intensity of the emotions being used and not the types of emotions being expressed.   Natural language processing shows promise in identification of subtle areas of bias that may influence an individual's likelihood of successful matching.

Sex bias in letters of recommendation and personal statements for otolaryngology residency

Objective

Application for otolaryngology residency is highly competitive, with letters of recommendation (LORs) and applicant personal statements (PSs) representing important components of the application process. However, their inherently subjective nature predisposes them to potential implicit bias. Otolaryngology has historically been predominated by male physicians and while implicit sex bias has been demonstrated in LORs for application to residency of multiple specialties, data is limited for otolaryngology.

Methods

LORs and PSs for all otolaryngology applicants to an academic medical center during the 2019-20 and 2020-21 cycles were abstracted. Quantitative analysis was performed using Linguistic Inquiry and Word Count 2015 (LIWC2015), a validated software application designed to analyze various emotional, cognitive, and structural components of written text.

Results

LORs written for females were found to be written from a perspective of higher expertise and confidence while LORs written for males were associated with a more honest, personal, and disclosing tone. Moreover, LORs written for female applicants were found to reference achievement and "grindstone" terminology more than those written for men. No differences were observed in any word category between PSs written by male and female applicants.

Conclusion

Minor linguistic differences exist in multiple domains between LORs written for male and female applicants for otolaryngology residency. These tended to favor female applicants, with their letters demonstrating higher clout, achievement, and grindstone scores. This trend was unexpected in this historically predominantly male specialty. While differences were statistically significant, the overall difference in an entire letter of recommendation is likely subtle.

Level of Evidence

3.

For What It's Worth: Medical School Reputation in The Integrated Plastic Surgery Match

Integrated Plastic and Reconstructive Surgery residency programs may use medical school reputation to help fill the gap of a pass/fail USMLE Step 1 in the match. The main objective of this manuscript was to consider if this shifting emphasis is warranted. Herein, a cross-sectional analysis of academic plastic surgeons found that medical school reputation did not predict career achievement. In the absence of evidence demonstrating its worth, residency programs should exercise caution in using medical school reputation in the match.

Gender differences in emergency medicine standardized letters of evaluation

OBJECTIVES

The Standardized Letter of Evaluation (SLOE) is a vital portion of any medical student's emergency medicine (EM) residency application. Prior literature suggests gender bias in EM SLOE comparative ranking, but there is limited understanding of the impact of gender on other SLOE components. The study objective was to evaluate the presence of gender differences in the 7 Qualifications for EM (7QEM), Global Assessment (GA), and anticipated Rank List (RL) position. A secondary objective was to evaluate the gender differences in 7QEM scores and their link to GA and anticipated RL position.

METHODS

We performed a cross-sectional study using SLOEs from a subset of United States applicants to three EM residency programs during the 2019-2020 application cycle. We collected self-reported demographics, 7QEM scores, GA, and anticipated RL position. We utilized linear regression analyses and repeated measures ANOVA to evaluate if the relationship between the 7QEM scores, GA score, and anticipated RL position was different for men and women.

RESULTS

2103 unique applicants were included (38.6% women, 61.4% men), with 4952 SLOEs meeting inclusion criteria. The average QEM (2.51 vs. 2.39; p < 0.001), GA (2.68 vs. 2.48; p < 0.001), and RL (2.68 vs. 2.47; p < 0.001) scores were statistically higher for women than men. When exploring the relationship between the 7QEM and GA, Ability to communicate a caring nature to patients was not found to be a statistically significant predictor for men, but it was for women. When exploring the relationship between 7QEM and RL, Commitment to EM was not a significant predictor for men, but it was for women.

CONCLUSIONS

Women scored higher than men on the 7QEM, GA, and anticipated RL position on SLOEs. The 7QEM scores factored differently for men and women.

Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum

BACKGROUND

With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need.

OBJECTIVES

To better understand the degree to which DOM LORs comply with published guidelines.

METHODS

We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines.

RESULTS

Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases.

CONCLUSIONS

Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.

Applying to residency: survey of neurosurgical residency applicants on virtual recruitment during COVID-19

OBJECTIVE

The COVID-19 pandemic caused a significant disruption to residency recruitment, including a sudden, comprehensive transition to virtual interviews. The authors sought to characterize applicant experiences and perceptions concerning the change in the application, interview, and match process for neurological surgery residency during the 2020-2021 recruitment cycle.

METHODS

A national survey of neurosurgical residency applicants from the 2020-2021 application cycle was performed. This survey was developed in cooperation with the Society of Neurological Surgeons (SNS) and the American Association of Neurological Surgeons Young Neurosurgeons Committee (YNC) and sent to all applicants (n = 280) who included academic video submissions to the SNS repository as part of their application package. These 280 applicants accounted for 69.6% of the total 402 neurosurgical applicants this year.

RESULTS

Nearly half of the applicants responded to the survey (44.3%, 124 of 280). Applicants favored additional reform of the interview scheduling process, including a centralized scheduling method, a set of standardized release dates for interview invitations, and interview caps for applicants. Less than 8% of students desired a virtual-only platform in the future, though the majority of applicants supported incorporating virtual interviews as part of the process to contain applicant costs and combining them with traditional in-person interview opportunities. Program culture and fit, as well as clinical and research opportunities in subspecialty areas, were the most important factors applicants used to rank programs. However, subjective program "fit" was deemed challenging to assess during virtual-only interviews.

CONCLUSIONS

Neurosurgery resident applicants identified standardized interview invitation release dates, centralized interview scheduling methods, caps on the number of interviews available to each candidate, and regulated opportunities for both virtual and in-person recruitment as measures that could significantly improve the applicant experience during and effectiveness of future neurosurgery residency application cycles. Applicants prioritized program culture and "fit" during recruitment, and a majority were open to incorporating virtual elements into future cycles to reduce costs while retaining in-person opportunities to gauge programs and their locations.

Linguistic Analysis of Letters of Recommendation for Vascular Surgery and Obstetrics and Gynecology Applicants Detects Differences in Attributable Strengths Based on Gender

OBJECTIVE

We asked whether letters of recommendation (LOR) written for applicants to vascular surgery (VS), a field where men have traditionally predominated, differentially highlight attributes based on applicant gender. For comparison, LOR for applicants to Obstetrics and Gynecology (Ob/Gyn), a surgical field where women are highly represented were evaluated.

DESIGN

We performed a cross-sectional review of LORs for students applying to VS or Ob/Gyn at our institution from 2017 to2018. Blinded to the gender of both the applicant and the letter author, investigators assessed word count per letter and used published rubrics to quantify how many words in the following categories: communal ("friendly"), able ("competent"), standout ("exceptional"), and grindstone ("hardworking"). Frequencies were reported as a function of specialty and gender.

SETTING

The study was performed at the University of Pittsburgh Medical Center and included letters written for applicants only to the stated residency programs at University of Pittsburgh Medical Center.

PARTICIPANTS

LOR written for self-identified women and men applying to both residencies from US-based allopathic medical schools were de-identified and evaluated by blinded reviewers.

RESULTS

One hundred and ninety-eight letters were reviewed for vascular surgery applicants. Two hundred letters were randomly selected from applications to Ob/Gyn as a comparison. Fifty-four vascular (27.8% women) and 63 Ob/Gyn (77.8% women) applicants were reviewed (p < 0.001 for gender). Licensing exam scores were higher for women than men applying to Ob/Gyn. Honor status was similar across fields and gender. Letters were shorter for VS applicants (p = 0.04). Gender-specific words (i.e., "lady" or "gentleman") were used more in VS letters (0.24 ± 0.50 vs 0.14 ± 0.42, p = 0.048). Ability words were more common (4.7 ± 2.6 vs 3.8 ± 2.1, p = 0.028) and grindstone adjectives were less common (3.4 ± 2.3 vs 4.5 ± 3.1, p = 0.024) in letters written for women compared to men VS applicants. Twenty-nine letters written for students applying to VS had honors status. While none written for women mentioned this achievement, 43% of those written for men did (p < 0.05). Letters for women applicants to Ob/Gyn contained more standout adjectives than those written for men (2.12 ± 2.2 vs 1.39 ± 1.25, p = 0.021).

CONCLUSIONS

Reference letters for both specialties highlighted attributes differently depending on the gender of the applicant. Although this likely represents an unconscious process, care should be taken to limit potential biases in LOR which are "gatekeepers" to access and advancement.

Gender Bias in Surgical Oncology Fellowship Recommendation Letters: Gaining Progress

OBJECTIVE

Gender bias has been identified in letters of recommendation (LOR) in many different surgical training fields. Among surgeons, women comprise over 30% of the full-time faculty positions nationally and surgical oncology is one of the most gender diverse surgical subspecialties. We sought to determine if bias existed in LOR submitted to a Complex General Surgical Oncology (CGSO) fellowship.

DESIGN

LOR for the CGSO fellowship were retrospectively analyzed from applicants at a single institution over an 8-year period (2013-2020). The linguistic content of the letters was analyzed using Linguistic Inquiry and Word Count (LIWC2015), a validated text analysis program. Using multivariable analysis, LOR were compared by gender of both applicant and letter writer to explore the association between gender and the characteristics of the applicants and letter writers.

SETTING

University of North Carolina at Chapel Hill (UNC), Division of Surgical Oncology and Endocrine Surgery.

PARTICIPANTS

Applicants interviewed for the CSGO fellowship program at the UNC from 2013 to 2020 as well as all applicants from the 2018 application cycle, regardless of interview status.

RESULTS

About 841 letters from 219 interviewed applicants throughout the 2013 to 2020 surgical oncology fellowship application cycles were included. No difference in authenticity, clout, analytic thinking, or emotional tone of the letters was seen when comparing men and women applicants. Of the 41 word categories analyzed, only "references to achievement" in LOR written for women was significantly higher when compared to LOR written for men (p = 0.01). Interestingly, significantly more women applicants had at least 1 LOR written by a woman (p = 0.04). A subset analysis of all applicants regardless of interview status from the 2018 cycle included 294 LOR from 77 applicants. With the inclusion of noninterviewed applicants, LOR for men had more analytic tone than LOR for women (p = 0.02), otherwise there were no significant differences between the groups.

CONCLUSIONS

Very few differences in LOR were found for applicants at a CGSO fellowship program based on applicant or letter writer gender. The lack of gender bias demonstrates progress within the field of surgical oncology, likely a result of recent work and educational effort in this area. Efforts to expand this progress into other surgical sub-specialties are necessary.

Assessment of gender differences in letters of recommendation for pharmacy residency applicants

Disclaimer

In an effort to expedite the publication of articles related to the COVID-19 pandemic, AJHP is posting these manuscripts online as soon as possible after acceptance. Accepted manuscripts have been peer-reviewed and copyedited, but are posted online before technical formatting and author proofing. These manuscripts are not the final version of record and will be replaced with the final article (formatted per AJHP style and proofed by the authors) at a later time.

Purpose

Letters of recommendation (LORs) are highly regarded components of pharmacy residency applications, as they provide insight into an applicant’s character and capabilities. In other medical fields, differences in language have been reported for letters written for female and male applicants; however, data on gender differences in LORs for pharmacy residency applications are currently lacking.

Methods

LORs for applicants to our institution’s postgraduate year 1 pharmacy residency program for the 2019-2020 academic year were extracted and processed by a natural language processing service. Words within 18 categories were identified and counted for each LOR. Total was also compared.

Results

Of the 473 LORs included for analysis, 320 (67.7%) were written for female applicants and 153 (32.3%) were written for male applicants. Approximately two-thirds of all writers were women for both female and male applicants. In comparing letters for women and men, there was a statistically significant difference in the percentage of LORs that contained terms in categories described as gendered, solitary/reserved, and desire. There was no statistically significant difference in total or in the presence of words in other categories such as grindstone, standout, agentic, or communal. When controlling for grade point average, writer gender, duration that the writer knew the applicant, and the writer’s professional position, there were no changes to the statistical findings.

Conclusion

Letters written for female and male applicants were largely similar with regard to length and word categories utilized. While no clear gender bias was found when evaluating pharmacy residency LORs, writers must continue to assess their implicit biases and how those biases might affect a candidate’s application.

Perspectives on the quality and utility of letters conforming to the AAIM guidelines

Background Standardized letters of recommendation (SLOR) have become common features of the medical school to residency transition. Research has shown many advantages over the narrative letter of recommendation including improved letter-writing efficiency, ease of interpretation, and improved reliability as performance predictors. Currently, at least four specialties require fellowship SLORs. Internal medicine adopted its SLOR in 2017. Previous research showed fellowship program directors’ satisfaction with the 2017 guidelines. Little is known about residency program directors’ acceptance and adherence to the guidelines.

Objectives The study sought to assess the adoption rate of each component, barriers to adoption, time commitment, and alignment with intended goals of the guidelines.

Methods Anonymous survey links were posted to an internal medicine discussion forum prior to the guidelines in spring 2017 and twice following the guidelines in fall 2018 and winter 2019. Two-sample tests of proportions were used to compare respondent characteristics with known survey population data. Pre- and post-survey comparisons were assessed for statistical significance with Pearson chi-squared statistic.

Results The response rate varied from 30% to 35% for each survey period. Medical knowledge, patient care, interpersonal and communication skills, professionalism, and scholarly activity were reported frequently (>96%) at baseline. Inclusion of residency program characteristics, systems-based practice, practice-based learning and improvement, and skills sought to master increased over the study period.

Conclusions The new guidelines improved uniform reporting of all core competency data. Overall, the gains were modest, as many pre-survey respondents reported high rates of including components within the guidelines.

Beyond Medical Knowledge and Patient Care: A Program Director's Perspective for the Role of General Surgery Subinternships

OBJECTIVE

Established primary goals of general surgery subinternships are improvement in patient care and medical knowledge. However, the secondary gains such as obtaining recommendation letters and forming relationships are apparent but poorly defined. We sought the opinion of general surgery program directors (PDs) on the secondary purposes of subinternships. Our aim is to aide mentors and students to optimize the subinternship experience relative to the residency application process.

DESIGN

A survey was administered in July 2019. This questionnaire consisted of 11 items and was broken down into 3 sections: demographics, PD perspective on the secondary goals of a general surgery subinternship, and the role of general surgery subinternships on the students' residency application.

SETTING

An online, national survey through the Association for Program Directors listserv.

PARTICIPANTS

United States general surgery PDs affiliated with the Association for Program Directors listserv.

RESULTS

Sixty-one PDs completed the survey from 42 (69%) academic and 14 (23%) community programs. The majority of PDs (n = 33, 54%) reported that assessment of a subintern's suitability for their residency was the most important secondary purpose. Furthermore, PDs (n = 24, 39%) valued a letter of recommendation from faculty the student worked with during a subinternship as the most important criteria in the interview selection process. Away rotations were perceived as of equal value to subinternships completed at the student's home institution. Overall, PD opinions were similar at academic and community programs.

CONCLUSIONS

Our study suggests subinternships significantly impact a student's application to general surgery residency, clarifying a secondary role for these rotations. Subinternships are important for PDs to assess a student for ranking at their program. All students should pursue a letter of recommendation from subinternship faculty, when possible, as they can heavily influence the interview selection process. Away rotations should only be recommended for those students who need to strengthen their application.

Impact of Advanced Pharmacy Practice Experiences on Residency Interview Invitations

Objective. To determine how postgraduate year one (PGY1) pharmacy residency program directors perceive factors related to advanced pharmacy practice experiences (APPEs) when selecting candidates for residency interviews. Methods. An online cross-sectional nationwide survey of 1,280 PGY1 residency program directors was conducted. Participants were asked to rank the overall influence of five APPE categories, including location, structure, elective type, timing, and preceptor references, as well as the desirability and necessity of APPE-related variables representing the five categories, in their assessment of residency candidates. Results. Program demographics and survey data were collected from 375 participants (29% response rate). The category most influential to program directors' decisions was APPE preceptor reference letters, while the category that was the least influential was APPE timing factors. An APPE's location, structure, and elective type ranked second, third, and fourth, respectively, as the most influential categories. Respondents perceived factors similar to their own residency environment as desirable, supporting the study's conceptual framework of person-environment fit. The variables that the majority of residency directors specifically desired were two reference letters from APPE preceptors and letter grades for each APPE completed. Completion of inpatient elective APPEs was considered more desirable than completion of a balanced mix of elective APPEs. Participants rarely indicated that a specific APPE variable was a necessity for a candidate to be considered. Conclusion. Applicants to pharmacy residency programs should consider the importance of person-environment fit when selecting APPEs and preparing applications as program directors desire candidates who possess attributes compatible with their organization and complete APPEs in settings similar to that of their organization. Conversely, the absence of desired APPE-related variables does not necessarily exclude an applicant from consideration.

What Are We REALLY Looking for in a Letter of Recommendation?

OBJECTIVES

The letter of recommendation (LOR) is an important component of a residency application. There is great subjectivity in the interpretation of a letter writer's narrative and many residencies have moved toward standardized LORs (sLOR). We aimed to scrutinize the importance afforded to specific content and applicant attributes in an LOR in hopes of decoding this time-honored process.

DESIGN

A 35-question survey comprised of nonidentifying general questions, and participant evaluation of applicant characteristics and LOR phrases were administered (cross-sectional design). Evaluations were scored both objectively on a 10-point Likert scale and subjectively using a relative ranking.

SETTING

Academic, tertiary care center with a large general surgery residency program (Mayo Clinic, Rochester, MN).

PARTICIPANTS

Surgery attendings and general surgery residents (n = 122).

RESULTS

Seventy-two attendings and 50 general surgery residents completed the survey. Faculty ranked strong work ethic/hard working (median rank = 1) and inquisitive/hungry learner (median rank = 3) as the top 2 attributes. "We will plan to recruit this candidate" (median rank = 1.5) and "I give my highest recommendation" (median rank = 2) were the top 2 phrases. Residents valued strong work ethic and collaborative/team player as the top 2 applicant attributes. Seventy-three percent of attendings and 82% of residents agreed that LOR allows the applicant pool to be divided into upper and lower halves. Only 17% of faculty and 18% of residents agreed that an LOR allowed candidate stratification into quartiles.

CONCLUSIONS

Elaborating the most favorable applicant characteristics and highly regarded content in an LOR will help truly exceptional candidates obtain letters that make them stand out in the eyes of their evaluators. Since LORs are mostly considered to be able to stratify only upper and lower halves of the applicant pool, it is imperative to move toward LORs which portray superior applicant qualities, and can provide more objective evaluation of a candidate.

Letters of Recommendation

Letters of recommendation (LORs) are a central element of an applicant's portfolio for the National Resident Matching Program (known as the "Match"). This is especially true when applying to competitive specialties like emergency medicine (EM). LORs convey an applicant's potential for success, and also highlight an applicant's qualities that cannot always be recognized from a curriculum vitae, test scores, or grades. Traditional LORs, also called narrative LORs, are written in prose and are therefore highly subjective. This led to the establishment of a task force by the Council of Emergency Medicine Residency Directors in 1995 to develop a standardized LOR. Revisions of this form are now referred to as a standardized letter of evaluation. These evaluations in this format have proven to increase inter-rater reliability, decrease interpretation time, and standardize the process used by EM faculty to prepare evaluations for EM applicants. In this article, we will discuss LORs; address applicants' concerns, including from whom to request LORs (EM faculty vs. non-EM faculty vs. non-clinical faculty), number of LORs an applicant should include in his or her application materials, the preferred manner of requesting and the timing in which to ask for an LOR, as well as the philosophy behind waiving the right to see the letter.

A Screening Tool to Identify Qualified Pharmacy Residency Candidates

Objective. To develop and evaluate the utility of a prescreening tool to assess candidates for PGY1 pharmacy residency programs. Methods. A scoring tool was developed to prescreen candidates who applied to two PGY1 pharmacy residency programs. The tool scored applicants based on six domains: community service, leadership experience, letters of intent, letters of recommendation, presentations/publication, and work experience. Results. Applicants who were chosen to interview based on results from the screening tool were those who had significantly higher scores for all domains except work experience, as compared to applicants who were not interviewed. Total scores were also significantly higher. The average overall scores for applicants increased each year. Conclusion. Each year, the competition for first-year pharmacy residency positions continues to increase. A tool that can assess and differentiate between candidates' strengths by screening their applications can be a valuable asset for program administrators when used correctly.

Use of Standard Guidelines for Department of Medicine Summary Letters

Phenomenon: Fourth-year medical students obtain Department of Medicine (DOM) letters ("Chair" letters) to support their residency applications. Writing and interpreting DOM letters are challenging. There is heterogeneity in the letters that makes it difficult to both write and read these letters.

APPROACH

The purpose of this study is to determine the value of new guidelines developed by a task force of clerkship directors and program directors in internal medicine and assess the implementation of these guidelines. The Clerkship Directors in Internal Medicine performed a cross-sectional survey of internal medicine clerkship directors at U.S. and Canadian medical schools in 2014. In addition, the primary author's institution reviewed 1,347 DOM letters between 2012 and 2014 to assess the implementation of these guidelines.

FINDINGS

The survey response rate was 78%. DOM letter writers reported the guidelines were better, easier to implement, and more compatible with the purpose of DOM letters than previously. Most letter readers reported that letters using the guidelines were more credible. Writers of DOM letters in lower academic ranks rated the letters with guidelines higher in several domains than those in higher academic ranks. Readers of DOM letters in higher academic ranks rated the letters with guidelines higher in several domains than those in lower academic ranks. In the DOM letters examined, the odds of meeting each guideline increased with each additional year. However, for 3 guidelines there was an initial decline in adherence from 2012 to 2013 before increasing again in 2014-the recommended length, clerkship description, and detailed narrative guidelines. Letters solely written by a chair were less likely to incorporate the guidelines. Insights: Clerkship directors often write the DOM letters and identify with the purpose of the guidelines. As writers, lower ranking academic faculty value the guidelines more than higher ranking academic faculty. As readers of DOM letters, higher academic ranking faculty value letters that incorporate the guidelines more than lower academic ranking faculty. DOM letters implemented more guideline criteria since the guidelines were released. If implementing the guidelines, chairs should solicit the help of their clerkship director or educational representative when writing DOM letters. Although many clerkship directors read letters for their residency programs, additional program directors' opinions are needed.