The Revised 2017 MSPE: Better, But Not "Outstanding"

Gender-Based Differences in Language Used by Students to Describe Their Noteworthy Characteristics in Medical Student Performance Evaluations

PURPOSE

The noteworthy characteristic (NC) section of the medical student performance evaluation (MSPE) was introduced to facilitate holistic review of residency applications and mitigate biases. The student-written aspect of the characteristics, however, may introduce biases resulting from gender differences in self-promotion behaviors. The authors conducted an exploratory analysis of potential gender-based differences in language used in NCs.

METHOD

The authors performed a single-center cohort analysis of all student-written NCs at the Perelman School of Medicine (2018-2022). NCs were converted into single words and characterized into word categories: ability (e.g., "talent"), standout ("best"), grindstone ("meticulous"), communal ("caring"), or agentic ("ambitious"). The authors qualitatively analyzed NC topic characteristics (i.e., focused on scholarship, community service). Logistic regression was used to identify gender differences in word categories and topics used in NCs.

RESULTS

The cohort included 2,084 characteristics from 783 MSPEs (47.5%, n = 371 written by women). After adjusting for underrepresented in medicine status, honor society membership, and intended specialty, men were more likely to use standout (OR = 2.00; 95% confidence interval [CI] = 1.35, 2.96; P = .001) and communal (OR = 1.40; 95% CI = 1.03, 1.90; P = .03) words in their NCs compared with women but less likely to use grindstone words (OR = 0.72; 95% CI = 0.53, 0.98; P = .04). Men were more likely than women to discuss scholarship (OR = 2.03; 95% CI = 1.27, 3.23; P = .003), hobbies (OR = 1.45; 95% CI = 1.07, 1.96; P = .02), and/or awards (OR = 1.59; 95% CI = 1.16, 2.16; P = .004) and less likely to highlight community service (OR = 0.66; 95% CI = 0.48, 0.92; P = .02).

CONCLUSIONS

The self-written nature of NCs permits language differences that may contribute to gender bias in residency application.

An Elite Privilege: Top-Ranked Medical Schools Provide Fewer Comparative Performance Data on Their Students

PURPOSE

The objective of this study was to determine differences in the reporting of performance data on medical student performance evaluations (MSPEs) by medical school ranking.

METHODS

MSPEs from all US allopathic and osteopathic medical schools received by a single diagnostic radiology residency program during the 2021-2022 application cycle were retrospectively reviewed. Preclinical class and core clerkship grades were categorized as pass/fail or multitiered. Comparative summative assessments provided in the MSPEs were recorded. Medical schools were grouped by their US News & World Report rankings, and the proportion of reported performance metrics for each group was compared.

RESULTS

Information from 95% of US allopathic medical schools (148 of 155) and 73% of osteopathic medical schools (27 of 37) was collected, on the basis of 1,046 applications received. For preclinical classes, multitiered grading was reported by no schools ranked in the top 10, 17% of schools ranked 11th to 50th, 52% of schools ranked 51st to 100th, and 59% of unranked schools (P < .001). For core clinical clerkships, multitiered grades were reported by 70% of the top 10 ranked schools, 90% of schools ranked 11th to 50th, 94% of those ranked 51st to 100th, and 94% of unranked schools (P = .0463). Comparative summative assessments were reported by none of the top 10 ranked schools, 56% of schools ranked 11th to 50th, 80% of those ranked 51th to 100th, and 81% of unranked schools (P < .001).

CONCLUSIONS

Higher ranked medical schools are less likely to provide comparative assessment data on their MSPEs, which may disadvantage students from lower ranked medical schools.

A Retrospective Analysis of Medical Student Performance Evaluations, 2014-2020: Recommend with Reservations

BACKGROUND

The Medical Student Performance Evaluations (MSPE) is a cornerstone of residency applications. Little is known regarding adherence to Association of American Medical Colleges (AAMC) MSPE recommendations and longitudinal changes in MSPE content.

OBJECTIVES

Evaluate current MSPE quality and longitudinal changes in MSPE and grading practices.

DESIGN

Retrospective analysis.

PARTICIPANTS

Students from all Liaison Committee on Medical Education (LCME)-accredited medical schools from which the Stanford University Internal Medicine residency program received applications between 2014-2015 and 2019-2020.

MAIN MEASURES

Inclusion of key words to describe applicant performance and metrics thereof, including distribution among students and key word assignment explanation; inclusion of clerkship grades, grade distributions, and grade composition; and evidence of grade inflation over time.

KEY RESULTS

MSPE comprehensiveness varied substantially among the 149 schools analyzed. In total, 25% of schools provided complete information consistent with AAMC recommendations regarding key word/categorization of medical students and clerkship grades in 2019-2020. Seventy-seven distinct key word terms appeared across the 139 schools examined in 2019-2020. Grading practices markedly varied, with 2-83% of students receiving the top internal medicine clerkship grade depending on the year and school. Individual schools frequently changed key word and grading practices, with 33% and 18% of schools starting and/or stopping use of key words and grades, respectively. Significant grade inflation occurred over the 6-year study period, with an average 14% relative increase in the proportion of students receiving top clerkship grades.

CONCLUSIONS

A minority of schools complies with AAMC MSPE guidelines, and MSPEs are inconsistent across time and schools. These practices may impair evaluation of students within and between schools.

Competency-Based Medical Education: Considering Its Past, Present, and a Post-COVID-19 Era

Advancement toward competency-based medical education (CBME) has been hindered by inertia and a myriad of implementation challenges, including those associated with assessment of competency, accreditation/regulation, and logistical considerations. The COVID-19 pandemic disrupted medical education at every level. Time-in-training sometimes was shortened or significantly altered and there were reductions in the number and variety of clinical exposures. These and other unanticipated changes to existing models highlighted the need to advance the core principles of CBME. This manuscript describes the impact of COVID-19 on the ongoing transition to CBME, including the effects on training, curricular, and assessment processes for medical school and graduate medical education programs. The authors outline consequences of the COVID-19 disruption on learner training and assessment of competency, such as conversion to virtual learning modalities in medical school, redeployment of residents within health systems, and early graduation of trainees based on achievement of competency. Finally, the authors reflect on what the COVID-19 pandemic taught them about realization of CBME as the medical education community looks forward to a postpandemic future.

Compliance with CDIM-APDIM Guidelines for Department of Medicine Letters: an Opportunity to Improve Communication Across the Continuum

BACKGROUND

With rising applications to internal medicine programs and pending changes in United States Medical Licensing Examination Step 1 score reporting, program directors desire transparent data for comparing applicants. The Department of Medicine Letters of Recommendation (DOM LORs) are frequently used to assess applicants and have the potential to provide clearly defined data on performance including stratification of a medical school class. Despite published guidelines on the expected content of the DOM LOR, these LORs do not always meet that need.

OBJECTIVES

To better understand the degree to which DOM LORs comply with published guidelines.

METHODS

We reviewed DOM LORs from 146 of 155 LCME-accredited medical schools in the 2019 Match cycle, assessing for compliance with published guidelines.

RESULTS

Adherence to the recommendation for DOM LORs to provide a final characterization of performance relative to peers was low (68/146, 47%). Of those that provided a final characterization, 19/68 (28%) provided a quantitative measure, and 49/68 (72%) provided a qualitative descriptor. Only 17/49 (35%) with qualitative terms described those terms, and thirteen distinct qualitative scales were identified. Ranking systems varied, with seven different titles given to highest performers. Explanations about determination of ranking groups were provided in 12% of cases.

CONCLUSIONS

Adherence to published guidelines for DOM LORs varies but is generally low. For program directors desiring transparent data to use in application review, clearly defined data on student performance, stratification groupings, and common language across schools could improve the utility of DOM LORs.

Current State of the Medical Student Performance Evaluation: A Tool for Reflection for Residency Programs

BACKGROUND

The Medical Student Performance Evaluation (MSPE) provides important information to residency programs. Despite recent recommendations for standardization, it is not clear how much variation exists in MSPE content among schools.

OBJECTIVES

We describe the current section content of the MSPE in US allopathic medical schools, with a particular focus on variations in the presentation of student performance.

METHODS

A representative MSPE was obtained from 95.3% (143 of 150) of allopathic US medical schools through residency applications to the Zucker School of Medicine at Hofstra/Northwell in select programs for the 2019-2020 academic year. A manual data abstraction tool was piloted in 2018-2019. After training, it was used to code all portions of the MSPE in this study. The results were analyzed, and descriptive statistics were reported.

RESULTS

In preclinical years, 30.8% of MSPEs reported data regarding performance of students beyond achieving "passes" in a pass/fail curriculum. Only half referenced performance in the fourth year including electives, acting internships, or both. About two-thirds of schools included an overall descriptor of comparative performance in the final paragraph. Among these schools, a majority provided adjectives such as "outstanding/excellent/very good/good," while one-quarter reported numerical data categories. Regarding clerkship grades, there were numerous nomenclature systems used.

CONCLUSIONS

This analysis demonstrates the existence of extreme variability in the content of MSPEs submitted by US allopathic medical schools in the 2019-2020 cycle, including the components and nomenclature of grades and descriptors of comparative performance, display of data, and inclusion of data across all years of the medical education program.

Medical School to Residency: How Can We Trust the Process?

Background

To say that the transition from undergraduate medical education (UME) to graduate medical education (GME) is under scrutiny would be an understatement. Findings from a panel discussion at the 2018 Association of American Medical Colleges Annual meeting entitled, “Pass-Fail in Medical School and the Residency Application Process and Graduate Medical Education Transition” addressed what and when information should be shared with residency programs, and how and when that information should be shared.

Materials and Methods

Over 250 participants representing UME and GME (e.g. leadership, faculty, medical students) completed worksheets addressing these questions. During report-back times, verbal comments were transcribed in real time, and written comments on worksheets were later transcribed. All comments were anonymous. Thematic analysis was conducted manually by the research team to analyze the worksheet responses and report back comments.

Results

Themes based on suggestions of what information should be shared included the following: 1) developmental/assessment benchmarks such as demonstrating the ability/competencies to do clinical work; 2) performance on examinations; 3) grades and class ranking; 4) 360 evaluations; 5) narrative evaluations; 6) failures/remediation/gaps in training; 7) professionalism lapses; 8) characteristics of students such as resiliency/reliability; and 9) service/leadership/participation. In terms of how this information should be shared, the participants suggested enhancements to the current process of transmitting documents rather than alternative methods (e.g., video, telephonic, face-to-face discussions) and information sharing at both the time of the match and again near/at graduation to include information about post-match rotations.

Discussion

Considerations to address concerns with the transition from medical school to residency include further enhancements to the Medical Student Performance Evaluation, viewing departmental letters as ones of evaluation and not recommendation, a more meaningful educational handoff, and limits on the number of residency applications allowed for each student. The current medical education environment is ready for meaningful change in the UME to GME transition.

Clerkship Grading and the U.S. Economy: What Medical Education Can Learn From America's Economic History

Clerkship grades (like money) are a social construct that function as the currency through which value exchanges in medical education are negotiated between the system's various stakeholders. They provide a widely recognizable and efficient medium through which learner development can be assessed, tracked, compared, and demonstrated and are commonly used to make decisions regarding progression, distinction, and selection for residency. However, substantial literature has demonstrated how grades imprecisely and unreliably reflect the value of learners. In this article, the authors suggest that challenges with clerkship grades are fundamentally tied to their role as currency in the medical education system. Associations are drawn between clerkship grades and the history of the U.S. economy; 2 major concepts are highlighted: regulation and stock prices. The authors describe the history of these economic concepts and how they relate to challenges in clerkship grading. Using lessons learned from the history of the U.S. economy, the authors then propose a 2-step solution to improve upon grading for future generations of medical students: (1) transition from grades to a federally regulated competency-based assessment model and (2) development of a departmental competency letter that incorporates competency-based assessments rather than letter grades and meets the needs of program directors.

Standardization in the MSPE: Key Tensions for Learners, Schools, and Residency Programs

The Medical Student Performance Evaluation (MSPE), which summarizes a medical student's academic and professional undergraduate medical education performance and provides salient information during the residency selection process, faces persistent criticisms regarding heterogeneity and obscurity. Specifically, MSPEs do not always provide the same type or amount of information about students, especially from diverse schools, and important information is not always easy to find or interpret. To address these concerns, a key guiding principle from the Recommendations for Revising the MSPE Task Force of the Association of American Medical Colleges (AAMC) was to achieve "a level of standardization and transparency that facilitates the residency selection process." Benefits of standardizing the MSPE format include clarification of performance benchmarks or metrics, consistency across schools to enhance readability, and improved quality. In medical education, standardization may be an important mechanism to ensure accountability of the system for all learners, including those with varied backgrounds and socioeconomic resources. In this article, members of the aforementioned AAMC MSPE task force explore 5 tensions inherent in the pursuit of standardizing the MSPE: (1) presenting each student's individual characteristics and strengths in a way that is relevant, while also working with a standard format and providing standard content; (2) showcasing school-specific curricular strengths while also demonstrating standard evidence of readiness for internship; (3) defining and achieving the right amount of standardization so that the MSPE provides useful information, adds value to the residency selection process, and is efficient to read and understand; (4) balancing reporting with advocacy; and (5) maintaining standardization over time, especially given the tendency for the MSPE format and content to drift. Ongoing efforts to promote collaboration and trust across the undergraduate to graduate medical education continuum offer promise to reconcile these tensions and promote successful educational outcomes.

Binary Reporting of USMLE Step 1 Scores: Resident Perspectives

The recent consensus from the Invitational Conference on USMLE Scoring has recommended a transition to a binary pass/fail reporting on the USMLE Step 1 exam to be implemented from January 22, 2022. While this change was instituted in an effort to decrease medical student stress and re-iterate the importance of the Step 1 as merely a licensing or qualifying exam, this decision has profound implications for medical graduates of both United States and foreign medical schools. In addition to compounding the difficulties of resident selection by residency programs, the new system could exert significant mental and financial burden on medical students, and potentially affect the diversity of graduate medical education in the United States. This article draws attention to the downstream effects of a pass/fail system on the future of medical and surgical education.

Prioritizing the Interview in Selecting Resident Applicants: Behavioral Interviews to Determine Goodness of Fit

From our initial screening of applications, we assess that the 10% to 15% of applicants whom we will interview are all academically qualified to complete our residency training program. This initial screening to select applicants to interview includes a personality assessment provided by the personal statement, Dean's letter, and letters of recommendation that, taken together, begin our evaluation of the applicant's cultural fit for our program. While the numerical scoring ranks applicants preinterview, the final ranking into best fit categories is determined solely on the interview day at a consensus conference by faculty and residents. We analyzed data of 819 applicants from 2005 to 2017. Most candidates were US medical graduates (62.5%) with 23.7% international medical graduates, 11.7% Doctors of Osteopathic Medicine (DO), and 2.1% Caribbean medical graduates. Given that personality assessment began with application review, there was excellent correlation between the preinterview composite score and the final categorical ranking in all 4 categories. For most comparisons, higher scores and categorical rankings were associated with applicants subsequently working in academia versus private practice. We found no problem in using our 3-step process employing virtual interviews during the COVID pandemic.

Just a Game: the Dangers of Quantifying Medical Student Professionalism

A medical student on her internal medicine clerkship says her numerical medical professionalism grade was "just a game." Building on this anecdote, we suggest there is good reason to believe that numerical summative assessments of medical student professionalism can, paradoxically, undermine medical student professionalism by sapping internal motivation and converting conversations about core professional values into just another hurdle to residency. We suggest better ways of supporting medical student professional development, including a portfolio comprised of written personal reflection and periodic 360° formative assessment in the context of longitudinal faculty coaching.

The Revised Medical School Performance Evaluation: Does It Meet the Needs of Its Readers?

BACKGROUND

The Medical School Performance Evaluation (MSPE) is an important factor for application to residency programs. Many medical schools are incorporating recent recommendations from the Association of American Medical Colleges MSPE Task Force into their letters. To date, there has been no feedback from the graduate medical education community on the impact of this effort.

OBJECTIVE

We surveyed individuals involved in residency candidate selection for internal medicine programs to understand their perceptions on the new MSPE format.

METHODS

A survey was distributed in March and April 2018 using the Association of Program Directors in Internal Medicine listserv, which comprises 4220 individuals from 439 residency programs. Responses were analyzed, and themes were extracted from open-ended questions.

RESULTS

A total of 140 individuals, predominantly program directors and associate program directors, from across the United States completed the survey. Most were aware of the existence of the MSPE Task Force. Respondents read a median of 200 to 299 letters each recruitment season. The majority reported observing evidence of adoption of the new format in more than one quarter of all medical schools. Among respondents, nearly half reported the new format made the MSPE more important in decision-making about a candidate. Within the MSPE, respondents recognized the following areas as most influential: academic progress, summary paragraph, graphic representation of class performance, academic history, and overall adjective of performance indicator (rank).

CONCLUSIONS

The internal medicine graduate medical education community finds value in many components of the new MSPE format, while recognizing there are further opportunities for improvement.