Evaluating situational judgment test use and diversity in admissions at a southern US medical school

Exploring institutional stratification: Minority-serving institutional pathways to medical school acceptance in the United States

INTRODUCTION

Recruiting and training a diverse pool of physicians from historically excluded groups is vital to solving complex scientific problems and increasing access to patient care. Disparate educational and health outcomes of COVID-19 amplified this need. In stratified higher education systems, underfunded institutions that serve greater numbers of underrepresented in medicine (URM) students face unique barriers to entering physician training. However, pathways from historically underfunded minority-serving institutions (MSIs) in the United States to medical school (MD) are not well understood.

METHOD

A hierarchical logistic regression analysis of 328 488 de-identified applicants to US MD-granting medical schools for academic years 2017-2018 through 2022-2023 was used to determine predictors of acceptance based on MSI attendance compared to predominantly White institutions (PWI) and other literature-informed predictors. An interrupted time series analysis determined the significance in changes in observed differences in application acceptance before and after the COVID-19 pandemic.

RESULTS

In model 1, MSI attendance was associated with significantly lower odds of acceptance, with a 42% to 52% chance of acceptance compared to PWI applicants. Adding MCAT and GPA significantly increased odds of acceptance for most MSI applicants in model 2. MSI attendance, MCAT, GPA and socio-economic status indicators were useful predictors, improving the acceptance model by 39% better than the null model. Although some predictors showed a change in odds over time, none changed significantly when comparing before and after COVID-19.

DISCUSSION

MSI attending students were less likely to be admitted to MD programmes, highlighting systemic stratification until other academic factors were introduced to the model that alleviated lower odds, and the COVID-19 pandemic did not substantially change these trends. Improved pathways and strengthened institutional relationships between institutions that have greater proportions of underrepresented students and medical schools may help improve MSI applicant odds furthering diversity within the physician-trained workforce.

Investigating the Road to Equity: A Scoping Review of Solutions to Mitigate Implicit Bias in Assessment within Medical Education

Introduction

In medical education, assessments have high-stakes implications. Yet, assessments are rife with unconscious bias, which contributes to inequitable social structures. Implicit bias in assessment must be addressed because medical educators use assessments to guide learning and promote development of physicians' careers. In this scoping review, the authors map the literature on implicit bias in assessment, as it applies to: 1) the types of implicit bias addressed, 2) the targets and types of interventions studied or proposed, and 3) how publications describe intervention efficacy.

Methods

The authors conducted a scoping review of the literature on interventions to mitigate implicit bias that was published between January 2010 and August 2023. Author pairs independently screened articles for inclusion and extracted data. Discrepancies were resolved with discussion and consensus. Qualitative and quantitative analysis was informed by Anderson et al's three assessment orientations: fairness, assessment for inclusion (AfI), and justice.

Results

7,831 articles were identified; 54 articles were included. The majority (n = 37; 69%) of articles focus on implicit bias toward those underrepresented in medicine. Interventions to mitigate implicit bias were targeted toward admissions and applications, faculty training, recruitment, summative assessments, and evaluation templates. Interventions had fairness (n = 43; 96%) and AfI (n = 22; 49%) orientations; no articles used a justice-orientation. For the sub-set of research studies (n = 40), almost all (n = 34; 85%) examined a single program/institution.

Discussion

This scoping review showed that more work is necessary to address different types of implicit biases, move scholarship beyond single-institution studies, refine existing interventions, and evaluate how efficacy is defined.

Multiple Mini Interviews vs Traditional Interviews: Investigating Racial and Socioeconomic Differences in Interview Processes

Aim

The study aims to compare traditional interviews with Multiple Mini Interviews (MMIs) to assess their reliability in evaluating applicants across racial and socioeconomic backgrounds.

Background

In the 2019-2020 admissions cycle, The University of Texas Medical Branch John Sealy School of Medicine (JSSOM) admissions committee observed inconsistencies in interview scoring, topics discussed during interviews, and interviewer comments using an unstructured interview format. Additionally, the recent Supreme Court decisions in Students for Fair Admissions, Inc. (SFFA) v. the University of North Carolina and SFFA v. President & Fellows of Harvard College, which ended race-conscious admissions, or affirmative action, and upheld holistic admissions practices, encouraged medical school admissions committee to reevaluate their approach to admissions.

Methods

Data from six admissions cycles (2018-2022), totaling 5799 interviewees, were analyzed to assess potential biases and the effectiveness of the admissions process. Spearman correlation examined relationships between Casper scores and both traditional interview and MMI outcomes. T-tests and Cohen's d explored demographic differences across URM, African American, Hispanic, and disadvantaged applicants in interview and academic metrics to evaluate fairness.

Results

When comparing MMIs and traditional interviews, both appear equal in reducing group differences (Underrepresented in Medicine(URM) versus non-URM, African American to non-URM, Hispanic to non-URM, and disadvantaged to non-disadvantaged). MMIs decreased demographic differences compared with traditional interviews for African-American candidates and slightly increased for URM, Hispanic, and disadvantaged candidates, but the effect size was small.

Conclusion

Future work includes conducting rigorous data analysis to better assess the MMI's utility, exploring the correlations between MMI scores, clinical evaluations, and objective structured clinical examination (OSCE) results. This multifaceted approach will provide a comprehensive view of how MMI performance aligns with real-world clinical assessments and standardized evaluation metrics, offering valuable insights into its effectiveness as a predictor of future medical proficiency.

Associations between an open-response situational judgment test and performance on OSCEs and fieldwork: implications for admissions decisions and matriculant diversity in an occupational therapy program

BACKGROUND

Casper, an online open-response situational judgement test that assesses social intelligence and professionalism [1], is used in admissions to health professions programs.

METHOD

This study (1) explored the incremental validity of Casper over grade point average (GPA) for predicting student performance on objective structured clinical examinations (OSCEs) and fieldwork placements within an occupational therapy program, (2) examined optimal weighting of Casper in GPA in admissions decisions using non-linear optimization and regression tree analysis to find the weights associated with the highest average competency scores, and (3) investigated whether Casper could be used to impact the diversity of a cohort selected for admission to the program.

RESULTS

Multiple regression analysis results indicate that Casper improves the prediction of OSCE and fieldwork performance over and above GPA (change in Adj. R2 = 3.2%). Non-linear optimization and regression tree analysis indicate the optimal weights of GPA and Casper for predicting performance across fieldwork placements are 0.16 and 0.84, respectively. Furthermore, the findings suggest that students with a slightly lower GPA (e.g., 3.5-3.6) could be successful in the program as assessed by fieldwork, which is considered to be the strongest indicator of success as an entry-level clinician. In terms of diversity, no statistically significant differences were found between those actually admitted and those who would have been admitted using Casper.

CONCLUSION

These results constitute preliminary validity evidence supporting the integration of Casper into applicant selection in an occupational therapy graduate program.

The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests

BACKGROUND

Situational Judgment Tests (SJTs) are commonly used in medical school admissions. However, it has been consistently found that native speakers tend to score higher on SJTs than non-native speakers, which can be particularly problematic in the admission context due to the potential risk of limited fairness. Besides type of SJT, awareness of time limit may play a role in subgroup differences in the context of cognitive load theory. This study examined the influence of SJT type and awareness of time limit against the background of language proficiency in a quasi high-stakes setting.

METHODS

Participants (N = 875), applicants and students in healthcare-related study programs, completed an online study that involved two SJTs: one with a text-based stimulus and response format (HAM-SJT) and another with a video-animated stimulus and media-supported response format (Social Shapes Test, SST). They were randomly assigned to a test condition in which they were either informed about a time limit or not. In a multilevel model analysis, we examined the main effects and interactions of the predictors (test type, language proficiency and awareness of time limit) on test performance (overall, response percentage).

RESULTS

There were significant main effects on overall test performance for language proficiency in favor of native speakers and for awareness of time limit in favor of being aware of the time limit. Furthermore, an interaction between language proficiency and test type was found, indicating that subgroup differences are smaller for the animated SJT than for the text-based SJT. No interaction effects on overall test performance were found that included awareness of time limit.

CONCLUSION

A SJT with video-animated stimuli and a media-supported response format can reduce subgroup differences in overall test performance between native and non-native speakers in a quasi high-stakes setting. Awareness of time limit is equally important for high and low performance, regardless of language proficiency or test type.

Racial and Ethnic Disparities in Situational Judgment Testing Among Applicants to an Anesthesiology Residency Program

Background The Computer-Based Assessment for Sampling Personal Characteristics (CASPer) is a situational judgment test (SJT) that assesses noncognitive skills like professionalism, communication, and empathy. There are no reports of the effects of race/ethnicity and sex on CASPer scores among residency applicants. Objective We examined the effects of race/ethnicity, sex, and United States vs international medical school attendance on CASPer performance. Methods Our anesthesiology residency program required all applicants for the 2021-2022 Match cycle to complete an online video and text-based SJT (CASPer). We compared these results, reported as z-scores, with self-identified race/ethnicity, sex, United States vs international medical school attendance, and United States Medical Licensing Examination (USMLE) Step 1 scores. Results Of the 1245 applicants who completed CASPer, 783 identified as male. The racial/ethnic distribution was 512 White, 412 Asian, 106 Black, 126 Hispanic, and 89 Other/No Answer. CASPer z-scores did not differ by sex. White candidates scored higher than Black (0.18 vs -0.57, P<.001) and Hispanic (0.18 vs -0.52, P<.001) candidates. Applicants attending US medical schools scored higher than those attending international medical schools (z-scores: 0.15 vs -0.68, P<.001). There was no correlation between CASPer z-scores and USMLE Step 1 scores. Conclusions Our results suggest that CASPer scores favor White applicants over Black and Hispanic ones and applicants attending US medical schools over those attending international medical schools.

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

BACKGROUND

Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

METHODS

In 2022, admissions data and information in the files of residents in difficulty (over three years pre- and post- Casper implementation) was used to determine the number of residents in difficulty, CanMEDS roles requiring a learning intervention, types of learning interventions (informal learning plans vs. formal remediation or probation), and impact on the utilization of institutional resource (costs and time). Professionalism concerns were mapped to the 4I domains of a professionalism framework, and their severity was considered in mild, moderate, and major categories. Descriptive statistics and between group comparisons were used for quantitative data.

RESULTS

In the pre- and post- Casper cohorts the number of residents in difficulty (16 vs. 15) and the number of learning interventions (18 vs. 16) were similar. Professionalism concerns as an outcome measure decreased by 35% from 12/16 to 6/15 (p < 0.05), were reduced in all 4I domains (involvement, integrity, interaction, introspection) and in their severity. Formal learning interventions (15 vs. 5) and informal learning plans (3 vs. 11) were significantly different in the pre- and post-Casper cohorts respectively (p < 0.05). This reduction in formal learning interventions was associated with a 96% reduction in costs f(rom hundreds to tens of thousands of dollars and a reduction in time for learning interventions (from years to months).

CONCLUSIONS

Justifiable from multiple stakeholder perspectives, use of an SJT (Casper) improves a clinical performance measure (professionalism concerns) and permits the institution to redirect its limited resources (cost savings and time) to enhance institutional endeavors and improve learner well-being and quality of programs.

Evaluating non-cognitive skills in medical school applicants

Medical school candidates must have both the cognitive and professional competencies required to become good physicians. In this commentary, we outline the evidence and outcomes associated with the implementation of these selection methodologies and evaluate their ability to assess non-cognitive skills.