Improving Our Ability to Predict Resident Applicant Performance: Validity Evidence for a Situational Judgment Test

Predicting Medical Student Performance With a Situational Judgment Test in Admissions

PURPOSE

To explore correlations between AAMC situational judgment test (SJT) scores, other admissions data, and learners' medical school performance.

METHOD

First- and second-year medical students from 8 U.S. MD-granting medical schools completed a prototype version of the AAMC SJT in 2017. Outcomes included research-only faculty ratings of student performance, final course grades, and faculty evaluations of student performance, 2017-2018 and 2018-2019 academic years. Bivariate correlations were used to investigate the relationship between SJT scores and student performance outcomes and hierarchical regressions to investigate whether SJT scores provided incremental validity over MCAT total scores and cumulative undergraduate grade point averages (UGPAs) for predicting student performance outcomes.

RESULTS

In general, there were small positive correlations with research-only faculty ratings from the first year of medical school, with the highest for social skills/service orientation ( rcorrected = .33, P < .05). Correlations were higher, with the highest for social skills/service orientation and cultural competence ( rcorrected = .33 and .36, respectively, P < .05) in the second year in medical school. SJT scores improved prediction of research-only faculty ratings over MCAT total scores and UGPAs for reliability and dependability/capacity for improvement, cultural competence, social skills/service orientation, and the overall composite score in the first year and for resilience and adaptability, social skills/service orientation, cultural competence, and the overall composite score in the second year. SJT scores demonstrated small correlations with course grades ( rsample-weighted = .10, P = ns) and faculty evaluations related to professionalism skills ( rsample-weighted = .14, P < .05); however, MCAT total scores explained most of the variance associated with course outcomes.

CONCLUSIONS

These studies provide initial evidence that SJT scores may add value to the medical school admissions process because scores were related to faculty ratings of professional behaviors and provided unique information relative to MCAT scores and UGPAs.

Opening the Black Box: Agreement and Reliability of a Situational Judgment Test Across Multiple Institutions

OBJECTIVE

The primary objective of this study was to examine the levels of agreement and reliability of a situational judgment test (SJT) using a diverse pool of pharmacy practice faculty as subject matter experts. Secondary aims included analyses to build support for test validity and fairness.

METHODS

An SJT containing 18 scenarios and 118 responses assessing empathy, integrity, and teamwork was developed and delivered to pharmacy practice faculty at 5 schools of pharmacy across the United States. Reliability was assessed by examining internal consistency, inter-rater reliability, and split-half reliability. Only responses which attained an inter-rater agreement>0.7 were included in the final version of the SJT. All responses were scored using a near-miss system, allowing higher scores for answers more closely aligned with the key, which was determined by the faculty who completed the SJT. Test fairness was reported using descriptive statistics.

RESULTS

Thirty-nine faculty across the 5 participating institutions completed the SJT. The final version of the SJT included 105 responses, achieving an inter-rater agreement of>0.7 (inter-rater reliability of 0.98). Split-half reliability was 0.72. The average score was 85.7%, and no differences in performance were observed based on demographic characteristics.

CONCLUSION

An SJT designed to assess empathy, integrity, and teamwork achieved reasonable levels of reliability among pharmacy practice faculty across the United States, and the results provided initial support for test validity and fairness. These results support a pilot to assess this SJT among students representing multiple institutions.

Maximizing Standardization While Ensuring Equity: Exploring the Role of Applicant Experiences, Attributes, and Metrics on Performance of a Surgery-Specific Situational Judgment Test

BACKGROUND

Situational judgment tests (SJT) are hypothetical but realistic scenario-based assessments that allow residency programs to measure judgment and decision-making among future trainees. A surgery-specific SJT was created to identify highly valued competencies among residency applicants. We aim to demonstrate a stepwise process for validation of this assessment for applicant screening through exploration of two often-overlooked sources of validity evidence - relations with other variables and consequences.

METHODS

This was a prospective multi-institutional study involving 7 general surgery residency programs. All applicants completed the SurgSJT, a 32-item test aimed to measure 10 core competencies: adaptability, attention to detail, communication, dependability, feedback receptivity, integrity, professionalism, resilience, self-directed learning, and team orientation. Performance on the SJT was compared to application data, including race, ethnicity, gender, medical school, and USMLE scores. Medical school rankings were determined based on the 2022 U.S. News & World Report rankings.

RESULTS

In total, 1491 applicants across seven residency programs were invited to complete the SJT. Of these, 1454 (97.5%) candidates completed the assessment. Applicants were predominantly White (57.5%), Asian (21.6%), Hispanic (9.7%), Black (7.3%), and 52% female. A total of 208 medical schools were represented, majority were allopathic (87.1%) and located in United States (98.7%). Less than a quarter of applicants (22.8%; N=337) were from a top 25 school based on U.S. News & World Report rankings for primary care, surgery, or research. Average USMLE Step 1 score was 235 (SD 37) and Step 2 score was 250 (SD 29). Sex, race, ethnicity, and medical school ranking did not significantly impact performance on the SJT. There was no relationship between SJT score and USMLE scores and medical school rankings.

CONCLUSIONS

We demonstrate the process of validity testing and importance of two specific sources of evidence-consequences and relations with other variables, in implementing future educational assessments.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values

PURPOSE

The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda.

METHOD

Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion.

RESULTS

Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process.

CONCLUSIONS

This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

Does it make sense to use written instruments to assess communication skills? Systematic review on the concurrent and predictive value of written assessment for performance

OBJECTIVES

To evaluate possible associations between learners' results in written and performance-based assessments of communication skills (CS), either in concurrent or predictive study designs.

METHODS

Search included four databases for peer-reviewed studies containing both written and performance-based CS assessment. Eleven studies met the inclusion criteria.

RESULTS

Included studies predominantly assessed undergraduate medical students. Studies reported mainly low to medium correlations between written and performance-based assessment results (Objective Structured Clinical Examinations or encounters with simulated patients), and gave correlation coefficients ranging from 0.13 to 0.53 (p < 0.05). Higher correlations were reported when specific CS, like motivational interviewing were assessed. Only a few studies gave sufficient reliability indicators of both assessment formats.

CONCLUSIONS

Written assessment scores seem to predict performance-based assessments to a limited extent but cannot replace them entirely. Reporting of assessment instruments' psychometric properties is essential to improve the interpretation of future findings and could possibly affect their predictive validity for performance.

PRACTICE IMPLICATIONS

Within longitudinal CS assessment programs, triangulation of assessment including written assessment is recommended, taking into consideration possible limitations. Written assessments with feedback can help students and trainers to elaborate on procedural knowledge as a strong support for the acquisition and transfer of CS to different contexts.

Identifying Potential Attrition during the Residency Applicant Screening Process Using a Situational Judgment Test

OBJECTIVE

As USMLE Step 1 moves to pass/fail, residency programs are seeking alternate interview selection processes. Attrition in general surgery is reported as high as 26%. Thus, it is important to ensure that programs are selecting and matching applicants with shared values. Situational judgment tests (SJTs) measure educational and cultural values by posing ambiguous situations and individuals rate the effectiveness of possible reactions. SJTs have previously been shown to identify job applicants with shared values while promoting diversity. Scoring categories are high, moderate, or low values congruence. We sought to explore predictive validity of the SJT relative to program attrition.

DESIGN

Residents who matched into our program between 2018 and 2021 completed the SJT. We tracked attrition.

SETTING

UT Health San Antonio, Texas PARTICIPANTS: Fifty-six categorical general surgery residents RESULTS: Per SJT ratings, the numbers of residents who had high, moderate, and low values congruence were 27, 16, and 13, respectively. Attrition numbers for residents who scored high and moderate congruence were similar, indicating that these ratings were indistinguishable. As such, we combined those 2 categories to create a 2 × 2 matrix and used signal detection theory as a framework for analysis. Overall attrition was 16.1% (9/56). Of the 43 residents who scored high or moderate congruence, 90.7% remained in the program. There was a 9.3% chance of attrition for these residents. Of the 13 residents who scored low congruence, 38.5% attrited. While scoring as low congruence on the SJT does not definitively indicate attrition, it does indicate that attrition is 4.14 times more likely for these residents (chi-square, p = 0.0121).

CONCLUSIONS

One of the most important aspects of residency applicant selection and interviewing is mitigating risk by identifying applicants who carry a high risk of attrition. The SJT significantly identifies at-risk applicants.

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

PURPOSE

To determine whether overall situational judgment test (SJT) scores are associated with programs' clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods.

METHOD

In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019-2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods.

RESULTS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings ( r = .21 and .14, P &lt; .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear ( P = .005) and a .22 SD increase at year-end ( P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear.

CONCLUSIONS

Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

Psychiatry Residency Directors' Attitudes Toward and Uses of the Medical Student Performance Evaluation and Other Potential Tools for Residency Selection

OBJECTIVE

A survey was conducted to determine US psychiatry residency directors' attitudes regarding current measures of medical student performance and their preferences for the future.

METHODS

A team of psychiatry medical student educators and residency program directors developed a 23-question survey. In July 2021, links to the survey were sent out to all program directors registered with the American Association of Directors of Psychiatric Residency Training.

RESULTS

Seventy program directors out of 223 initiated the survey, resulting in a response rate of 31.4%. Forty percent of respondents reported that the most important use of the Medical Student Performance Evaluation (MSPE) is in screening out applicants for interviews, and only 26.1% reported that the MSPE in its current form could be trusted to provide a valid and reliable assessment of a student's medical school performance. Most respondents agreed that in the absence of United States Medical Licensing Examination (USMLE) step 1 numerical scores, the existing MSPE format/content requirements should be modified, use a set of ranking categories that are uniform across all medical schools, and be supplemented with additional measures of the student's character and ability specific to psychiatry.

CONCLUSIONS

US psychiatry program directors are eager for change when it comes to the MSPE and how it reports rankings, grades, and professionalism. The transition of the USMLE step 1 score reporting to pass/fail presents an opportunity to pursue this change and for stakeholders from all medical specialties to work together toward a shared goal of an improved residency selection process.

The role of the emotive, moral, and cognitive components for the prediction of medical students' empathic behavior in an Objective Structured Clinical Examination (OSCE)

OBJECTIVES

Investigate whether medical students' emotive abilities, attitudes, and cognitive empathic professional abilities predict empathic behavior in an Objective Structured Clinical Examination (OSCE).

METHODS

Linear and multiple regressions were used to test concurrent validity between Interpersonal Reactivity Index (IRI), Jefferson Scale of Physician Empathy (JSPE-S), Situational Judgement Test (SJT-expert-based score (SJT-ES), SJT-theory-based score (SJT-TS)) and empathic behavior in an OSCE measured by Berlin Global Rating (BGR) and Verona Coding Definitions for Emotion Sequences (VR-CoDES).

RESULTS

Highest amounts of explained variance of empathic behavior measured by VR-CoDES were found for the SJT-ES (R2 = 0.125) and SJT-TS (R2 = 0.131). JSPE-S (R2 = 0.11) and SJT-ES (R2 = 0.10) explained the highest amount of variance in empathic behavior as measured by BGR. Stepwise multiple regression improved the model for BGR by including SJT-ES and JSPE-S, explaining 16.2% of variance.

CONCLUSIONS

The instrument measuring the emotive component (IRI) did not significantly predict empathic behavior, whereas instruments measuring moral (JSPE-S) and cognitive components (SJT) significantly predicted empathic behavior. However, the explained variance was small.

PRACTICE IMPLICATIONS

The instrument measuring the emotive component (IRI) did not significantly predict empathic behavior, whereas instruments measuring moral (JSPE-S) and cognitive components (SJT) significantly predicted empathic behavior. However, the explained variance was small. In a longitudinal assessment program, triangulation of different instruments assessing empathy offers a rich perspective of learner's empathic abilities. Empathy training should include the acquisition of knowledge, attitudes, and behavior to support learner's empathic behaviors.

Incorporating Holistic Review in Recruitment in a Physical Medicine and Rehabilitation Residency Program

ABSTRACT

Traditional residency recruitment practices are vulnerable to unconscious biases, inequity, lack of diversity, and have limited ability to predict future clinical competency. Holistic review and evidence-based strategies, such as structured interviews and situational judgment tests, can mitigate these limitations. A physical medicine and rehabilitation residency program restructured its residency recruitment process using holistic review principles and evidence-based strategies during the 2020-2021 academic year. A subcommittee developed a weighted screening rubric based on Association of American Medical Colleges core competencies and semistructured interview questions targeting specific domains. Formal implicit bias training was provided. Screening scores determined interview invitations. Applicants participated in three different domain-focused semistructured interviews. Screening and interview scores were combined to form the program rank list. A postinterview anonymous questionnaire was sent to interviewees to obtain feedback. Four hundred eighty-nine applicants were screened (82 interview invitations, 80 interviewed, 8 matched). The respondents to the postinterview questionnaire found that interviews evaluated them objectively (90%) and improved their impression of the program (90%). The program's match was consistent with previous matches. Interviewed applicants represented a racially diverse group. Most questionnaire respondents had a positive impression of the interview process. This article demonstrates incorporating holistic review into residency recruitment and provides specific resources to aid other institutions pursuing similar goals.

A video-based situational judgement test of medical students' communication competence in patient encounters: Development and first evaluation

OBJECTIVE

We developed and evaluated the Video-Based Assessment of Medical Communication Competence (VA-MeCo), a construct-driven situational judgement test measuring medical students' communication competence in patient encounters.

METHODS

In the construction phase, we conducted two expert studies (n_panel1 = 6, n_panel2 = 13) to ensure curricular and content validity and sufficient expert agreement on the answer key. In the evaluation phase, we conducted a cognitive pre-test (n = 12) and a pilot study (n = 117) with medical students to evaluate test usability and acceptance, item statistics and test reliability depending on the applied scoring method (raw consensus vs. pairwise comparison scoring).

RESULTS

The results of the expert interviews indicated good curricular and content validity. Expert agreement on the answer key was high (ICCs> .86). The pilot study showed favourable usability and acceptance by students. Irrespective of the scoring method, reliability for the complete test (Cronbach's α >.93) and its subscales (α >.83) was high.

CONCLUSION

There is promising evidence that medical communication competence can be validly and reliably measured using a construct-driven and video-based situational judgement test.

PRACTICE IMPLICATIONS

Video-based SJTs allow efficient online assessment of medical communication competence and are well accepted by students and educators.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

OBJECTIVE

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences.

DESIGN

PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded.

RESULTS

Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation.

CONCLUSIONS

As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

What do we know about written assessment of health professionals' communication skills? A scoping review

OBJECTIVE

The aim of this scoping review was to investigate the published literature on written assessment of communication skills in health professionals' education.

METHODS

Pubmed, Embase, Cinahl and Psychnfo were screened for the period 1/1995-7/2020. Selection was conducted by four pairs of reviewers. Four reviewers extracted and analyzed the data regarding study, instrument, item, and psychometric characteristics.

RESULTS

From 20,456 assessed abstracts, 74 articles were included which described 70 different instruments. Two thirds of the studies used written assessment to measure training effects, the others focused on the development/validation of the instrument. Instruments were usually developed by the authors, often with little mention of the test development criteria. The type of knowledge assessed was rarely specified. Most instruments included clinical vignettes. Instrument properties and psychometric characteristics were seldom reported.

CONCLUSION

There are a number of written assessments available in the literature. However, the reporting of the development and psychometric properties of these instruments is often incomplete. Practice implications written assessment of communication skills is widely used in health professions education. Improvement in the reporting of instrument development, items and psychometrics may help communication skills teachers better identify when, how and for whom written assessment of communication should be used.

Evaluating the Whole Applicant: Use of Situational Judgment Testing and Personality Testing to Address Disparities in Resident Selection

PURPOSE OF REVIEW

Urology program directors are faced with increasing numbers of applications annually, making holistic review of each candidate progressively more difficult. Efforts to streamline evaluation using traditional cognitive metrics have fallen short as these do not predict overall resident performance. Situational judgment tests (SJTs) and personality assessment tools (PATs) have been used in business and industry for decades to evaluate candidates and measure non-cognitive attributes that better predict subsequent performance. The purpose of this review is to describe what these assessments are and the current literature on the use of these metrics in medical education.

RECENT FINDINGS

SJTs relative to PATs have more original research. Data suggests that SJTs decrease bias, increase diversity, and may be predictive of performance in residency. PATs are also emerging with data to support use with ability to assess fit to program and certain traits identified more consistently among high-performing residents and correlation to performance on ACGME milestones. PATs may be more coachable than SJTs. SJTs and PATs are emerging as techniques to supplement the current resident application review process. Early evidence supports their use in undergraduate medical education as does some early preliminary results in graduate medical education.