How effective are selection methods in medical education? A systematic review

Does academic excellence predict non-cognitive competencies? Insights from a situational judgement test among medical students in Nigeria

BACKGROUND

Situational judgment tests (SJTs) are becoming more widely accepted for assessing medical students' behavioral (non-cognitive) competencies. However, the correlation between behavioral and clinical competencies is under-explored, particularly in sub-Saharan Africa. Using a validated SJT questionnaire, this study examined the association between SJT and the academic performance of clinical medical undergraduates in Enugu state, Nigeria.

METHOD

This was a cross-sectional study of 279 clinical medical students at the Enugu State University of Science, Management and Technology (ESUT), Nigeria. The results of the SJT of behavioral competencies were collected using a pre-tested, validated, and self-administered questionnaire. Cronbach's alpha for SJT competencies was below 0.7, which reflects domain heterogeneity rather than poor tool precision. The association between the dependent variable (SJT performance) and individual-level predictors was assessed using both bivariate and multivariate linear regressions. Statistical significance was set at p < 0.01.

RESULTS

An increase in students' academic performance predicted a decrease in their SJT performance by 3% (β= -0.03; p = 0.21), but this finding was not statistically significant. Having a grade of A or B in chemistry, compared to a grade of C4 or below, predicted a statistically significant increase in SJT performance by 15% (β = 0.15; p = 0.01). Being a male student predicted a decrease in the SJT score by 8% (β= -0.08; p = 0.09). The results also showed that students' state of origin (β = 0.28; p = 0.04) correlated with their SJT performance.

CONCLUSION

Our study findings suggest that students who perform better clinically are less likely to have higher behavioral competence. However, the negative association between clinical performance and SJTs is not always statistically significant. This has implications for training and implies that excellent clinical education may not facilitate the production of better doctors, but this is not always the case. Hence, there may be a case for incorporating mechanisms to specifically build behavioral competencies in medical students being trained. There is a need for studies that examine these associations in other contexts and test the impact of trainings and other mechanisms in building behavioral competencies in medical professionals.

CLINICAL TRIAL REGISTRATION

Not applicable.

Evidence of Differential Attainment in Canadian Medical School Admissions: A Scoping Review

PURPOSE

In Canada, many groups (e.g., Black, Indigenous, rural backgrounds) have historically faced and continue to encounter systemic barriers in accessing the medical profession. These barriers often manifest in performance disparities, known as differential attainment, during medical school admissions. This scoping review summarizes the nature and extent of evidence on the association of differential attainment in medical school admissions selection tools and outcomes with applicant social identity in the Canadian context.

METHOD

The authors used Arksey and O'Malley's scoping review framework to summarize research studies published between 2000 and 2022 with empirical evidence of differential attainment in admissions selection tools and outcomes with respect to a range of applicant social identity categories. The authors recorded whether studies adopted a structuralist and/or intersectional perspective.

RESULTS

Ultimately, 15 studies were included in the review. While the evidence on differential attainment associated with social identity in Canadian medical education was heterogeneous, numerous studies highlight differential attainment in the admissions process associated with applicant race and/or ethnicity (6 studies), age (5 studies), gender (4 studies), socioeconomic status (3 studies), geographic location (4 studies), and rural or urban background (5 studies). These attainment differences were reported at 3 phases of the admissions process (invitation to interview, offer of admission, and acceptance of offer) and were driven by several admissions selection tools, including grade point average, Medical College Admission Test score, and interview performance.

CONCLUSIONS

The review highlights evidence that suggests systemic, structural inequities in admissions systems manifest as differential attainment in Canadian medical school admissions. Based on this evidence, those who identify as Black or Indigenous and those with low socioeconomic status or rural backgrounds were generally more adversely affected. Admission practices must be studied and improved so medical education systems can better avow equality and human dignity and achieve equity goals.

Can all roads lead to competency? School levels effects in Licensing examinations scores

At the foundation of research concerned with professional training is the idea of an assumed causal chain between the policies and practices of education and the eventual behaviours of those that graduate these programs. In medicine, given the social accountability to ensure that teaching and learning gives way to a health human resource that is willing and able to provide the healthcare that patients and communities need, it is of critical importance to generate evidence regarding this causal relationship. One question that medical education scholars ask regularly is the degree to which the unique features of training programs and learning environments impact trainee achievement of the intended learning outcomes. To date, this evidence has been difficult to generate because data pertaining to learners is only rarely systematically brought together across institutions or periods of training. We describe new research which leverages an inter-institutional data-driven approach to investigate the influence of school-level factors on the licensing outcomes of medical students. Specifically, we bring together sociodemographic, admissions, and in-training assessment variables pertaining to medical trainee graduates at each of the six medical schools in Ontario, Canada into multilevel stepwise regression models that determine the degree of association between these variables and graduate performances on the Medical Council of Canada Qualifying Examinations (Part 1, n = 1097 observations; Part 2, n = 616 observations), established predictors of downstream physician performance. As part of this analysis, we include an anonymized school-level (School 1, School 2) independent variable in each of these models. Our results demonstrate that the largest variable associated with performance on both the first and second parts of the licensing examinations is prior academic achievement, notably clerkship performance. Ratings of biomedical knowledge were also significantly associated with the first examination, while clerkship OSCE scores and enrollment in a family medicine residency were significantly associated with the Part 2. Small significant school effects were realized in both models accounting for 4% and 2% of the variance realized in the first and second examinations, respectively. These findings highlight that school enrollment plays a minor role relative to individual student performance in influencing examination outcomes.

Tolerance for uncertainty and medical students' specialty choices: A myth revisited

BACKGROUND

In 1962, the idea emerged that medical students' tolerance of uncertainty could determine their specialty choice. While some studies supported this claim, others refuted it, often using independently developed instruments. We explored whether the reported link between specialty choice and uncertainty tolerance is more myth than evidence by employing established instruments to investigate whether specialty choice could be explained by variance in uncertainty tolerance.

METHOD

We conducted a cross-sectional online survey at two periods of time. From February to June 2023, we queried 563 final-year medical students from 34 German medical universities (1) on their uncertainty tolerance using three validated tools (the modified tolerance for ambiguity scale, the physicians' reaction to uncertainty scale and the uncertainty intolerance scenario method) and (2) on their intended specialty choice. In a follow-up 1 year later (May to June 2024), 263 of those medical students responded to our query on their final specialty choice and again on their uncertainty tolerance.

RESULTS

Participants' (N = 563) median age was 26.0 years (mean: 27.2; SD = 3.8), and 70% (n = 396) were female. Originally reported differences and rank orders in uncertainty tolerance among medical students with different intended specialty choices could not be replicated for any of the three scales. Instead, our results suggest different rank orders of uncertainty tolerance by different tools, as well as nonsignificant differences between intended medical specialties. Intercorrelation coefficient analyses demonstrated that, depending on the scale, only 0.3% to 1.5% of the variance in uncertainty tolerance could be attributed to specialty choice. Follow-up data using actual instead of intended medical choices left findings unchanged.

DISCUSSION

Our findings suggest that the presumed link between uncertainty tolerance and specialty choice is more myth than evidence. Instead of teaching this link or using it as an admissions criterion, medical schools should equip students with the skills needed to navigate uncertainty across their careers.

Evaluating factors that impact scoring an open response situational judgment test: a mixed methods approach

Introduction

Situational judgment tests (SJT) are commonly used in admissions to measure skills associated with professionalism. Although open-response SJTs have shown strong psychometric properties, assessors' personal beliefs, experiences, and cultural backgrounds may influence how they perceive, organize and evaluate information within test takers' diverse responses. Additionally, SJT research typically focuses on reliability and predictive validity, whereas the construct validity of open response SJTs remains underexplored. This mixed methods study aims to address this gap by exploring the construct-(ir)relevant factors that may impact assessors' evaluation of professionalism in open response SJTs.

Methods

For this study, we used data from Casper, an open response SJT commonly used in professional program admissions. In Study I, a quantitative content analysis was conducted on 160 responses to identify factors which were significant predictors of low and high scores. Correlation coefficients and logistic regression models were used to evaluate the relationship between each factor and response scores. In Study II, think-aloud activities were conducted with 23 Casper assessors to directly observe how they evaluated responses. All interviews were transcribed verbatim, which were then thematically analyzed using an inductive coding technique.

Results

Results from both the content analyses and think-aloud activities revealed that several construct relevant factors influenced scores. Scores were impacted by the extent to which test takers demonstrated the competencies probed for by the SJT, engaged with the context of the presented ethical dilemma, provided in-depth justifications for their response, considered various perspectives relevant to the presented dilemma, and provided creative solutions or insightful arguments for the suggested approach. Mixed results were found with respect to construct irrelevant factors, such as the flow, cohesion, and kinds of phrases used in the response.

Conclusion

This mixed methods study contributes to the construct validity of SJTs by investigating construct relevant and irrelevant factors that may impact assessors' evaluation of open responses. The findings of this study provide evidence that open-response SJTs are valid approaches to measure professional competencies more broadly, both in terms of what test takers focus on in their responses, as well as in terms of how they construct their responses.

Medical school selection is a sociohistorical embedded activity: A comparison of five countries

INTRODUCTION

The medical school selection literature comes mostly from a few countries in the Global North and offers little opportunity to consider different ways of thinking and doing. Our aim, therefore, was to critically consider selection practices and their sociohistorical influences in our respective countries (Brazil, China, Singapore, South Africa and the UK), including how any perceived inequalities are addressed.

METHODS

This paper summarises many constructive dialogues grounded in the idea of he er butong () (harmony with diversity), learning about and from each other.

RESULTS

Some practices were similar across the five countries, but there were differences in precise practices, attitudes and sociohistorical influences thereon. For example, in Brazil, South Africa and the UK, there is public and political acknowledgement that attainment is linked to systemic and social factors such as socio-economic status and/or race. Selecting for medical school solely on prior attainment is recognised as unfair to less privileged societal groups. Conversely, selection via examination performance is seen as fair and promoting equality in China and Singapore, although the historical context underpinning this value differs across the two countries. The five countries differ in respect of their actions towards addressing inequality. Quotas are used to ensure the representation of certain groups in Brazil and regional representation in China. Quotas are illegal in the UK, and South Africa does not impose them, leading to the use of various, compensatory 'workarounds' to address inequality. Singapore does not take action to address inequality because all people are considered equal constitutionally.

DISCUSSION

In conclusion, medical school selection practices are firmly embedded in history, values, societal expectations and stakeholder beliefs, which vary by context. More comparisons, working from the position of acknowledging and respecting differences, would extend knowledge further and enable consideration of what permits and hinders change in different contexts.

Physicians' Social Skills - Conceptualization, Taxonomy, and Behavioral Assessment

Social skills (e.g., assertiveness, empathy, ability to accept criticism) are essential for the medical profession and therefore also for the selection and development of medical students. However, the term "social skills" is understood differently in different contexts. There is no agreed upon taxonomy for classifying physicians' social skills, and skills with the same meaning often have different names. This conceptual ambiguity presents a hurdle to cross-context communication and to the development of methods to assess social skills. Drawing from behavioral psychology, we aim to contribute to a better understanding of social skills in the medical context. To this end, we introduce a theoretically and empirically informed taxonomy that can be used to integrate the large number of different social skills. We consider how skills manifest at the behavioral level to ensure that we focus only on skills that are actually observable, distinguishable, and measurable. Here, behavioral research has shown that three overarching skill dimensions can be seen in interpersonal situations and are clearly distinguishable from each other: agency skill (i.e., getting ahead in social situations), communion skill (i.e., getting along in social situations), and interpersonal resilience (i.e., staying calm in social situations). We show that almost all social skills relevant for physicians fit into this structure. The approach presented allows redundant descriptions to be combined under three clearly distinguishable and behavior-based dimensions of social skills. This approach has implications for the assessment of social skills in both the selection and development of students.

Developing institution-specific admission competency criteria for prospective health sciences students

BACKGROUND

Given the critical role of health professionals in societal health, the development of robust and effective selection methods is of fundamental concern for educational institutions within the field of health sciences education. Conventionally, admission competencies have been determined by institutional authorities. Developing institution-specific competency criteria enables an admission process that is mission- and value-aware, evidence-based, and strategically adaptable. However, few schools have established their admission competency criteria, although the majority possess their own models of graduation competencies. This study reports the process of developing and validating an institution-specific admission competency model that addresses the need for evidence-based and mission-aligned selection processes that are distinct from standardized models.

METHODS

This study was conducted in two phases, using both qualitative and quantitative analyses. Phase I involved constructing an admission competency model through a qualitative approach facilitated by workshops with 17 faculty members and 92 first-year pre-doctoral students of a dental school. Through constant comparative analysis, this phase focused on the extraction and refinement of competencies for entering dental students. In Phase II, a questionnaire developed from the workshops asked respondents to rate the importance of 47 attributes across 10 constructs on a 5-point Likert scale. A total of 301 individuals participated in the survey. Exploratory Factor Analysis (EFA) identified the factor structure, and Confirmatory Factor Analysis (CFA) examined construct validity and assessed the model fit with the data.

RESULTS

The EFA of the 47 attributes identified 10 factors, and the CFA results indicated a good-to-acceptable level of fit for the ten-factor model. Aligned with the American Association of Medical Colleges Premed competencies, this study identified unique attributes specific to the institution, such as confidence, leadership, and entrepreneurship. These findings highlight the importance of developing tailored competencies reflecting the unique needs of institutions and their fields.

CONCLUSIONS

This study demonstrates the feasibility and value of creating institution-specific admission competency models, offering a methodology that aligns with evidence-based mission-driven selection processes. The distinct competencies identified emphasize the need for educational institutions to consider unique institutional and field-specific requirements and move beyond standardized models to enhance the selection of medical students.

Simulation in admissions interviews: applicant experiences and programmatic performance prediction Simulation dans les entrevues d'admission : expériences des candidats et prédiction du rendement au sein des programmes de formation

Background

Admissions interviews are frequently used to assess personal and interpersonal attributes required for successful medical practice. Using simulation in interviews to engage applicants in realistic medical scenarios to assess these attributes is novel. This study evaluates applicant perceptions of simulation within multiple mini-interviews (MMI) and reports on subsequent student program performance.

Methods

Physician assistant (PA) program applicants were invited to complete an anonymous post-interview survey that included one free-response question about their admissions experience. We chose to qualitatively analyze the free-response question. Additionally, success metrics of students who experienced simulation-based MMI were compared to prior cohorts who were admitted using traditional interviews.

Results

Applicants undergoing simulation-based interviews in MMI had decreased incidences of major professionalism events, greater on-time program progression, and similar board pass rates compared to applicants who experienced traditional interviews. Several themes, highlighting the applicants' varied responses to the simulation-based MMI, emerged including showcasing strengths and passion, feelings of fairness, accessing program faculty, and impacts on certainty.

Conclusions

The use of simulation in admissions interviews is a valuable tool for assessing an applicant's personal attributes in a clinical setting. Applicants admitted using simulation had improved programmatic performance compared to applicants admitted using traditional interviews. Applicants' perceptions of simulation in interviews are helpful when designing the admissions experience.

Reframing selection as a learning experience: Insights from a residency selection assessment

PURPOSE

Selection for postgraduate medical training is high-stakes and complex. The learning value of assessments for selection has, thus far, been underexplored, limiting their uptake as potentially meaningful learning experiences. The aim of this study was to explore the learning value residency applicants derive from an intelligence, personality, motivation and competency selection assessment and what factors influence the experienced learning value.

METHODS

In Autumn 2020 and Spring 2021, we conducted individual semi-structured interviews with sixteen applicants for pediatric residency training. Selection outcomes were unknown at the time of the interview. Interviews were transcribed verbatim and thematically analyzed.

RESULTS

Participants reported that the assessment was valuable in fostering self-reflection and self-awareness, embracing self-acceptance, pursuing development goals, assessing professional fit, and harnessing motivational drivers in work. The experienced learning value was influenced by applicants' ability to interpret its results, their focus on the high-stakes selection process and concerns regarding the acceptability and credibility of the selection tool.

CONCLUSIONS

While the selection assessment showed learning potential, its learning value was impeded by a preoccupation with the high-stakes nature of the selection procedure. Intentional integration of the selection assessment in the learning curriculum may play a pivotal role in realizing its learning potential.

Does voluntary practice improve the outcome of an OSCE in undergraduate medical studies? A Propensity Score Matching approach

In Objective Structured Clinical Examinations (OSCE), students have to demonstrate proficiency in a wide array of medical knowledge and different skills, ranging from physical examinations to medical-practical skills and doctor-patient interactions. In this study, we empirically test the concept 'assessment drives learning' and investigate whether an OSCE motivates extracurricular, voluntary free practice (FP) of specific skills in a Skills Lab, and whether this has positive treatment effects on exam success in the respective parts of the OSCE. To explore causal inference with observational data, we used Propensity Score Matching (PSM) to generate a control and a treatment group that only differed in their practice behavior. For internal examinations and practical skills such as venous catheter placement and IM injections, we find strong, positive effects of FP that can result in a grade-jump. We further show that the presence and strength of effects depends on the complexity and type of the task. For instance, we find no effect for practicing venipuncture, and performance in communicative skills is associated with the willingness to repeatedly engage with instructional contents inside and outside the Skills Lab, and not with targeted practice of specific skills. We conclude that the anticipation of the complex OSCE is effective in motivating students to engage with a wide range of competencies crucial to the medical profession, and that this engagement has positive effects on exam success. However, consistent practice throughout the study program is necessary to sustain and nurture the acquired skills.

Protocol for the SELECT study: a sequential mixed methods study of the selection of UK medical students into clinical academic training

BACKGROUND

Internationally, there has been a move towards fostering diverse healthcare workforces that are representative of the patient populations they serve. Selection criteria for academic-clinicians often aim to capture skills and attributes that demonstrate both clinical and academic excellence. Currently, it is not known whether the selection criteria for early academic-clinical careers advantage or disadvantage certain ethnic or socioeconomic groups. The UK has a structured route of integrated clinical academic training with entry level training for newly qualified doctors administered through the 'Specialised Foundation Programme' which provides protected time for research within the first two years of postgraduate clinical training. In this study, we aim to identify what selection criteria are used within the UK Specialised Foundation Programme, and how these relate to demographic factors.

METHODS

We will perform a mixed methods study consisting of a document analysis of person specifications and selection criteria used in the 2024 UK Specialised Foundation Programme, and a national cross-sectional survey of current medical students in the UK. We will obtain the person specifications, selection criteria, white space (open ended questions used during shortlisting) and interview questions and mark schemes from each Specialised Unit of Applications via information available on their websites or through Freedom of Information requests. Our survey will collect information relating to demographic data, selection criteria, and perceptions of specialised foundation programme selection.

DISCUSSION

International literature has demonstrated inequity in academic markers used in selection of post-graduate clinicians and that disadvantages caused by selection can compound over time. As such it is important to understand what inequity exists within the selection of early academic-clinicians, as this can help inform more equitable selection practices and help nurture a more diverse academic-clinical workforce.

Emotional intelligence and LEGO-based communication assessments as indicators of peer evaluations

INTRODUCTION

Much attention and effort are devoted to general surgery residency applicant interview strategies and ranking. However, few interview strategies are positively associated with applicant communication performance. The purpose of this study was to assess the potential of a LEGO-based communication assessment and a standardized patient-based emotional intelligence assessment to serve as an indicator for communication and interpersonal skills among peers in residency.

METHODS

We examined general surgery residents who interviewed virtually at our institution in 2021-2022, then matriculated in 2022-2023. Residents' LEGO-based communication and emotional intelligence interview assessments were evaluated for associations with peer evaluations in intern year. A regression analysis was conducted testing the association between each assessment and peer-evaluation metrics, controlling for resident track (preliminary, categorical).

RESULTS

Performance on LEGO-based communication assessments was significantly associated with peer evaluation of cross-cover performance (B = 0.09, standard error 0.044, 95% confidence interval 0.001-0.187, P = .048), whereas emotional intelligence scores had no significant association (P = .155). In contrast, performance on the LEGO-based communication assessments were associated with peer evaluations for desirability as a coworker (B = 0.098, standard error 0.038, 95% confidence interval 0.017-0.178, P = .021), whereas emotional intelligence scores were negatively associated with desirability as a coworker (B = -0.255, standard error 0.107, 95% confidence interval -0.482 to -0.029], P = .029).

CONCLUSION

Although LEGO-based communication assessments before residency are associated with better resident peer evaluations 1 year into residency, our emotional intelligence applicant assessment during recruitment did not provide clear insight into resident performance. Future research should consider the ability of standardized assessments to predict performance.

Insights into undergraduate medical student selection tools: a systematic review and meta-analysis

PURPOSE

Evaluating medical school selection tools is vital for evidence-based student selection. With previous reviews revealing knowledge gaps, this meta-analysis offers insights into the effectiveness of these selection tools.

METHODS

A systematic review and meta-analysis were conducted applying the following criteria: peer-reviewed articles available in English, published from 2010 and which include empirical data linking performance in selection tools with assessment and dropout outcomes of undergraduate entry medical programs. Systematic reviews, meta-analyses, general opinion pieces, or commentaries were excluded. Effect sizes (ESs) of the predictability of academic and clinical performance within and by the end of the medicine program were extracted, and the pooled ESs were presented.

RESULTS

Sixty-seven out of 2,212 articles were included, which yielded 236 ESs. Previous academic achievement predicted medical program academic performance (Cohen’s d=0.697 in early program; 0.619 in end of program) and clinical exams (0.545 in end of program). Overall aptitude tests predicted academic achievement in both the early and last years (0.550 & 0.371, respectively). Within aptitude tests, verbal reasoning and quantitative reasoning best predicted academic achievement in the early program (0.704 & 0.643, respectively). Neither panel interviews, multiple mini-interviews, nor situational judgement tests (SJT) yielded statistically significant pooled ES.

CONCLUSION

Current evidence suggests that learning outcomes are predicted by previous academic achievement and aptitude tests. The predictive value of SJT and topics such as selection algorithms, features of interview (e.g., content of the questions) and the way the interviewers’ reports are used, warrant further research.

Evaluating the Role of Competency-Based Behavioral Interviewing in Holistic Medical School Admissions

PROBLEM

Holistic review is a multifaceted concept that aims to increase diversity and applicant fit with program needs by complementing traditional academic requirements with appraisal of a wider range of personal characteristics and experiences. Behavioral interviewing has been practiced and studied in human resources, business, and organizational psychology for over 50 years. Its premise is that future performance can be anticipated from past actions. However, many of the interview approaches within the holistic framework are resource intensive and logistically challenging.

APPROACH

The Vanderbilt University School of Medicine instituted a competency-based behavioral interview (CBBI) to augment the selection process in 2012. Behavioral interviews are based on key competencies needed for entering students and require applicants to reflect on their actual experiences and what they learned from them. The authors reviewed 5 years of experience (2015-2019) to evaluate how CBBI scores contributed to the overall assessment of applicants for admission.

OUTCOMES

The final admission committee decision for each applicant was determined by reviewing multiple factors, with no single assessment determining the final score. The CBBI and summary interview scores showed a strong association ( P < .005), suggesting that the summary interviewer, who had access to the full applicant file, and the CBBI interviewer, who did not, assessed similar strengths despite the 2 different approaches, or that the strengths assessed tracked in the same direction. Students whose 2 interview scores were not aligned were less likely to be accepted to the school.

NEXT STEPS

The review raised awareness about the cultural aspects of interpreting the competencies and the need to expand our cultural framework throughout interviewer training. Findings indicate that CBBIs have the potential to reduce bias related to overreliance on standardized metrics; however, additional innovation and research are needed.

Examining interviewer bias in medical school admissions: The interplay between applicant and interviewer gender and its effects on interview outcomes

Selection interviews have long been integral to medical school admissions, yet their limited predictive validity and susceptibility to bias raise concerns. This study delves into potential interviewer bias within the dynamics of interviewee and interviewer gender. We analyze a dataset of 5,200 applicants and over 370 selection committees engaged in semi-structured interviews from 2006 to 2019 at a large German medical school with multiple linear and non-linear regression analyses. Our findings reveal that all-female committees tended to award male candidates, on average, one point more than their female counterparts, significantly enhancing the chances of submission for male applicants despite lower academic grades, which constituted 51% of the selection process points. All-male and mixed-gender committees exhibited similar ratings for both genders. The role of valuing voluntary services emerged prominently: all-male and mixed committees acknowledged women's volunteer work but not men's, while all-female committees demonstrated the opposite pattern. Our results attribute variations in interview outcomes to the absence of standardization, such as insufficient interviewer training, divergent rating strategies, variations in interviewer experience, and imbalances in candidate allocation to selection committees, rather than to a "gender bias", for example by favoritism of males because of their gender.

Improving selection procedures in health professions education from the applicant perspective: an interview study

INTRODUCTION

Applicant perceptions of selection impact motivation and performance during selection, and student diversity. However, in-depth insight into which values underly these perceptions is lacking, creating challenges for aligning selection procedures with applicant perceptions. This qualitative interview study aimed to identify values applicants believe should underlie selection, and how, according to applicants, these values should be used to make specific improvements to selection procedures in undergraduate health professions education (HPE).

METHODS

Thirty-one applicants to five undergraduate HPE programs in the Netherlands participated in semi-structured interviews using Appreciative Inquiry, an approach that focuses on what goes well to create vision for improvement, to guide the interviews. Transcriptions were analyzed using thematic analysis, adopting a constructivist approach.

RESULTS

Applicants' values related to the aims of selection, the content of selection, and the treatment of applicants. Applicants believed that selection procedures should aim to identify students who best fit the training and profession, and generate diverse student populations to fulfill societal needs. According to applicants, the content of selection should be relevant for the curriculum and profession, assess a comprehensive set of attributes, be of high quality, allow applicants to show who they are, and be adapted to applicants' current developmental state. Regarding treatment, applicants believed that selection should be a two-way process that fosters reflection on study choice, be transparent about what applicants can expect, safeguard applicants' well-being, treat all applicants equally, and employ an equitable approach by taking personal circumstances into account. Applicants mentioned specific improvements regarding each value.

DISCUSSION

Applicants' values offer novel insights into what they consider important preconditions for the design of selection procedures. Their suggested improvements can support selection committees in better meeting applicants' needs.

New evidence on the validity of the selection methods for recruitment to general practice training: a cohort study

BACKGROUND

Selection into UK-based GP training has used the Multi-Specialty Recruitment Assessment (MSRA) and a face-to-face selection centre (SC). The MSRA comprises of a situational judgement test and clinical problem-solving test. The SC was suspended during the COVID-19 pandemic. Evidence is needed to guide national and international selection policy.

AIM

To evaluate the validity of GP training selection.

DESIGN & SETTING

A retrospective cohort study using data from UK-based national recruitment to GP training, from 2015-2021.

METHOD

Data were available for 32 215 GP training applicants. The ability of scores from the specialty selection process to predict subsequent performance in the Clinical Skills Assessment (CSA) of the Membership of the Royal College of General Practitioners examination was modelled using path analysis. The effect sizes for sex, professional family background, and world region of qualification were estimated.

RESULTS

All component scores of the selection process demonstrated statistically significant independent relationships with CSA performance (P<0.001), thus establishing their predictive validity. All were sensitive to demographic factors. The SC scores had the weakest relationship with future CSA performance. However, for candidates with MSRA scores below the lowest quartile, the relative contribution of the SC scores to predicting CSA performance was similar to that observed for MSRA components.

CONCLUSION

The MSRA has predictive validity in this context. Re-instituting an SC for those with relatively low MSRA scores should be considered. However, the relative costs and potential advantages and disadvantages should be carefully weighed.

How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal

BACKGROUND

Peer review represents a cornerstone of the scientific process, yet few studies have evaluated its association with scientific impact. The objective of this study is to assess the association of peer review scores with measures of impact for manuscripts submitted and ultimately published.

METHODS

3173 manuscripts submitted to Regional Anesthesia & Pain Medicine (RAPM) between August 2018 and October 2021 were analyzed, with those containing an abstract included. Articles were categorized by topic, type, acceptance status, author demographics and open-access status. Articles were scored based on means for the initial peer review where each reviewer's recommendation was assigned a number: 5 for 'accept', 3 for 'minor revision', 2 for 'major revision' and 0 for 'reject'. Articles were further classified by whether any reviewers recommended 'reject'. Rejected articles were analyzed to determine whether they were subsequently published in an indexed journal, and their citations were compared with those of accepted articles when the impact factor was <1.4 points lower than RAPM's 5.1 impact factor. The main outcome measure was the number of Clarivate citations within 2 years from publication. Secondary outcome measures were Google Scholar citations within 2 years and Altmetric score.

RESULTS

422 articles met inclusion criteria for analysis. There was no significant correlation between the number of Clarivate 2-year review citations and reviewer rating score (r=0.038, p=0.47), Google Scholar citations (r=0.053, p=0.31) or Altmetric score (p=0.38). There was no significant difference in 2-year Clarivate citations between accepted (median (IQR) 5 (2-10)) and rejected manuscripts published in journals with impact factors >3.7 (median 5 (2-7); p=0.39). Altmetric score was significantly higher for RAPM-published papers compared with RAPM-rejected ones (median 10 (5-17) vs 1 (0-2); p<0.001).

CONCLUSIONS

Peer review rating scores were not associated with citations, though the impact of peer review on quality and association with other metrics remains unclear.

Measuring personal characteristics in applicants to German medical schools: Piloting an online Situational Judgement Test with an open-ended response format

Objectives

Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada ("Casper").

Methods

German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).

Results

The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (α=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (d=0.37) or did not have a migration background (d=0.40) received higher scores. Casper scores correlated with HAM-SJT (r=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (r=-.15), the TMS (r=.18), and HAM-Nat logical reasoning scores (r=.23).

Conclusion

This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.

Investigating the internal structure of multiple mini interviews-A perspective from Pakistan

BACKGROUND

Healthcare professionals require many personal attributes in addition to cognitive abilities and psychomotor skills for competent practice. Multiple Mini- Interviews are being employed globally to assess personality attributes of candidates for selection in health professions education at all level of entry; these attributes are namely, communication skills, critical thinking, honesty, responsibility, health advocacy, empathy and sanctity of life. Considering the high stakes involved for students, faculty, institutions and the society, rigorous quality assurance mechanisms similar to those used for student assessment must be employed for student selection, throughout the continuum of medical education. It is a difficult undertaking as these psychological constructs are difficult to define and measure. Though considered to yield reliable and valid scores, studies providing multiple evidences of internal structure especially dimensionality of Multiple Mini-Interviews are sparse giving rise to questions if they are measuring a single or multiple constructs and even if they are measuring what they are purported to be measuring.

OBJECTIVE

The main objective is to provide statistical support of the multi-dimensional nature of our Multiple Mini Interviews, hypothesized a-priori, through CFA. Another objective is to provide multiple evidences for the internal structure. Our study highlights the link between content and internal structure evidences of the constructs, thus establishing that our Multiple Mini Interviews measure what they were intended to measure.

METHOD

After securing permission from the Institutional review board, an a-priori seven factor-model was hypothesized based on the attributes considered most essential for the graduating student of the institution. After operationally defining the attributes through extensive literature search, scenarios were constructed to assess them. A 5-point rating scale was used to rate each item on the station. A total 259 students participated in the multiple mini interviews over a period of three days. A training workshop had been arranged for the participating faculty.

RESULTS

The reliability coefficient using Cronbach's alpha were calculated (range from 0.73 to 0.94), Standard Error of Measurement (ranged from 0.80 to1.64), and item to station-total correlation ranged from 0.43-0.50 to 0.75-0.83. Inter-station correlation was also determined. Confirmatory factor analysis endorsed the results of Exploratory factor analysis in the study revealing a seven model fit with multiple indices of Goodness-of-fit statistics such as Root mean square error of approximation (RMSEA) value 0.05, Standardized root mean square residual (SRMR) value with less than 0.08. All these indices showed that model fit is good. The Confirmatory factor analysis confirmed the multi-dimensional nature of our MMIs and also confirmed that our stations measured the attributes that they were supposed to measure.

CONCLUSION

This study adds to the validity evidence of Multiple Mini-Interviews, in selection of candidates, with required personality traits for healthcare profession. It provides the evidence for the multi-dimensional structure of Multiple Mini interviews administered with multiple evidences for its internal structure and demonstrates the independence of different constructs being measured.

Effect of Performance-Based Nonfinancial Incentives on Data Quality in Individual Medical Records of Institutional Births: Quasi-Experimental Study

BACKGROUND

Despite the potential of routine health information systems in tackling persistent maternal deaths stemming from poor service quality at health facilities during and around childbirth, research has demonstrated their suboptimal performance, evident from the incomplete and inaccurate data unfit for practical use. There is a consensus that nonfinancial incentives can enhance health care providers' commitment toward achieving the desired health care quality. However, there is limited evidence regarding the effectiveness of nonfinancial incentives in improving the data quality of institutional birth services in Ethiopia.

OBJECTIVE

This study aimed to evaluate the effect of performance-based nonfinancial incentives on the completeness and consistency of data in the individual medical records of women who availed institutional birth services in northwest Ethiopia.

METHODS

We used a quasi-experimental design with a comparator group in the pre-post period, using a sample of 1969 women's medical records. The study was conducted in the "Wegera" and "Tach-armacheho" districts, which served as the intervention and comparator districts, respectively. The intervention comprised a multicomponent nonfinancial incentive, including smartphones, flash disks, power banks, certificates, and scholarships. Personal records of women who gave birth within 6 months before (April to September 2020) and after (February to July 2021) the intervention were included. Three distinct women's birth records were examined: the integrated card, integrated individual folder, and delivery register. The completeness of the data was determined by examining the presence of data elements, whereas the consistency check involved evaluating the agreement of data elements among women's birth records. The average treatment effect on the treated (ATET), with 95% CIs, was computed using a difference-in-differences model.

RESULTS

In the intervention district, data completeness in women's personal records was nearly 4 times higher (ATET 3.8, 95% CI 2.2-5.5; P=.02), and consistency was approximately 12 times more likely (ATET 11.6, 95% CI 4.18-19; P=.03) than in the comparator district.

CONCLUSIONS

This study indicates that performance-based nonfinancial incentives enhance data quality in the personal records of institutional births. Health care planners can adapt these incentives to improve the data quality of comparable medical records, particularly pregnancy-related data within health care facilities. Future research is needed to assess the effectiveness of nonfinancial incentives across diverse contexts to support successful scale-up.

Detecting Artificial Intelligence-Generated Personal Statements in Professional Physical Therapist Education Program Applications: A Lexical Analysis

OBJECTIVE

The objective of this study was to compare the lexical sophistication of personal statements submitted by professional physical therapist education program applicants with those generated by OpenAI's Chat Generative Pretrained Transformer (ChatGPT).

METHODS

Personal statements from 152 applicants and 20 generated by ChatGPT were collected, all in response to a standardized prompt. These statements were coded numerically, then analyzed with recurrence quantification analyses (RQAs). RQA indices including recurrence, determinism, max line, mean line, and entropy were compared with t-tests. A receiver operating characteristic curve analysis was used to examine discriminative validity of RQA indices to distinguish between ChatGPT and human-generated personal statements.

RESULTS

ChatGPT-generated personal statements exhibited higher recurrence, determinism, mean line, and entropy values than did human-generated personal statements. The strongest discriminator was a 13.04% determinism rate, which differentiated ChatGPT from human-generated writing samples with 70% sensitivity and 91.4% specificity (positive likelihood ratio = 8.14). Personal statements with determinism rates exceeding 13% were 8 times more likely to have been ChatGPT than human generated.

CONCLUSION

Although RQA can distinguish artificial intelligence (AI)-generated text from human-generated text, it is not absolute. Thus, AI introduces additional challenges to the authenticity and utility of personal statements. Admissions committees along with organizations providing guidelines in professional physical therapist education program admissions should reevaluate the role of personal statements in applications.

IMPACT

As AI-driven chatbots like ChatGPT complicate the evaluation of personal statements, RQA emerges as a potential tool for admissions committees to detect AI-generated statements.

Incorporating a situational judgement test in residency selections: clinical, educational and organizational outcomes

BACKGROUND

Computer-based assessment for sampling personal characteristics (Casper), an online situational judgement test, is a broad measure of personal and professional qualities. We examined the impact of Casper in the residency selection process on professionalism concerns, learning interventions and resource utilization at an institution.

METHODS

In 2022, admissions data and information in the files of residents in difficulty (over three years pre- and post- Casper implementation) was used to determine the number of residents in difficulty, CanMEDS roles requiring a learning intervention, types of learning interventions (informal learning plans vs. formal remediation or probation), and impact on the utilization of institutional resource (costs and time). Professionalism concerns were mapped to the 4I domains of a professionalism framework, and their severity was considered in mild, moderate, and major categories. Descriptive statistics and between group comparisons were used for quantitative data.

RESULTS

In the pre- and post- Casper cohorts the number of residents in difficulty (16 vs. 15) and the number of learning interventions (18 vs. 16) were similar. Professionalism concerns as an outcome measure decreased by 35% from 12/16 to 6/15 (p < 0.05), were reduced in all 4I domains (involvement, integrity, interaction, introspection) and in their severity. Formal learning interventions (15 vs. 5) and informal learning plans (3 vs. 11) were significantly different in the pre- and post-Casper cohorts respectively (p < 0.05). This reduction in formal learning interventions was associated with a 96% reduction in costs f(rom hundreds to tens of thousands of dollars and a reduction in time for learning interventions (from years to months).

CONCLUSIONS

Justifiable from multiple stakeholder perspectives, use of an SJT (Casper) improves a clinical performance measure (professionalism concerns) and permits the institution to redirect its limited resources (cost savings and time) to enhance institutional endeavors and improve learner well-being and quality of programs.

How professional and academic pre-qualifications relate to success in medical education: Results of a multicentre study in Germany

OBJECTIVE

Every year, many applicants want to study medicine. Appropriate selection procedures are needed to identify suitable candidates for the demanding curriculum. Although research on medical school admissions has shown good predictive validity for cognitive selection methods (undergraduate GPA, aptitude tests), the literature on applicants with professional and/or academic experience prior to entering medical school remains slim. In our study, we therefore aimed to examine the association between academic success in medical school and having previously completed vocational training in the medical field, voluntary service (≥11 months) or an academic degree.

METHODS

Data were collected in a multicentre, cross-sectional study at five medical schools in Germany (Baden-Wuerttemberg) from students during medical school (i.e. 3rd-, 6th-, and 10th-semester and final-year students). Academic success was assessed according to scores on the first and second state examinations, the total number of examinations repeated and the number of semesters beyond the standard period of study. For the analysis we calculated ordinal logistic regression models for each outcome variable of academic success.

RESULTS

A total of N = 2,370 participants (response rate: RR = 47%) participated in the study. Having completed vocational training was associated with a higher amount of repeated examinations (small effect), while having an academic degree was associated with worse scores on the second state examination (medium effect). No significant association emerged between voluntary service and academic success.

CONCLUSION

The results indicate that professional and academic pre-qualifications pose no advantage for academic success. Possible associations with the financing of study and living conditions of students with pre-qualifications were analysed and discussed in an exploratory manner. However, the operationalisation of academic success from objective and cognitive data should be critically discussed, as the benefits of prior experience may be captured by personal qualities rather than examination results.

Follow the policy: An actor network theory study of widening participation to medicine in two countries

INTRODUCTION

The slow pace of change in respect of increasing the diversity of medical students suggests powerful actors are reproducing practices to support the status quo. Opening up medicine to embrace diversity thus requires the deconstruction of entrenched processes and practices. The first step in doing so is to understand how the actor-network of widening participation and access to medicine (WP/WA) is constructed. Thus, here we examine how the connections among actors in WP/WA in two different networks are assembled.

METHODS

A comparative case study using documents (n = 7) and interviews with staff and students (n = 45) from two medical schools, one United Kingdom and one Australian, was used. We used Callon's moments of translation (problematisation, interessement/operationalisation, enrolment, mobilisation) to map the network of actors as they are assembled in relation to one another. Our main actant was institutional WP to medicine policy (actor-as-policy).

RESULTS

Our actor-as-policy introduced five other actors: the medical school, medical profession, high schools, applicants and medical school staff. In terms of problematisation, academic excellence holds firm as the obligatory passage point and focal challenge for all actors in both countries. The networks are operationalised via activities such as outreach and admissions policy (e.g., affirmative action is apparent in Australia but not the UK). High schools play (at best) a passive role, but directed by the policy, the medical schools and applicants work hard to achieve WP/WA to medicine. In both contexts, staff are key mobilisers of WP/WA, but with little guidance in how to enact policy. In Australia, policy drivers plus associated entry structures mean the medical profession exerts significant influence.

CONCLUSIONS

Keeping academic excellence as the obligatory passage point to medical school shapes the whole network of WP/WA and perpetuates inequality. Only by addressing this can the network reconfigure.

Patient Engagement in Medical Trainee Selection: A Scoping Review

PURPOSE

The stakes of medical trainee selection are high, making it ironic and somewhat paradoxical that patients and the public often get little say in selection practices. The authors sought to undertake a knowledge synthesis to uncover what is known about patient engagement across the medical trainee selection continuum.

METHOD

The authors conducted a scoping review aimed at exploring the current state of practice and research on patient engagement in medical trainee selection in 2017-2021. MeSH headings and keywords were used to capture patient, community, and standardized patient engagement in selection processes across multiple health professions. The authors employed broad inclusion criteria and iteratively refined the corpus, ultimately, limiting study selection to those reporting engagement of actual patients in selection within medicine, but maintaining a broad focus on any patient contributions across the entire selection continuum. The Cambridge Framework was adapted and used to organize the included studies.

RESULTS

In total, 2,858 abstracts were reviewed, and ultimately, 28 papers were included in the final corpus. The included studies were global but nascent. Most of the literature on this topic appears in the form of individual projects advocating for patient engagement in selection rather than cohesive programs with empirical exploration of patient engagement in selection. Job analysis methodology was particularly prominent for incorporating the patient voice into identifying competencies of relevance to selection. Direct patient engagement in early selection activities allowed the patient voice to assist candidates in determining their fit for medicine.

CONCLUSIONS

Patient engagement has not been made a specific focus of study in its own right, leading the authors to encourage researchers to turn their lens more directly on patient engagement to explore how it complements the professional voice in medical trainee selection.

'Selection into training will always be an inexact process': A survey of Directors of Physician Education on selection into Basic Physician Training in Australia and New Zealand

BACKGROUND

Despite being one of the largest medical specialty training programmes in Australasia, there is no standardised method for selection into Basic Physician Training (BPT), and limited data exist regarding current practices.

AIMS

To address existing knowledge gaps, we aimed to create a 'snapshot' of current BPT selection practices and explore the perspectives of Directors of Physician Education (DPEs) regarding trainee selection.

METHODS

An electronic survey of DPEs from adult and paediatric medicine BPT sites in Australia and New Zealand was undertaken in January-February 2022. A combination of free text, multiple-choice and yes/no answers were analysed using descriptive statistics and qualitative content analysis.

RESULTS

A total of 70 responses were received, achieving a response rate of 35% (70/198). Selection practices were found to be heterogenous across BPT sites. Respondents had varying opinions regarding the utility of selection tools and desirable candidate attributes. A heavy reliance upon interviews and the reported use of subjective assessments raise concerns for selection process bias.

CONCLUSION

BPT sites should critically evaluate their selection methods, and more research in this field is needed to establish best practice.

'Click, I Guess I'm Done': Applicants' and Assessors' Experiences Transitioning to a Virtual Multiple Mini Interview Format

Introduction

During the COVID-19 pandemic, medical schools were forced to suspend in-person interviews and transition to a virtual Multiple Mini Interview (vMMI) format. MMIs typically comprise multiple short assessments overseen by assessors, with the aim of measuring a wide range of non-cognitive competencies. The adaptation to vMMI required medical schools to make swift changes to their MMI structure and delivery. In this paper, we focus on two specific groups greatly impacted by the decision to transition to vMMIs: medical school applicants and MMI assessors.

Methods

We conducted an interpretive qualitative study to explore medical school applicants' and assessors' experiences transitioning to an asynchronous vMMI format. Ten assessors and five medical students from one Canadian medical school participated in semi-structured interviews. Data was analyzed using a thematic analysis framework.

Results

Both applicants and assessors shared a mutual feeling of longing and nostalgia for an interview experience that, due to the pandemic, was understandably adapted. The most obvious forms of loss experienced - albeit in different ways - were: 1) human connection and 2) missed opportunity. Applicants and assessors described several factors that amplified their grief/loss response. These were: 1) resource availability, 2) technological concerns, and 3) the virtual interview environment.

Discussion

While virtual interviewing has obvious advantages, we cannot overlook that asynchronous vMMIs do not lend themselves to the same caliber of interaction and camaraderie as experienced in in-person interviews. We outline several recommendations medical schools can implement to enhance the vMMI experience for applicants and assessors.

Residency Selection in Otolaryngology: Past, Present, & Future

OBJECTIVES

To examine the otolaryngology residency selection process, including past experiences based on the medical literature and survey analysis of our present practices to generate recommendations for future selection system design.

METHODS

A mixed-methods study, including a scoping review and a cross-sectional survey, was completed. Four databases were assessed for articles on otolaryngology residency selection published from January 1, 2016 through December 31, 2020. A 36-question survey was developed and distributed to 114 otolaryngology program directors. Descriptive and thematic analysis was performed.

RESULTS

Ultimately, 67 of 168 articles underwent data abstraction and assessment. Three themes surfaced during the analysis: effectiveness, efficiency, and equity. Regarding the survey, there were 62 participants (54.4% response rate). The three most important goals for the selection process were: (1) to fit the program culture, (2) to make good colleagues, and (3) to contribute to the program's diversity. The three biggest 'pain points' were as follows: (1) Large volume of applications, (2) Lack of reliable information about personal characteristics, and (3) Lack of reliable information about a genuine interest in the program.

CONCLUSIONS

Within this study, the depth and breadth of the literature on otolaryngology residency selection have been synthesized. Additionally, baseline data on selection practices within our specialty has been captured. With an informed understanding of our past and present, we can look to the future. Built upon the principles of person-environment fit theory, our proposed framework can guide research and policy discussions regarding the design of selection systems in otolaryngology, as we work to achieve more effective, efficient, and equitable outcomes.

LEVEL OF EVIDENCE

N/A Laryngoscope, 133:2929-2941, 2023.

A Comparison of the Academic Achievement at the End of the Medicine Undergraduate Degree Program Between Students Who Only Used the University Admission Test and Those Who Used the University Admission Test Plus Marks from the High School National Exam (ENEM) at a Single Brazilian Center

Purpose

The role of marks in the University Admission Test (UAT) plus the marks from pre-university academic records in predicting academic achievement at the end of the Medicine undergraduate degree program is not completely known. This study was undertaken to compare the performance of marks in the UAT alone with those of the UAT plus marks from the National High School Exam (ENEM in Brazil) regarding students' outcomes at the end of the Medicine undergraduate degree program.

Methods

Fifty-one (51) students from the last semester (12th) of our Medicine undergraduate degree program were included in the study. They were divided into a group of those who used the marks obtained in the UAT plus the marks obtained in the ENEM (ENEM group, n=9), and those who only used the marks in the UAT (non-ENEM group, n=42). We compared the academic achievement of the non-ENEM group with that of the ENEM group regarding the mean marks obtained in the clerkship, in the Progress Test (PT), and in the Objective Structured Clinical Examination (OSCE).

Results

The mean scores obtained in the disciplines of the clerkship were higher in the non-ENEM group compared to the ENEM group (7.32 ± 0.41 vs 6.98 ± 0.31, p= 0.01). Both groups obtained similar mean marks in the OSCE and in the PT. A moderate correlation was observed between the marks in the clerkship with those of the UAT from the non-ENEM group (p=0.00006; r=0.45).

Conclusion

Marks of the UAT alone appear to be associated with a higher academic achievement in the clerkship than marks of the UAT plus scores obtained from the ENEM at the end of the Medicine undergraduate degree program.

Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Selection tools and student diversity in health professions education: a multi-site study

Student diversity in health professions education (HPE) can be affected by selection procedures. Little is known about how different selection tools impact student diversity across programs using different combinations of traditional and broadened selection criteria. The present multi-site study examined the chances in selection of subgroups of applicants to HPE undergraduate programs with distinctive selection procedures, and their performance on corresponding selection tools. Probability of selection of subgroups (based on gender, migration background, prior education, parental education) of applicants (N = 1935) to five selection procedures of corresponding Dutch HPE undergraduate programs was estimated using multilevel logistic regression. Multilevel linear regression was used to analyze performance on four tools: prior-education grade point average (pe-GPA), biomedical knowledge test, curriculum-sampling test, and curriculum vitae (CV). First-generation Western immigrants and applicants with a foreign education background were significantly less likely to be selected than applicants without a migration background and with pre-university education. These effects did not vary across programs. More variability in effects was found between different selection tools. Compared to women, men performed significantly poorer on CVs, while they had higher scores on biomedical knowledge tests. Applicants with a non-Western migration background scored lower on curriculum-sampling tests. First-generation Western immigrants had lower CV-scores. First-generation university applicants had significantly lower pe-GPAs. There was a variety in effects for applicants with different alternative forms of prior education. For curriculum-sampling tests and CVs, effects varied across programs. Our findings highlight the need for continuous evaluation, identifying best practices within existing tools, and applying alternative tools.

Effects of candidates' demographics and evaluation of the virtual Multiple Mini Interview (vMMI) as a tool for selection into paediatric training in Queensland

INTRODUCTION

The Queensland Basic Paediatric Training Network (QBPTN) is responsible for the selection of candidates into paediatric training in Queensland. The COVID-19 pandemic necessitated interviews to be conducted 'virtually' as virtual Multiple-Mini-Interviews (vMMI). The study aimed to describe the demographic characteristics of candidates applying for selection into paediatric training in Queensland, and to explore their perspectives and experiences with the vMMI selection tool.

METHODOLOGY

The demographic characteristics of candidates and their vMMI outcomes were collected and analysed with a mixed methods approach. The qualitative component was comprised of seven semi-structured interviews with consenting candidates.

RESULTS

Seventy-one shortlisted candidates took part in vMMI and 41 were offered training positions. The demographic characteristics of candidates at various stages of selection were similar. The mean vMMI scores were not statistically different between candidates from the Modified Monash Model 1 (MMM1) location and others [mean (SD): 43.5 (5.1) versus 41.7 (6.7), respectively, p = 0.26]. However, there was a statistically significant difference (p value 0.03) between being offered and not offered a training position for candidates from MMM2 and above. The analysis of the semi-structured interviews suggested that candidate experiences of the vMMI were influenced by the quality of the management of the technology used. Flexibility, convenience, and reduced stress were the main factors that influenced candidates' acceptance of vMMI. Perceptions of the vMMI process focused on the need to build rapport and facilitate communication with the interviewers.

DISCUSSION

vMMI is a viable alternative to face-to-face (FTF) MMI. The vMMI experience can be improved by facilitating enhanced interviewer training, by making provision for adequate candidate preparation and by having contingency plans in place for unexpected technical challenges. Given government priorities in Australia, the impact of candidates' geographical location on the vMMI outcome for candidates from MMM >1 location needs to be further explored.

Effects of the alternative medical curriculum at the Hannover Medical School on length of study and academic success

Objective

The model curriculum HannibaL (Hannoversche integrierter berufsorientierter und adaptiver Lehrplan) differs significantly from other medical study programs in Germany in terms of its structure with which, among other factors, the Hannover Medical School (MHH) saw an opportunity to positively influence the length of study. We investigate how the length of medical study is influenced by the curriculum's structure and whether this has any impact on academic success.

Methods

We use data from over 2,500 students who studied medicine at MHH between 2011 and 2021. We measure study time as the number of years which pass until completion of the respective study phases and academic success as the grades achieved on final exams.

Results

Since they more often fail or postpone exams, students admitted based on special quotas (VQ) or a waiting list (WQ) need significantly more time to complete the first study phase (M1) compared to students who were admitted based on a selection process (AdH) or who belong to the "best school graduates" quota (AQ) because they earned the highest scores on the final secondary school exam. Yet, students from all admission groups reach the written state exam (M2) almost simultaneously. In HannibaL, WQ and VQ manage to catch up on delays from M1 with no negative impact on success in M2. In general, however, VQ and WQ achieve lower grades and drop out more often than students from AQ and AdH.

Discussion

In the regular curriculum, students can only proceed with their studies once M1 has been entirely completed. HannibaL, on the other hand, allows for the catching up of delays from the first two years of study by integrating both study phases. The curricular structure thus accommodates students with lower academic performance who accumulate delays early on in their studies. By contrast, delays in the AQ and AdH groups arise during the second phase of study (M2).

Impact of medical students' socioeconomic backgrounds on medical school application, admission and migration in Japan: a web-based survey

OBJECTIVES

The aim of this study was to clarify and compare the socioeconomic backgrounds of medical students with those of other health professions and non-health faculty students in an era of increasing inequity in Japanese society.

DESIGN

This was a quantitative nationwide study. Data were collected by a cross-sectional web-based anonymous questionnaire.

SETTING

Data from years 3 and 4 medical, health professions and non-health faculty students across Japan were collected in 2021.

PARTICIPANTS

Participants were 1991 students from medical schools, 224 from dental, 419 from pharmacy, 326 from nursing, 144 from other health professions and 207 from non-health faculties.

RESULTS

The proportion of high-income families (>18 million yen: ca. US$140 000) among medical students was 25.6%, higher than that of pharmacy (8.7%) and nursing students (4.1%) (p<0.01). One-third of medical students had a physician parent, more common than in non-medical students (p<0.01). Students who only applied to public medical schools and a regional quota 'Chiiki-waku' students with scholarship had lower family income and physician parents compared with those who applied to private medical schools (p<0.01), but they still had higher physician parents compared with non-medical students (p<0.01). Logistic regression revealed that having a physician parent (p<0.01), aspiring to the present profession during elementary school (p<0.01) and private upper secondary school graduation (p<0.01) predicted the likelihood of studying medicine. There were regional differences of backgrounds among medical students, and 80% of medical students with urban backgrounds intended to work in urban localities after graduation.

CONCLUSIONS

This study provides evidence that medical students in Japan hail from urban and higher income classes and physicians' families. This finding has implications for the health workforce maldistribution in Japan. Widening the diversity of medical students is essential for solving physician workforce issues and meeting broad healthcare needs.

What does shared decision making ask from doctors? Uncovering suppressed qualities that could improve person-centered care

OBJECTIVE

Shared decision making (SDM) is infrequently seen in clinical practice despite four decades of efforts. We propose a need to explore what SDM asks from doctors in terms of enabling competencies and necessary, underlying qualities, and how these can be nurtured or suppressed in medical training.

DISCUSSION

Key SDM tasks call for doctors to understand communication and decision mechanisms to carry them out well, including reflecting on what they know and do not know, considering what to say and how, and listening unprejudiced to patients. Different doctor qualities can support accomplishing these tasks; humility, flexibility, honesty, fairness, self-regulation, curiosity, compassion, judgment, creativity, and courage, all relevant to deliberation and decision making. Patient deference to doctors, lack of supervised training opportunities with professional feedback, and high demands in the work environment may all inflate the risk of only superficially involving patients.

CONCLUSIONS

We have identified ten professional qualities and related competencies required for SDM, with each to be selected based on the specific situation. The competencies and qualities need to be preserved and nurtured during doctor identity building, to bridge the gap between knowledge, technical skills, and authentic efforts to achieve SDM.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

The association between how medical students were selected and their perceived stress levels in Year-1 of medical school

BACKGROUND

The prevalence of medical students' mental distress is high. While schools apply various methods to select a well-performing and diverse student population, little is known about the association between different selection methods and the well-being of these students during medical school. The present retrospective multi-cohort study assessed whether students selected by high grades, assessment, or weighted lottery showed different stress perception levels in Year-1 of medical school.

METHODS

Of 1144 Dutch Year-1 medical students, 650 (57%) of the cohorts 2013, 2014, and 2018 who were selected by high grades, assessment, or weighted lottery completed a stress perception questionnaire (PSS-14). A multilevel regression analysis assessed the association between selection method (independent variable) and stress perception levels (dependent variable) while controlling for gender and cohort. In a post-hoc analysis, academic performance (optimal vs. non-optimal) was included in the multilevel model.

RESULTS

Students selected by assessment (B = 2.25, p < .01, effect size (ES) = small) or weighted lottery (B = 3.95, p < .01, ES = medium) had higher stress perception levels than students selected by high grades. Extending the regression model with optimal academic performance (B=-4.38, p < .001, ES = medium), eliminated the statistically significant difference in stress perception between assessment and high grades and reduced the difference between weighted lottery and high grades from 3.95 to 2.45 (B = 2.45, p < .05, ES = small).

CONCLUSIONS

Selection methods intended to create a diverse student population - assessment and lottery - are associated with higher stress perception levels in Year-1 of medical school. These findings offer medical schools insights into fulfilling their responsibility to take care of their students' well-being.

Admission into healthcare education in France: Half-baked reform that further complicates the system

France has undertaken the most ambitious reform of healthcare education in more than 10 years. It has resulted in a hybrid system with multiple pathways, granting admission into the healthcare professions after competitive exams. The reform continues the trend to increase the quotas limiting the number of second year healthcare students, and also creates new local access options to healthcare education. However, the heterogeneity in implementation has led, in conjunction with the difficulties caused by the Covid-19 pandemic, to great dismay among students and parents. This article seeks to outline the historical underpinnings of the reform program(s) and argues that the core question - selecting students from the very high number of candidates in a fair and effective manner - remains largely unresolved.

What is the evidence for biology as the 'heart of eligibility' to study medicine? A retrospective analysis

The first, sometimes the only, selection tool for entry into undergraduate medicine is prior educational attainment (PEA). This is often further specified to include certain subjects, for example, biology is a prerequisite for entry into medicine in many Asian countries. However, there seems no clear evidence base for this prerequisite. Our aim, therefore, was to carry out a retrospective quantitative study comparing the performances of five cohorts of students (2015-2019 entry; n = 588) with and without biology PEA in Years 1 and 2 Bachelor of Medicine and Bachelor of Surgery (MBBS) integrated written assessments (n = 3) and anatomy practical examinations (APE) (n = 5). The study was conducted at one of Singapore's three medical schools. Data were analyzed using independent t-tests and Mann-Whitney U with p values of less than 0.05 were considered significant. There were no significant differences in performance on any Years 1 or 2 integrated written assessments. Similarly, in one of the APE, a significant difference was found for one cohort (academic year [AY] 2015-2016) out of five assessments. These results suggest that having a prior biology qualification does not make a difference in assessment performance in the early years of medical school. This information may help stakeholders and admissions committees decide whether biology is required for medical school entrance.

CARDA: Guiding document analyses in health professions education research

INTRODUCTION

Documents, from policies and procedures to curriculum maps and examination papers, structure the everyday experiences of health professions education (HPE), and as such can provide a wealth of empirical information. Document analysis (DA) is an umbrella term for a range of systematic research procedures that use documents as data.

METHODS

A meta-study review was conducted with the aims of describing the current state of DA in HPE, guiding researchers engaging in DA and improving methodological, analytical and reporting rigour. Structured searches were conducted, returns were filtered for inclusion and the 115 remaining articles were critically analysed for their use of DA methods and methodologies.

RESULTS

There was a significant increase in the number of articles reporting the use of DA over time. Sixty-three articles were single method (DA only), while the others were mixed methods research (MMR). Overall, there were major lacunae in terms of why documents were used, how documents were identified, what the authors did and what they found from the documents. This was particularly apparent in MMR where DA reporting was typically poorer than the reporting of other methods in the same paper.

DISCUSSION

Given these many lacunae, a framework for reporting on DA research was developed to facilitate rigorous DA research and transparent, complete and accurate reporting of the same, to help readers assess the trustworthiness of the findings from document use and analysis in HPE and, potentially, other domains. It was also noted that there are gaps in HPE knowledge that could be addressed through DA, particularly where documents are conceptualised as more than passive holders of information. Scholars are encouraged to reflect more deeply on the applications and practices of DA, with the ultimate aim of ensuring more substantive and more rigorous use of documents for understanding and constructing meaning in our field.

[Multimodal selection of medical students: The predictive power of individual process components in the two-stage selection process at Witten/Herdecke University (UW/H)]

OBJECTIVE

According to the legislator's ideas, the selection of medical students in Germany should no longer be based on the Abitur grade alone. This approach has already been implemented in the two-stage selection process at Witten/Herdecke University (UW/H) using several criteria. On the one hand, the present study aims to determine the prognostic value of the procedural components for the overall performance on the selection day. On the other hand, the different strategies of the applicants in processing the written task (phase 1) will be examined with regard to their application success.

METHODOLOGY

Data on applications for the summer semester 2020 (N = 819 phase 1; N = 233 phase 2) were available retrospectively. A stepwise regression analysis was conducted to determine the predictive power of each procedural component. Using a summary content analysis, the four essays from the applicants' motivation letters were structured and categories were identified, and an extreme group comparison (Group 1: Not invited; Group 2: University acceptance; N = 60 essays) was conducted.

RESULTS

As the stepwise regression analysis shows, the individual biographical interview emerged as the strongest predictor in terms of overall performance, followed by lecture, group interview, and multiple mini interviews. Content analysis extracted content and scaling categories for the individual essays, as well as an additional meta-category (Impression Management, IM). Successful applicants demonstrated, among other things, better judgment skills, more sophisticated reasoning skills, and an internalized role model as a physician. In addition, they used defensive IM strategies, e.g., subjectification and self-deprecation, more frequently.

CONCLUSION

Biographical interview is considered the strongest predictor of overall performance. The dimensions of impression management, reasoning quality and judgment proved to be reliable predictors of successful performance in the selection process. In addition, role image as a physician and professional commitment had a favorable effect on the selection decision.

Does doctors' personality differ from those of patients, the highly educated and other caring professions? An observational study using two nationally representative Australian surveys

OBJECTIVES

Personality differences between doctors and patients can affect treatment outcomes. We examine these trait disparities, as well as differences across medical specialities.

DESIGN

Retrospective, observational statistical analysis of secondary data.

SETTING

Data from two data sets that are nationally representative of doctors and the general population in Australia.

PARTICIPANTS

We include 23 358 individuals from a representative survey of the general Australian population (with subgroups of 18 705 patients, 1261 highly educated individuals and 5814 working in caring professions) as well as 19 351 doctors from a representative survey of doctors in Australia (with subgroups of 5844 general practitioners, 1776 person-oriented specialists and 3245 technique-oriented specialists).

MAIN OUTCOME MEASURES

Big Five personality traits and locus of control. Measures are standardised by gender, age and being born overseas and weighted to be representative of their population.

RESULTS

Doctors are significantly more agreeable (a: standardised score -0.12, 95% CIs -0.18 to -0.06), conscientious (c: -0.27 to -0.33 to -0.20), extroverted (e: 0.11, 0.04 to 0.17) and neurotic (n: 0.14, CI 0.08 to 0.20) than the general population (a: -0.38 to -0.42 to -0.34, c: -0.96 to -1.00 to -0.91, e: -0.22 to -0.26 to -0.19, n: -1.01 to -1.03 to -0.98) or patients (a: -0.77 to -0.85 to -0.69, c: -1.27 to -1.36 to -1.19, e: -0.24 to -0.31 to -0.18, n: -0.71 to -0.76 to -0.66). Patients (-0.03 to -0.10 to 0.05) are more open than doctors (-0.30 to -0.36 to -0.23). Doctors have a significantly more external locus of control (0.06, 0.00 to 0.13) than the general population (-0.10 to -0.13 to -0.06) but do not differ from patients (-0.04 to -0.11 to 0.03). There are minor differences in personality traits among doctors with different specialities.

CONCLUSIONS

Several personality traits differ between doctors, the population and patients. Awareness about differences can improve doctor-patient communication and allow patients to understand and comply with treatment recommendations.

The relationship between required physician letters of recommendation and decreasing diversity in osteopathic medical school admissions

CONTEXT

Some racial and ethnic groups are underrepresented in the medical field because they face unique barriers to admission to medical school. One admission requirement that can present a barrier for applicants is the physician letter of recommendation (PLOR). Undergraduate students report confusion with the application process and lack of mentorship to be two of their biggest challenges to becoming a doctor. It is especially challenging to those who already have limited access to practicing physicians. Therefore, we hypothesized that in the presence of a PLOR requirement, the diversity of students who apply and matriculate into medical school will be decreased.

OBJECTIVES

This study aims to determine if a relationship exists between a PLOR requirement for the medical school application and the proportion of underrepresented in medicine (URM) students applying and matriculating to that school.

METHODS

A retrospective study was conducted utilizing data published by the American Association of Colleges of Osteopathic Medicine Application Services (AACOMAS) on the race and ethnicity of applicants and matriculants to osteopathic medical schools during the years 2009-2019. In total, 35 osteopathic schools with 44 campuses were included in the study. Schools were grouped based on whether they required a PLOR. For each group of schools, descriptive statistics were performed for the following variables: number of total applicants, class size, application rate per ethnicity, matriculation rate per ethnicity, number of applicants per ethnicity, number of matriculants per ethnicity, and percentage of student body per ethnicity. The Wilcoxon rank-sum test was utilized to detect differences between the two groups. Statistical significance was assessed at the α=0.05 level.

RESULTS

Schools that required a PLOR showed decreases in the number of applicants across all races and ethnicities. Black students showed the greatest difference between groups and were the only ethnicity to show significant reductions across all outcomes in the presence of a PLOR requirement. On average, schools that required a PLOR have 37.3% (185 vs. 295; p<0.0001) fewer Black applicants and 51.2% (4 vs. 8.2; p<0.0001) fewer Black matriculants.

CONCLUSIONS

This study strongly suggests a relationship between requiring a PLOR's and decreasing racial and ethnic diversity in medical school matriculants, specifically the Black applicants. Based on this result, it is recommended that the requirement of a PLOR be discontinued for osteopathic medical schools.

Clinical experience can compensate for inferior academic achievements in an undergraduate objective structured clinical examination

BACKGROUND

Practical and non-cognitive skills are essential to medical professions; yet, success in medical studies is primarily assessed with cognitive criteria. We show that practical exams can benefit students who have only average high school final grades, but working experience in medical professions.

METHODS

With a cross-sectional study, we compare the performance of undergraduate medical students with working experience in adjacent health-care professions (and below-average school leaving-grades) with students who entered medical school directly based on their excellent school records in an Objective Structured Clinical Examination (OSCE). For a sample of more than 1,200 students, we use information on OSCE scores in medical and practical skills, doctor-patient communication/interaction, performance in MC-exams, and core sociodemographic variables.

RESULTS

Waiting list students outperformed their classmates in the demonstration of practical skills. Students admitted via their excellent school grades scored best overall. This difference vanishes once we control for school-leaving grade and age, the two main factors separating the analysed groups. Students from the waiting list have a significantly smaller overall chance to reach excellent grades in the first two years of study.

CONCLUSIONS

Students who gathered experiences in health-care professions before enrolling at medical school can benefit from an expanded role of practical elements in medical studies. Student selection instruments should take these different starting positions and qualities of applicants into account, for example with a quota for the professionally experienced.

Investigating Students’ Pre-University Admission Requirements and Their Correlation with Academic Performance for Medical Students: An Educational Data Mining Approach

Medical education is one of the most sought-after disciplines for its prestigious and noble status. Institutions endeavor to identify admissions criteria to register bright students who can handle the complexity of medical training and become competent clinicians. This study aims to apply statistical and educational data mining approaches to study the relationship between pre-admission criteria and student performance in medical programs at a public university in Saudi Arabia. The present study is a retrospective cohort study conducted at the College of Computer Science, King Khalid University, Abha, Kingdom of Saudi Arabia between February and November 2022. The current pre-admission criterion is the admission score taken as the weighted average of high school percentage (HSP), general aptitude test (GAT) and standard achievement admission test (SAAT), with respective weights of 0.3, 0.3 and 0.4. Regression and optimization techniques have been applied to identify weightages that better fit the data. Five classification techniques—Decision Tree, Neural Network, Random Forest, Naïve Bayes and K-Nearest Neighbors—are employed to develop models to predict student performance. The regression and optimization analyses show that optimized weights of HSP, GAT and SAAT are 0.3, 0.2 and 0.5, respectively. The results depict that the performance of the models improves with admission scores based on optimized weightages. Further, the Neural Network and Naïve Bayes techniques outperform other techniques. Firstly, this study proposes to revise the weights of HSP, GAT and SAAT to 0.3, 0.2 and 0.5, respectively. Secondly, as the evaluation metrics of models remain less than 0.75, this study proposes to identify additional student features for calculating admission scores to select ideal candidates for medical programs.

A mixed methods evaluation of Multiple Mini Interviews for entry into the Bachelor of Midwifery

BACKGROUND

Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.

AIM

To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.

METHODS

A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.

FINDINGS

Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant's MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.

CONCLUSION

MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.

Evaluating situational judgment test use and diversity in admissions at a southern US medical school

INTRODUCTION

Situational judgment tests have been adopted by medical schools to assess decision-making and ethical characteristics of applicants. These tests are hypothesized to positively affect diversity in admissions by serving as a noncognitive metric of evaluation. The purpose of this study was to evaluate the performance of the Computer-based Assessment for Sampling Personal Characteristics (CASPer) scores in relation to admissions interview evaluations.

METHODS

This was a cohort study of applicants interviewing at a public school of medicine in the southeastern United States in 2018 and 2019. Applicants took the CASPer test prior to their interview day. In-person interviews consisted of a traditional interview and multiple-mini-interview (MMI) stations. Between subjects, analyses were used to compare scores from traditional interviews, MMIs, and CASPer across race, ethnicity, and gender.

RESULTS

1,237 applicants were interviewed (2018: n = 608; 2019: n = 629). Fifty-seven percent identified as female. Self-identified race/ethnicity included 758 White, 118 Black or African-American, 296 Asian, 20 Native American or Alaskan Native, 1 Native Hawaiian or Other Pacific Islander, and 44 No response; 87 applicants identified as Hispanic. Black or African-American, Native American or Alaskan Native, and Hispanic applicants had significantly lower CASPer scores than other applicants. Statistically significant differences in CASPer percentiles were identified for gender and race; however, between subjects, comparisons were not significant.

CONCLUSIONS

The CASPer test showed disparate scores across racial and ethnic groups in this cohort study and may not contribute to minimizing bias in medical school admissions.