Selection into medicine: the predictive validity of an outcome-based procedure

A qualitative study of social accountability translation: from mission to living it

BACKGROUND

Medical schools are increasingly adopting socially accountable mission and curricula, the realisation of which are dependent on engaging individuals to embody the mission's principles in their everyday activities as doctors. However, little is known about how graduates perceive the efforts taken by their medical school to sensitise them to social accountability values, and how they translate this into their working lives. Our aim was to explore and understand graduate perceptions of how their medical school influenced them to embody a social accountability mission in their working lives.

METHODS

This was a qualitative interview study carried out with graduates/alumni [n = 51] of Christian Medical College, Vellore [CMCV], India, a school with a long-established and explicit social-accountability mission. Data coding and analysis were initially inductive and thematic using Braun and Clarke's six step framework. MacIntyre's virtue ethics theory framed secondary analysis, allowing us to consider the relationships between individual and contextual factors.

RESULTS

Our participants perceived that CMCV invested heavily in selecting personal qualities aligned with the CMCV mission. They saw that these qualities were reinforced through various practices: [e.g., placements in resource limited and/or remote and rural settings]; community engagement and expectations [e.g., student self-governance]; role modelling [staff and more senior students]. Much emphasis was placed on sustaining these traditions and practices over time, creating a strong sense of identity and belonging among participants, traditions which were fostered further by the alumni network and continued engagement with CMCV post-graduation.

CONCLUSIONS

Ensuring social accountable medical education depends on alignment and interactions over time between context and structures, systems and human agents. Further studies are needed to extend understanding of how students from diverse contexts experience socially accountable medical education and translate their educational experience into their thinking and practice after graduation.

What makes a 'good doctor'? A critical discourse analysis of perspectives from medical students with lived experience as patients

What constitutes a 'good doctor' varies widely across groups and contexts. While patients prioritise communication and empathy, physicians emphasise medical expertise, and medical students describe a combination of the two as professional ideals. We explored the conceptions of the 'good doctor' held by medical learners with chronic illnesses or disabilities who self-identify as patients to understand how their learning as both patients and future physicians aligns with existing medical school curricula. We conducted 10 semistructured interviews with medical students with self-reported chronic illness or disability and who self-identified as patients. We used critical discourse analysis to code for dimensions of the 'good doctor'. In turn, using concepts of Bakhtinian intersubjectivity and the hidden curriculum we explored how these discourses related to student experiences with formal and informal curricular content.According to participants, dimensions of the 'good doctor' included empathy, communication, attention to illness impact and boundary-setting to separate self from patients. Students reported that formal teaching on empathy and illness impact were present in the formal curriculum, however ultimately devalued through day-to-day interactions with faculty and peers. Importantly, teaching on boundary-setting was absent from the formal curriculum, however participants independently developed reflective practices to cultivate these skills. Moreover, we identified two operating discourses of the 'good doctor': an institutionalised discourse of the 'able doctor' and a counterdiscourse of the 'doctor with lived experience' which created a space for reframing experiences with illness and disability as a source of expertise rather than a source of stigma. Perspectives on the 'good doctor' carry important implications for how we define professional roles, and hold profound consequences for medical school admissions, curricular teaching and licensure. Medical students with lived experiences of illness and disability offer critical insights about curricular messages of the 'good doctor' based on their experiences as patients, providing important considerations for curriculum and faculty development.

Robust, defensible, and fair: The AMEE guide to selection into medical school: AMEE Guide No. 153

Selection is the first assessment of medical education and training. Medical schools must select from a pool of academically successful applicants and ensure that the way in which they choose future clinicians is robust, defensible, fair to all who apply and cost-effective. However, there is no comprehensive and evidence-informed guide to help those tasked with setting up or rejuvenating their local selection process. To address this gap, our guide draws on the latest research, international case studies and consideration of common dilemmas to provide practical guidance for designing, implementing and evaluating an effective medical school selection system. We draw on a model from the field of instructional design to frame the many different activities involved in doing so: the ADDIE model. ADDIE provides a systematic framework of Analysis (of the outcomes to be achieved by the selection process, and the barriers and facilitators to achieving these), Design (what tools and content are needed so the goals of selection are achieved), Development (what materials and resources are needed and available), Implementation (plan [including piloting], do study and adjust) and Evaluation (quality assurance is embedded throughout but the last step involves extensive evaluation of the entire process and its outcomes).HIGHLIGHTSRobust, defensible and fair selection into medical school is essential. This guide systematically covers the processes required to achieve this, from needs analysis through design, development and implementation, to evaluation of the success of a selection process.

Does medical education reform change who is selected? A national cross-sectional survey from China

OBJECTIVES

Approximately 10 years ago, China introduced an education plan to improve the overall quality of medical education and to better serve the population's health needs. Many medical schools were then recognised and financed by China's Ministry of Education to develop and operationalise new pilot programmes (PPs) aligned with this plan. These ran in parallel with the traditional programmes (TPs). One way to achieve the plan's first aim, improving the quality of medical education, is to select academically stronger candidates. We, thus, examined and compared who were selected into PPs and TPs.

DESIGN

Cross-sectional study.

SETTING

Data were collected from 123 medical schools across China via the 2021 China Medical Student Survey.

PARTICIPANTS

Participants were undergraduate clinical medicine students across all year groups.

PRIMARY AND SECONDARY OUTCOME MEASURES

Medical school selection was via the National College Entrance Examination (NCEE). Medical students' NCEE performance and their sociodemographics were used as the primary and secondary outcome measures. Mann-Whitney or χ2 tests were used to compare the means between educational programmes (PPs vs TPs) and various selection outcomes. Multilevel mixed-effects regressions were employed to account for school idiosyncratic selection results.

RESULTS

Of the 204 817 respondents, 194 163 (94.8%) were in a TP and 10 654 (5.2%) a PP. PP respondents (median=75.2, IQR=69.5-78.8) had significantly higher NCEE scores than their TP counterparts (median=73.9, IQR=68.5-78.7). Holding constant their NCEE score, PP respondents were significantly more likely to come from urban areas, not be first-generation college students, and have parents with higher occupational status and income.

CONCLUSIONS

Assuming quality can be indicated by prior academic achievement at the point of selection, PPs achieved this mission. However, doing so limited medical students' diversity. This may be unhelpful in achieving the Education Plan's goal to better serve China's health needs.

Relationship between selection criteria and trainee performance in medical specialty training: A retrospective longitudinal study

BACKGROUND

Investigation of tools used for candidate selection as predictors of future performance in medical education has been primarily undertaken within the undergraduate setting, but little is known about the selection to medical specialist programs. This retrospective longitudinal study aims to explore correlations between selection tools and the performance of trainees enrolled in the Australasian College of Dermatologists (ACD), the accredited specialist medical college for training in dermatology in Australia.

METHODS

Data were collected from consecutive cohorts of ACD trainees commencing from 2007 to 2015. Predictive variables were trainee demographics and selection tools (prior academic qualification; research experience; clinical experience; curriculum vitae [CV] rating; interview scores). The outcome variables were exam performance. First Year ([Pharmacology and Clinical Sciences] and Fellowship Exam [Written, Clinical, and Overall Score]). Statistical analyses included: descriptive statistics summaries; bivariate correlation of selection criteria and demographic data with exam performance; and linear regression analysis to identify predictors of exam performance.

RESULTS

Demographic analysis (N = 172) showed that 64% of trainees were female, the average age was 30.5 years (± 3.47) and trainees living in high socioeconomic status (SES) areas were over-represented. Using Pearson correlation analysis, interview scores were significantly positively correlated with First Year Pharmacology results (p = 0.018), Fellowship Written results (p = 0.002), and Fellowship Overall Scores (p = 0.006). First Year Pharmacology Exam performance was most highly correlated with Fellowship Exam performance (p = 0.000). No association was identified between exam performance and gender, SES, prior academic qualification, research experience or CV rating. Linear regression analysis showed that interview score (p = 0.012), entry age (p = 0.026) and First Year Pharmacology score (p = 0.002) were predictors of Fellowship exam performance.

CONCLUSION

These results suggest that the comprehensive selection tools used by ACD are effective, with interviews being a highly valid tool linked to exam performance. These findings have important practical implications for assessing the selection process in specialist dermatology training.

The predictive validity of admission criteria for the results of clinical competency assessment with an emphasis on family medicine in the fifth year of medical education: an observational study

BACKGROUND

In many countries, the number of applicants to medical schools exceeds the number of available places. This offers the need, as well as the opportunity to medical schools to select those applicants most suitable for later work as a doctor. However, there is no generally accepted definition of a 'good doctor'. Clinical competencies may serve as surrogates. The aim of this study was to compare medical students in Germany selected based either on their pre-university grade point average alone or based on the result of a university-specific selection procedure regarding their clinical competencies with an emphasis on family medicine in the later years of training.

METHODS

We used the 'Allgemeinarztbarometer Ausbildung' (Undergraduate Family Medicine Barometer), an instrument developed to assess clinical competencies with an emphasis on family medicine, to compare students in the pre-university grade point average admission-quota and the university-specific selection procedure admission-quota in the fifth year of training. Students were judged by their supervising general practitioners after a two-week practical course. Competencies were rated on a five-point Likert-scale (1 = 'totally agree' i.e. the student is very competent to 5 = 'totally disagree' i.e. the student is not competent at all).

RESULTS

We included 94 students (66% female). Students in the university-specific selection procedure quota (n = 80) showed better mean scores in every item of the Undergraduate Family Medicine Barometer. We found a statistically significant difference between the two groups for the item assessing communication skills (M [university-specific selection procedure quota] = 1.81, SD = 0.84 vs. M [pu-GPA quota] = 2.38, SD = 0.96; t[91] = -2.23, p = .03; medium effect size). Logistic regression revealed no statistically significant age or gender contribution.

CONCLUSIONS

Despite the small sample-size, our results indicate, that students selected via an university-specific selection procedure show better communicative competencies in the later years of training.

Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years

PURPOSE

Medical school selection committees aim to identify the best possible students and, ultimately, the best future doctors from a large, well-qualified, generally homogeneous pool of applicants. Constructive alignment of medical school selection, curricula, and assessment with the ultimate outcomes (e.g., CanMEDS roles) has been proposed as means to attain this goal. Whether this approach is effective has not yet been established. The authors addressed this gap by assessing the relationship between performance in an outcomes-based selection procedure and performance during the clinical years of medical school.

METHOD

Two groups of students were compared: (1) those admitted into Maastricht University Medical School via an outcomes-based selection procedure and (2) those rejected through this procedure who were admitted into the program through a national, grade-point-average-based lottery. The authors compared performance scores of students from the 2 groups on all 7 CanMEDS roles, using assessment data gathered during clinical rotations. The authors examined data from 3 cohorts (2011-2013).

RESULTS

Students admitted through the local, outcomes-based selection procedure significantly outperformed the initially rejected but lottery-admitted students in all years, and the differences between groups increased over time. The selected students performed significantly better in the CanMEDS roles of Communicator, Collaborator, and Professional in the first year of clinical rotations; in these 3 roles-plus Organizer-in the second year; and in 2 additional roles (Advocate and Scholar-all except Medical Expert) at the end of their clinical training.

CONCLUSIONS

A constructively aligned selection procedure has increasing predictive value across the clinical years of medical school compared with a GPA-based lottery procedure. The data reported here suggest that constructive alignment of selection, curricula, and assessment to ultimate outcomes is effective in creating a selection procedure predictive of clinical performance.

Opening the black box of selection

Medical school selection is currently in the paradoxical situation in which selection tools may predict study outcomes, but which constructs are actually doing the predicting is unknown (the 'black box of selection'). Therefore, our research focused on those constructs, answering the question: do the internal structures of the tests in an outcome-based selection procedure reflect the content that was intended to be measured? Downing's validity framework was applied to organize evidence for construct validity, focusing on evidence related to content and internal structure. The applied selection procedure was a multi-tool, CanMEDS-based procedure comprised of a video-based situational judgement test (focused on (inter)personal competencies), and a written aptitude test (reflecting a broader array of CanMEDS competencies). First, we examined content-related evidence pertaining to the creation and application of the competency-based selection blueprint and found that the set-up of the selection procedure was a robust, transparent and replicable process. Second, the internal structure of the selection tests was investigated by connecting applicants' performance on the selection tests to the predetermined blueprint using cognitive diagnostic modeling. The data indicate 89% overlap between the expected and measured constructs. Our results support the notion that the focus placed on creating the right content and following a competency-blueprint was effective in terms of internal structure: most items measured what they were intended to measure. This way of linking a predetermined blueprint to the applicants' results sheds light into the 'black box of selection' and can be used to support the construct validity of selection procedures.

Do Medical Students in Their Fifth Year of Undergraduate Training Differ in Their Suitability to Become a "Good Doctor" Depending on Their Admission Criteria? A Pilot Study

BACKGROUND

In Germany, the pre-university grade point average (pu-GPA) has to be the main criterion for medical school applicant selection. This is also mandatory in the university-specific selection procedures (Auswahlverfahren der Hochschulen [AdH]). The admission framework has now been reworked following a judgement by the German Federal Constitutional Court. From 2020, more students will be admitted based solely on the pu-GPA and at least two selection criteria independent of the pu-GPA have to be considered in the AdH. However, the question whether an AdH (the core of the AdH at Lübeck Medical School [LMS], Germany, is a 30-mins panel interview led by two faculty members and one student) leads to better doctors as compared to pu-GPA-based selection, remains unanswered.

OBJECTIVE

To compare students selected based either on their pu-GPA alone ("pu-GPA-students") or based on the result of the AdH at LMS ("AdH-students") regarding their suitability to become a good doctor.

DESIGN

We conducted a cross-sectional observational pilot study at LMS. Students were judged regarding their overall suitability to become a good doctor by their supervising general practitioners after a two-week internship in their last year of theoretical medical education. The scores were matched to the selection procedure and compared between the pu-GPA-students and AdH-students.

RESULTS

In all, 79% of the AdH-students were rated as "absolutely suitable" for the medical profession, as compared to 42% of the pu-GPA-students (p = 0.01, odds ratio 5.17, 95% confidence interval = 1.41, 18.99). We did not find any association between gender or age and the suitability rating.

CONCLUSION

Despite the small sample size, our results indicate that it could be favourable to select medical students not only based on their pu-GPA but also using additional selection criteria.

Bachelor completion and dropout rates of selected, rejected and lottery-admitted medical students in the Netherlands

BACKGROUND

Evidence for the effectiveness of the selection of medical students is weak. This study aimed to examine the added value of a two-step selection procedure (first step non-academic, second step academic tests) to a pre-university GPA-based lottery procedure. Because previous research has suggested that participation in selection (regardless of the outcome) is a predictor of study success, this study is the first to include students who initially applied for selection, then refrained from (actively) participating in selection and were eventually admitted through lottery.

METHODS

Bachelor completion and dropout rates of selected (n = 416) and lottery-admitted students from four cohorts (2006-2009) were compared using logistic regression analysis. Four groups of lottery-admitted students were distinguished: students who were rejected after step 2 (n = 57), were rejected after step 1 (n = 169), withdrew during selection step 1 (n = 42) and students who only applied for lottery (n = 366). Covariates included gender, age, pre-university GPA and cohort.

RESULTS

There was a significant association between admission group and obtaining a bachelor degree in three years. Selected students were more likely to obtain a bachelor degree within three years (64.2% versus 51.6%; OR = 1.7) or four years (81.5% versus 74.3%; OR = 1.6) than students who only applied to a lottery (p <  0.05); selected students also seemed more likely to obtain all Year-1 course credits than students who withdrew during step 1 (40.4% versus 21.4%; OR = 2.3; p <  0.05). We found no significant association between dropout and admission groups. Students rejected at step 1 or 2 did not perform significantly different from selected students on any of the outcome measures.

CONCLUSIONS

The findings indicated that students at risk for study delay in the preclinical phase in our context were more likely to refrain from applying to a demanding selection procedure when a less demanding alternative was available. We found no significant associations between the non-academic and academic selection steps and bachelor completion and dropout rates. These findings suggest that the presence of the selection was more important than these specific selection components. In follow-up research, we plan to investigate the associations between the admission groups and outcome measures in the clinical phase.

Selection and lottery in medical school admissions: who gains and who loses?

This article was migrated. The article was marked as recommended. Concerns related to fairness of medical school admissions through selection have led some scholars to consider selection as an expensive lottery and suggest that lottery may be fairer. This paper considers the issue of selection versus lottery from the perspectives of three groups of stakeholders: 1) applicants, 2) medical schools, and 3) society. This paper contributes to the discussion by addressing advantages and disadvantages of the use of selection and lottery for these stakeholder groups, grounded in the findings from research. Themes that are discussed are reliability and validity issues, perceived influence on selection outcomes and student uptake, effects on student diversity, financial costs, impact on rejected applicants, transparency, and strategic behaviour. For each stakeholder group both lottery and selection yield a combination of advantages and disadvantages, which implies that none of the currently available admissions strategies completely fulfils stakeholders' needs. Research indicates that selection yields only small gains compared to a lottery procedure, while the student diversity, necessary for serving the increasingly diverse patient population, may be compromised. We argue that society's needs should drive admissions policies rather than institutional gains, which means that until a selection procedure is developed that does not disadvantage certain types of students, a lottery procedure should be preferred.

Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems

CONTEXT

Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection.

OBJECTIVES

We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school.

METHODS

The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations.

RESULTS

The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000.

CONCLUSIONS

This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.