A systematic review of stakeholder views of selection methods for medical schools admission

Artificial Intelligence Screening of Medical School Applications: Development and Validation of a Machine-Learning Algorithm

PURPOSE

To explore whether a machine-learning algorithm could accurately perform the initial screening of medical school applications.

METHOD

Using application data and faculty screening outcomes from the 2013 to 2017 application cycles (n = 14,555 applications), the authors created a virtual faculty screener algorithm. A retrospective validation using 2,910 applications from the 2013 to 2017 cycles and a prospective validation using 2,715 applications during the 2018 application cycle were performed. To test the validated algorithm, a randomized trial was performed in the 2019 cycle, with 1,827 eligible applications being reviewed by faculty and 1,873 by algorithm.

RESULTS

The retrospective validation yielded area under the receiver operating characteristic (AUROC) values of 0.83, 0.64, and 0.83 and area under the precision-recall curve (AUPRC) values of 0.61, 0.54, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The prospective validation yielded AUROC values of 0.83, 0.62, and 0.82 and AUPRC values of 0.66, 0.47, and 0.65 for the invite for interview, hold for review, and reject groups, respectively. The randomized trial found no significant differences in overall interview recommendation rates according to faculty or algorithm and among female or underrepresented in medicine applicants. In underrepresented in medicine applicants, there were no significant differences in the rates at which the admissions committee offered an interview (70 of 71 in the faculty reviewer arm and 61 of 65 in the algorithm arm; P = .14). No difference in the rate of the committee agreeing with the recommended interview was found among female applicants (224 of 229 in the faculty reviewer arm and 220 of 227 in the algorithm arm; P = .55).

CONCLUSIONS

The virtual faculty screener algorithm successfully replicated faculty screening of medical school applications and may aid in the consistent and reliable review of medical school applicants.

A mixed methods evaluation of Multiple Mini Interviews for entry into the Bachelor of Midwifery

BACKGROUND

Multiple-mini interviews (MMI) are increasingly used as part of the admission process into healthcare degrees. Evaluations have found MMIs to be a fair assessment tool in terms of reliability and validity and viewed positively by those involved in the MMI process. The use of MMIs in midwifery is novel and evaluation is lacking.

AIM

To evaluate the use of MMIs as part of the admission process for the Bachelor of Midwifery in one Australian university.

METHODS

A basic convergent mixed methods study design was utilised. Data included linked data sets, Likert scale responses to survey questions, focus groups and open-ended survey questions. Integration took place at the interpretation and reporting stage.

FINDINGS

Participants viewed the MMI experience positively. The study confirmed the reliability of the MMIs as an assessment tool. Most variance in MMI scores was attributed to the candidate at 31.4 % with the interviewer and the interview station having less influence on the MMI score at 11 % and 6.4 % variance. Older applicants on average achieved higher MMI scores, and those who spoke a language other than English at home or were first in family to attend university had lower on average MMI scores. Being born overseas did not impact an applicant's MMI score. The overall experience was seen as fair, offering further opportunity to gain entry into the Bachelor of Midwifery.

CONCLUSION

MMIs were viewed positively and findings support the use of MMIs as part of an admissions process for the Bachelor of Midwifery.

Relationship between selection criteria and trainee performance in medical specialty training: A retrospective longitudinal study

BACKGROUND

Investigation of tools used for candidate selection as predictors of future performance in medical education has been primarily undertaken within the undergraduate setting, but little is known about the selection to medical specialist programs. This retrospective longitudinal study aims to explore correlations between selection tools and the performance of trainees enrolled in the Australasian College of Dermatologists (ACD), the accredited specialist medical college for training in dermatology in Australia.

METHODS

Data were collected from consecutive cohorts of ACD trainees commencing from 2007 to 2015. Predictive variables were trainee demographics and selection tools (prior academic qualification; research experience; clinical experience; curriculum vitae [CV] rating; interview scores). The outcome variables were exam performance. First Year ([Pharmacology and Clinical Sciences] and Fellowship Exam [Written, Clinical, and Overall Score]). Statistical analyses included: descriptive statistics summaries; bivariate correlation of selection criteria and demographic data with exam performance; and linear regression analysis to identify predictors of exam performance.

RESULTS

Demographic analysis (N = 172) showed that 64% of trainees were female, the average age was 30.5 years (± 3.47) and trainees living in high socioeconomic status (SES) areas were over-represented. Using Pearson correlation analysis, interview scores were significantly positively correlated with First Year Pharmacology results (p = 0.018), Fellowship Written results (p = 0.002), and Fellowship Overall Scores (p = 0.006). First Year Pharmacology Exam performance was most highly correlated with Fellowship Exam performance (p = 0.000). No association was identified between exam performance and gender, SES, prior academic qualification, research experience or CV rating. Linear regression analysis showed that interview score (p = 0.012), entry age (p = 0.026) and First Year Pharmacology score (p = 0.002) were predictors of Fellowship exam performance.

CONCLUSION

These results suggest that the comprehensive selection tools used by ACD are effective, with interviews being a highly valid tool linked to exam performance. These findings have important practical implications for assessing the selection process in specialist dermatology training.

Millennials Medical Students Generation at the Crosswalks: Motivations and Attitudes Towards Study and Future Career - A Mixed-Method Study

BACKGROUND AND OBJECTIVES

The aim of this study was to investigate in the millennium medical student generation the influence of the curriculum (problem-based curriculum [PBC] vs science-based curriculum [SBC]), gender and semester level on medical students' motives to study medicine, their attitudes toward their career and in this regard their view about their study condition in university.

METHODS

Semi-structured qualitative interviews with 28 medical students were performed and analyzed using Mayring's content analysis. Based on these results, a quantitative questionnaire for a nationwide survey was developed using a mixed-method-approach and send to most medical faculties in Germany. Data from n=1053 students entered statistical analysis.

RESULTS

Humanistic ideals prevailed in the choice to enter medical school and to become a physician. PBC students were found to be significantly (p<0.001) more idealistic and patient oriented, and they regard their curriculum more competitive than SBC-students (p<0.001). A balanced work and family life is essential for all students but particularly important for the PBC - group, male and undergraduate students. The majority of students wanted to work with patients and omitted patient-distant line of work. Undergraduate SBC-students saw their studies as old-fashioned citing lack of patient contact (p<0.001 compared to PBC), which eased in the graduate study part.

CONCLUSION

This study found major differences in student's perceptions depending on curriculum type. PBC-students were more idealistic, and humanistic ideals prevailed in comparison to SBC. For both, close patient contact is essential in their training. Particularly for female students, lifestyle factors and a balanced work-life-integration outweigh career ambitiousness. This study offers an important insight to policy makers and educators to understand the motivation and perceptions of the millennial student generation regarding their studies and future career plans, which should be considered in educational policies.

BMAT's predictive validity for medical school performance: A retrospective cohort study

BACKGROUND

Although used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.

METHODS

This was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.

RESULTS

When BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.

DISCUSSION

Schools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

Medical Student Selection

Medical school admissions committees are tasked with fulfilling the values of their institutions through careful recruitment. Making accurate predictions regarding the enrollment behavior of admitted students is critical to intentionally formulating class composition and impacts long-term physician representation.

Relationship between admission criteria and academic performance in basic science courses in health science colleges in KAU

BACKGROUND

At King Abdulaziz University, medical and health science schools depend on admission exams (aptitude and achievement) and preparatory year scores in their students' selection. However, with the growing number of applicants and the drastic changes in teaching and assessment in these colleges, continuous assessment and development of admission criteria are needed. In this study, we aimed to evaluate the correlation of admission exam scores, in addition to the preparatory year Grade Point Average (GPA), with academic performance in the basic science subjects such as Clinical Biochemistry and Clinical Pharmacology in health science colleges.

METHODS

The study was conducted on four cohort studies, two faculty of nursing cohorts; nursing students (2017-2018, n=146) nursing students (2018-2019, n=81), and two faculty of applied medical sciences cohorts, clinical nutrition students (2017-2018, n=33), and clinical nutrition students (2018-2019, n=28). The students' scores of General Aptitude Test (GAT), Scholastic Achievement Admission Test (SAAT), and preparatory year GPA were all recorded at the beginning of each semester before the beginning of courses. Clinical Biochemistry and Clinical Pharmacology exam results were recorded at the end of the semester. Correlation was done for each cohort and all cohorts pooled.

RESULTS

Results showed only a weak correlation detected between SAAT and the overall achievement in Clinical Biochemistry (r= 0.192, P= 0.042) in nursing students (2017-2018), but no correlation was seen with SAAT or preparatory year scores. There was also no significant correlation between admission exams scores and the students' academic achievement in Clinical Biochemistry or Clinical Pharmacology. On the other hand Clinical Pharmacology exam results showed a significant positive correlation with Clinical Biochemistry results (r=0.688, P=0.000).

CONCLUSION

Our results could indicate the need to revisit the admission criteria for these colleges. Furthermore, specific preparatory year tracks for health science colleges can ensure that students improve the specific skills and knowledge required for their future college years3.

The economic costs of selecting medical students: An Australian case study

CONTEXT

The design of selection methods must balance, amongst a range of factors, the desire to select the best possible future doctors with the reality of our resource-constrained environment. Examining the cost of selection processes enables us to identify areas in which efficiencies may be gained.

METHODS

A cost description study was conducted based on selection for 2018 entry into medical school directly from secondary school. The perspectives of applicants, volunteer interviewers and the admitting institution were considered. Costs were modelled based on the Monash University (Australia) selection process, which uses a combination of secondary school matriculation score, aptitude test score (Undergraduate Medicine and Health Sciences Admission Test) and multiple mini-interview score. A variety of data sources were utilised, including bespoke surveys, audit data and existing literature. All costs are expressed in 2018 Australian dollars (AU$). Applicant behaviours in preparing for selection tests were also evaluated.

RESULTS

A total of 381 of 383 applicants returned the survey. Over 70% of applicants had utilised commercial preparation materials. The median total cost to applicants was AU$2586 (interquartile range [IQR] AU$1574-3999), including costs to both prepare for and attend selection tests. Of 217 volunteer interviewers, 108 returned the survey. These were primarily health professional clinicians at a mid-career stage. The median total cost to interviewers was AU$452 (IQR AU$252-715) for participation in a half-day interview session, largely due to the loss of income. The cost to the admitting institution was AU$269 per applicant, accounted for by the costs of equipment and consumables (52%), personnel (34%) and facilities (14%).

CONCLUSIONS

The costs of student selection for medical school are substantial. Understanding costs facilitates achievement of the objective of selecting the desired future medical workforce within the constraints of the resources available. Opportunities for change may arise from changes in applicant preparation behaviours, opportunities for economies of scale, and efficiencies driven by technological solutions.

National recruitment scheme for pre-registration pharmacist training in England and Wales: a mixed method evaluation of experiences of applicant pharmacy students

BACKGROUND

A national pre-registration pharmacist training recruitment scheme, which replaces local recruitment models, was introduced in England and Wales in 2017. The national recruitment system allows pharmacy students to apply for the 52 weeks training programmes (mandatory requirement for registration as a pharmacist), through a single application system prior to undertaking a nationally administered assessment. This study aimed to explore experiences of pharmacy students on the national recruitment scheme, particularly their views on the selection methodology, application process, and offer outcomes.

METHODS

This mixed method study involved a) an online survey of all (approximate n = 2800) year 4 (final year of MPharm degree) pharmacy students in England and Wales and b) a qualitative focus group with four students. The study population was eligible to participate in the 2017/18 national recruitment scheme. Survey respondents were invited to participate in a focus group. Quantitative data were analysed using descriptive and inferential analysis. Qualitative data were analysed using the framework technique. Participation was voluntary. Ethical approval from University of Birmingham was obtained.

RESULTS

A total of 307 completed surveys were returned (approximate response rate 11%). Respondents were generally satisfied with the application process and commended the fairness of the selection methodology and convenience in allowing them to apply to multiple training providers. Most survey respondents (n = 181, 72.9%) were either satisfied or highly satisfied with the training programme they were offered based on their assessment performances. Three themes and eight sub-themes obtained from the analysis of over 200 open comments data from the survey and transcript of a focus group with four participants. Results suggested the need to widen the timeframe available for applicants to shortlist their preferred employers, improve the method of programme listing in the application system, and consideration of prior achievements including academic performances and placement experiences to be included in the selection methodology.

CONCLUSIONS

Experiences of pharmacy students on the national recruitment scheme suggest that respondents considered the selection methodology to be fair. Student engagement and satisfaction with the recruitment system can be maximised through improved listing of employers and widening the timescales for students to shortlist their preferred employers during application process. Inclusion of University achievements in the selection methodology will require consideration of evidence based approaches. Low response rate limits generalisation of findings.

One Small Step for Step 1

Step 1 of the United States Medical Licensing Examination (USMLE) is a multiple-choice exam primarily measuring knowledge about foundational sciences and organ systems. The test was psychometrically designed as pass/fail for licensing boards to decide whether physician candidates meet minimum standards they deem necessary to obtain the medical licensure necessary to practice. With an increasing number of applicants to review, Step 1 scores are commonly used by residency program directors to screen applicants, even though the exam was not intended for this purpose. Elsewhere in this issue, Chen and colleagues describe the "Step 1 climate" that has evolved in undergraduate medical education, affecting learning, diversity, and well-being.Addressing issues related to Step 1 is a challenge. Various stakeholders frequently spend more time demonizing one another rather than listening, addressing what lies under their respective control, and working collaboratively toward better long-term solutions. In this Invited Commentary, the author suggests how different constituencies can act now to improve this situation while aspirational future solutions are developed.One suggestion is to report Step 1 and Step 2 Clinical Knowledge scores as pass/fail and Step 2 Clinical Skills scores numerically. Any changes must be carefully implemented in a way that is mindful of the kind of unintended consequences that have befallen Step 1. The upcoming invitational conference on USMLE scoring (InCUS) will bring together representatives from all stakeholders. Until there is large-scale reform, all stakeholders should commit to taking (at least) one small step toward fixing Step 1 today.

How can the perceptions and experiences of medical educator stakeholders inform selection into medicine? An interpretative phenomenological pilot study

This article was migrated. The article was marked as recommended. Background Attempts to utilise the experiences of stakeholders to better inform selection into medicine are rare in the literature. Published scholarship to date reflects a myriad of competing goals for selecting and graduating 'good doctors' amidst increasingly complex health care environments. This includes debates around what is the 'good doctor', selection methods, health care decision-making, the doctor-patient relationship, patient-centredness, professionalism and stakeholder experiences with doctors. Within the complexity manifested by these multiple dimensions, decisions about the characteristics and capabilities on which selection should be based may have privileged some stakeholder groups over others, with patient experiences particularly de-emphasised. The aims of this pilot study were to focus on front-line medical educators as stakeholders whose experiences might be valuable for informing selection into medicine and to inform a larger-scale study of the topic from the perspectives of a more diverse group of stakeholders, including patients. Method Fourteen (14) medical educator participants were recruited for a semi-structured group interview at an international conference for health professional educators. The audio-recording was transcribed verbatim and the raw data were de-identified and organised with the aid of computer assisted data analysis software. Coding was initiated and Smith's interpretative phenomenological analytical (IPA) method employed ( Smith, Flowers, & Larkin, 2009). Results Initial analysis yielded four broad phenomenological themes: perceptions of 'good doctors', selection processes, selection-related challenges and possible solutions. The more deeply experiential data were captured in an analytical commentary of first-person accounts that may be useful for informing future selection strategies. Participant experiences mirrored the major debates in medical selection but their accounts revealed a negativity and cynicism about the topic that was concerning and warrants further investigation. Conclusion This study contributes to medical student selection research through offering an account of the 'lived experiences' of front-line medical educator stakeholders.

Correction to: A systematic review of stakeholder views of selection methods for medical schools admission

Following publication of the original article [1], the author report typographical errors in the.