Academic Performance on First-Year Medical School Exams: How Well Does It Predict Later Performance on Knowledge-Based and Clinical Assessments?

'Early identification of struggling pre-clerkship learners using formative clinical skills OSCEs: an assessment for learning program.'

Multiple experts in clinical skills remediation recommend early identification to support struggling learners, but there is minimal documentation on implementation of these programs. We share one school's outcomes-based research utilizing the formative assessment for learning model to early-identify pre-clerkship students struggling with clinical skills using formative OSCEs (F-OSCE). Student scores were monitored over longitudinal F-OSCE experiences as part of a curricular innovation. Points towards early identification accumulated when a student's score fell below the 80% threshold for each section of an OSCE. Students who accumulated enough points were advised of the need for intervention, and coaching was recommended. Students were surveyed about their experiences with the program. The objective was to explore whether this early identification program and coaching intervention had a positive impact on subsequent OSCE performance. Of 184 students in 2 cohorts who completed F-OSCEs, 38 (20.7%) were flagged for early identification. Of these, 17 (44.7%) sought additional help by voluntarily participating in the coaching program. Students who participated in extra clinical skills coaching demonstrated statistically significant improvements in performance on subsequent FOSCEs, as did the early identified students who did not participate in extra coaching. The greatest impact of coaching intervention was noted in the physical examination domain. This program was effective in identifying students struggling with clinical skills on formative OSCEs. Early identified students demonstrated improvements in subsequent OSCE performance, with those who sought coaching faring slightly better. Development of robust early identification programs as formative assessments of clinical skills and follow-up coaching programs to guide skills development are important implications of this work. Monitoring short- and long-term results for students identified through this approach to see if improvement is sustained is planned.

Medical school grades may predict future clinical competence

BACKGROUND

In real-world medical education, there is a lack of reliable predictors of future clinical competencies. Hence, we aim to identify the factors associated with clinical competencies and construct a prediction model to identify "improvement required" trainees.

METHODS

We analyzed data from medical students who graduated from National Yang-Ming University with clerkship training and participated in the postgraduate year (PGY) interview at Taipei Veterans General Hospital. Clinical competencies were evaluated using grades of national objective structured clinical examination (OSCEs). This study used data from medical students who graduated in July 2018 as the derivation cohort (N = 50) and those who graduated in July 2020 (n = 56) for validation.

RESULTS

Medical school grades were associated with the performance of national OSCEs (Pearson r = 0.34, p = 0.017), but the grades of the structured PGY interviews were marginally associated with the national OSCE (Pearson r = 0.268, p = 0.06). A prediction model was constructed to identify "improvement required" trainees, defined: trainees with the lowest 25% of scores in the national OSCEs. According to this model, trainees with the lowest 25% medical school grades predicted a higher risk of the "improvement required" clinical performance (Q1-Q3 vs Q4 = 15% vs 60%, odds ratio = 8.5 [95% confidence interval = 1.8-39.4], p = 0.029). In the validation cohort, our prediction model could accurately classify 76.7% "improvement required" and "nonimprovement required" students.

CONCLUSION

Our study suggests that interventions for students with unsatisfactory medical school grades are warranted to improve their clinical competencies.

Predicting students' performance in English and Mathematics using data mining techniques

This study attempts to predict secondary school students' performance in English and Mathematics subjects using data mining (DM) techniques. It aims to provide insights into predictors of students' performance in English and Mathematics, characteristics of students with different levels of performance, the most effective DM technique for students' performance prediction, and the relationship between these two subjects. The study employed the archival data of students who were 16 years old in 2019 and sat for the Malaysian Certificate of Examination (MCE) in 2021. The learning of English and Mathematics is a concern in many countries. Three main factors, namely students' past academic performance, demographics, and psychological attributes were scrutinized to identify their impact on the prediction. This study utilized the Orange software for the DM process. It employed Decision Tree (DT) rules to determine the characteristics of students with low, moderate, and high performance in English and Mathematics subjects. DT and Naïve Bayes (NB) techniques show the best predictive performance for English and Mathematics subjects, respectively. Such characteristics and predictions may cue appropriate interventions to improve students' performance in these subjects. This study revealed students' past academic performance as the most critical predictor, as well as a few demographics and psychological attributes. By examining top predictors derived using four different classifier types, this study found that students' past Mathematics performance predicts their MCE English performance and students' past English performance predicts their MCE Mathematics performance. This finding shows students' performances in both subjects are interrelated.

Can we predict failure in licensure exams from medical students' undergraduate academic performance?

BACKGROUND

In 2015, the Medical Council of Canada increased the minimum pass level for the Medical Council of Canada Qualifying Examination Part I, and students had a higher rate of failure than in previous years. The purpose of this study was to predict students at an increased odds of examination failure to allow for early, targeted interventions.

METHODS

We divided our dataset into a derivation cohort and two validation cohorts and used multiple logistic regression to predict licensing examination failure. We then performed receiver operating characteristics and a sensitivity analysis using different cutoffs for explanatory variables to identify the cutoff threshold with the best predictive value at identifying students at increased odds of failure.

RESULTS

After multivariate analysis, only pre-clerkship GPA was a significant independent predictor of failure (OR 0.76, 95% CI [0.66, 0.88], p < 0.001). The probability of failure increased steeply when the pre-clerkship GPA fell below 80% and 76% was found to be the most efficient cutoff for predicting failure (OR 9.37, 95% CI [3.08, 38.41]).

CONCLUSIONS

Pre-clerkship performance can predict students at increased odds of licensing examination failure. Further studies are needed to explore whether early interventions for at-risk students alter their examination performance.

Three Decades Later: A Scoping Review of the Literature Related to the United States Medical Licensing Examination

PURPOSE

To conduct a scoping review of the timing, scope, and purpose of literature related to the United States Medical Licensing Examination (USMLE) given the recent impetus to revise USMLE scoring.

METHOD

The authors searched PubMed, PsycInfo, and ERIC for relevant articles published from 1990 to 2019. Articles selected for review were labeled as research or commentaries and coded by USMLE Step level, sample characteristics (e.g., year(s), single/multiple institutions), how scores were used (e.g., predictor/outcome/descriptor), and purpose (e.g., clarification/justification/description).

RESULTS

Of the 741 articles meeting inclusion criteria, 636 were research and 105 were commentaries. Publication totals in the past 5 years exceeded those of the first 20 years.Step 1 was the sole focus of 38%, and included in 84%, of all publications. Approximately half of all research articles used scores as a predictor or outcome measure related to other curricular/assessment efforts, with a marked increase in the use of scores as predictors in the past 10 years. The overwhelming majority of studies were classified as descriptive in purpose.

CONCLUSIONS

Nearly 30 years after the inception of the USMLE, aspirations for its predictive utility are rising faster than evidence supporting the manner in which the scores are used. A closer look is warranted to systematically review and analyze the contexts and purposes for which USMLE scores can productively be used. Future research should explore cognitive and noncognitive factors that can be used in conjunction with constrained use of USMLE results to inform evaluation of medical students and schools and to support the residency selection process.

Portable Video Media Versus Standard Verbal Communication in Surgical Teaching: A Prospective, Multicenter, and Randomized Controlled Crossover Trial

OBJECTIVE

This study aims to evaluate the effectiveness of portable video media (PVM) compared to standard verbal communication (SVC) as a novel adjunct for surgical education of junior medical officers and medical students, in terms of knowledge acquisition and participant satisfaction.

BACKGROUND

The effective, continued education of final year medical students and junior doctors is the foundation of quality healthcare. The development of new media technologies and rapid internet streaming has resulted in an opportunity for the integration of PVM into medical education. PVM is an educational platform with the advantage of being standardized, efficient, and readily available.

DESIGN

This is a multicenter, prospective, and randomized controlled crossover study. Participants completed a preintervention knowledge test and were then randomized in an allocation ratio of 1:1 to receive surgical education regarding cystoscopy and ureteric stenting for acute renal colic via either PVM or SVC. A 32-point knowledge test and a modified Client Satisfaction Questionnaire-8 were then administered and the participants were then crossed over to the other educational method. The knowledge and satisfaction tests were then readministered.

RESULTS

Fifty-four participants were recruited for this study with 27 participants in each group. Both groups had a 18% to 20% increase in knowledge scores following the first intervention p < 0.001 and on crossover there was a further 4% increase in knowledge scores, p < 0.01. There was no significant difference between the groups in knowledge scores before intervention, p = 0.23 after first intervention p = 0.74 or following crossover p = 0.09. After first intervention, participants in the group receiving PVM education first had a significant 8% higher satisfaction score compared to the SVC group, p = 0.023.

CONCLUSIONS

Our study has shown that PVM shows similar efficacy in information uptake to traditional forms of education. Furthermore, PVM was shown to have higher satisfaction scores compared to SVC. Further studies will need to evaluate the use of PVM for education in other surgical and medical domains and assess the long-term knowledge retention.

Adaptation of Medical Students During Clinical Training: Effects of Holistic Preclinical Education on Clerkship Performance

THEORY

Taiwan's medical undergraduate program at a university or medical center is a continuation of 12 years of compulsory citizenship education rooted in holistic philosophies. Students acquire both technical knowledge and nontechnical attributes, which are necessary for success in further work and life. The early clinical learning experiences of medical students are primarily acquired through clerkships. These clerkships require medical students to apply and extend what they learned during their preclinical education; however, previous studies have explored this issue through examining fragmentary factors such as preclinical course grades and traits but not undertaking comprehensive, whole-person investigations.

HYPOTHESES

To account for the potential benefits of a holistic approach in medical students' learning, we propose three hypotheses: Medical students' preclinical performance on Taiwan's technical and nontechnical higher education assessments are positively associated with their clinical competence (Hypothesis 1) and psychological well-being (Hypothesis 2) during clerkships, and medical students' psychological well-being during clerkships is positively associated with their clinical competence (Hypothesis 3).

METHOD

We studied a cohort of 65 medical students engaged in clerkships from September 2013 to April 2015. Their preclinical technical knowledge scores-formal curricular grades received from course instructors-were obtained from their medical school's archival dataset. Their nontechnical attributes-moral and social performance scores received from student mentors and physical performance scores received from course instructors-were also obtained from the school's archival data set. The medical students' competence in their 2-year clinical clerkships was measured using the objective structured clinical examination scores from the end of both clerkship years. The medical students' psychological well-being during their 2-year clerkships was measured according to burnout level, which was determined using routine online surveys that employed validated, structured, and self-administered questionnaires at each specialty rotation. Multiple regressions and linear mixed-effects model were employed for statistical analysis.

RESULTS

Our study revealed that higher preclinical technical knowledge predicted superior clinical competence and a higher level of burnout during clerkships. By contrast, higher preclinical nontechnical attributes (i.e., higher preclinical moral, social, and physical performance) predicted lower level of burnout. However, no relationship was discovered between clerkship burnout and the clinical competence of the medical students.

CONCLUSIONS

Our study verified the value of a holistic education that encompasses both technical knowledge and nontechnical attributes during the preclinical learning stage for medical students. Our findings can serve as a reference for medical educators designing preclinical educational programs for medical students.