How much do differences in medical schools influence student performance? A longitudinal study employing hierarchical linear modeling

Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review

PURPOSE

To synthesize the evidence of the factors leading to successful performance on knowledge-based national licensure exams (NLEs) for medical students.

METHOD

The authors conducted a scoping review to summarize the peer-reviewed empiric literature that used United States Medical Licensing Examination (USMLE) Step 1 or Step 2 Clinical Knowledge or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 or Level 2 Cognitive Evaluation scores as outcomes. The authors searched PubMed and Scopus without date restrictions through April 30, 2021. Two reviewers independently screened and selected studies for inclusion. Data were summarized narratively and with descriptive statistics.

RESULTS

The authors screened 1,185 unique citations and included 233 full-text studies in their review. Of these, 201 (86%) were studies of USMLE exams, 31 (13%) were studies of COMLEX exams, and 1 (0.4%) reported on both. The authors classified 29 studies (12%) as informing NLE preparation, 163 (70%) as attempting to identify predictive variables, and 76 (33%) as using NLE scores for program evaluation. Preparation studies found that the number of practice test items, practice exam scores, and less time in dedicated preparation correlated with higher NLE scores. Use of other commercial resources or study strategies was not consistently associated with higher scores. Predictive studies found the strongest relationships between individuals' performance on past assessments and their NLE scores.

CONCLUSIONS

The factors leading to successful performance on knowledge-based NLEs align with well-known principles from the cognitive sciences. Learners build on existing foundations of knowledge (reflected in their prior academic performance) and are likely to learn more efficiently with testing and spaced learning over time. While commercial test preparation resources are ubiquitous, there is no evidence that a single resource gives students a competitive advantage on NLEs. Developing habits of regular and continuous learning is necessary for clinical practice and successful NLE performance.

Does performance at the intercollegiate Membership of the Royal Colleges of Surgeons (MRCS) examination vary according to UK medical school and course type? A retrospective cohort study

OBJECTIVES

The knowledge, skills and behaviours required of new UK medical graduates are the same but how these are achieved differs given medical schools vary in their mission, curricula and pedagogy. Medical school differences seem to influence performance on postgraduate assessments. To date, the relationship between medical schools, course types and performance at the Membership of the Royal Colleges of Surgeons examination (MRCS) has not been investigated. Understanding this relationship is vital to achieving alignment across undergraduate and postgraduate training, learning and assessment values.

DESIGN AND PARTICIPANTS

A retrospective longitudinal cohort study of UK medical graduates who attempted MRCS Part A (n=9730) and MRCS Part B (n=4645) between 2007 and 2017, using individual-level linked sociodemographic and prior academic attainment data from the UK Medical Education Database.

METHODS

We studied MRCS performance across all UK medical schools and examined relationships between potential predictors and MRCS performance using χ2 analysis. Multivariate logistic regression models identified independent predictors of MRCS success at first attempt.

RESULTS

MRCS pass rates differed significantly between individual medical schools (p<0.001) but not after adjusting for prior A-Level performance. Candidates from courses other than those described as problem-based learning (PBL) were 53% more likely to pass MRCS Part A (OR 1.53 (95% CI 1.25 to 1.87) and 54% more likely to pass Part B (OR 1.54 (1.05 to 2.25)) at first attempt after adjusting for prior academic performance. Attending a Standard-Entry 5-year medicine programme, having no prior degree and attending a Russell Group university were independent predictors of MRCS success in regression models (p<0.05).

CONCLUSIONS

There are significant differences in MRCS performance between medical schools. However, this variation is largely due to individual factors such as academic ability, rather than medical school factors. This study also highlights group level attainment differences that warrant further investigation to ensure equity within medical training.

Key Assumptions Underlying a Competency-Based Approach to Medical Sciences Education, and Their Applicability to Veterinary Medical Education

This perspective explores six key assumptions of a competency-based approach to medical-sciences education, as they relate to veterinary medical education. Those assumptions, derived from characteristics of competency based medical education (CBME) identified by CBME proponents are: (1) There are sufficient shortcomings in the medical competence of graduate veterinarians that solutions are necessary, and changes in the way we teach veterinarians will address those problems. (2) It is feasible to identify generally accepted core competencies in veterinary medical practice. (3) Teaching to defined learning outcomes will produce greater achievement for learners than approaches that do not emphasize clearly defined outcomes. (4) In veterinary medical education, it is possible to articulate the development of competence sequentially in a manner that is relatively consistent across learners, and carefully planning and sequencing learning activities will produce better learning outcomes. (5) Competency-focused instruction, which tailors the pace and progression of instruction to learners, is feasible in veterinary medical education, and will produce better outcomes than instruction that moves all students through an equivalent process in a set time frame. (6) Programmatic Assessment, including numerous direct observations with feedback, will improve learning outcomes, and is feasible in veterinary medical education. While available research does not unequivocally support all six assumptions, overall the potential benefits of adopting a competency-based approach seem promising for veterinary medical education.

Where we've come from, where we might go

The paper reviews 50 years of research in health sciences education and identifies several recurring controversies-formative versus summative assessment, high and low fidelity simulation, expertise as knowledge versus skills, and the impact of teaching versus curriculum. I then look at the role these may play in the current situation where COVID has necessitated rapid change to distance learning. I then posit an essential role for research in teaching and learning, using multiple methods from qualitative to neuropsychological to better understand the dimensions of effective teaching. The ultimate goal is to operationalize these findings in creation of distance learning modules.

A Study of the Validity of the New MCAT Exam

PURPOSE

To conduct a study of the validity of the new Medical College Admission Test (MCAT).

METHOD

Deidentified data for first- and second-year medical students (185 women, 54.3%; 156 men, 45.7%) who matriculated in 2016 and 2017 to the University of Minnesota Medical School-Twin Cities were included. Of those students, 220 (64.5%) had taken the new MCAT exam and 182 (53.4%) had taken the old MCAT exam (61 [17.9%] had taken both). The authors calculated descriptive statistics and Pearson product moment correlations (r) between new and old MCAT section scores. They conducteda regression analysis of MCAT section scores with Step 1 scores and with preclerkship course performance. They also conducted an exploratory factor analysis (principal component analysis with varimax rotation) of MCAT scores, undergraduate grade point average, Step 1 scores, and course performance.

RESULTS

The new MCAT exam section mean score percentiles ranged from 72 to 78 (mean composite score percentile of 80). The old MCAT exam section mean score percentiles ranged from 84 to 88 (mean composite score percentile of 83). The pattern of correlations among and between new and old MCAT exam section scores (range of r: 0.03-0.67; P < .01) provided evidence of both divergent and convergent validities. Backward multiple regression of new MCAT exam section scores and Step 1 scores resulted in a multiple R of .440; the same analysis with Human Behavior course performance as the dependent variable provided a similar solution with the expected sections of the new MCAT exam (multiple R = .502). The factor analysis resulted in 4 cohesive, theoretically meaningful factors: biomedical knowledge, basic science concepts, cognitive reasoning, and general achievement.

CONCLUSIONS

This study provided empirical evidence of multiple types of validity for the new MCAT exam.

Predictors of USMLE Step 1 Outcomes: Charting Successful Study Habits

BACKGROUND

The United States Medical Licensing Examination Step 1 is a test that affects many aspects of medical students' careers. The aim of this study was to assess the predictive value of various studying habits and academic traits.

ACTIVITY

A survey concerning Step 1 study habits and scores was collected and analyzed.

RESULTS AND DISCUSSION

Study results showed that preclinical curriculum grades, practice test scores, and the number of practice questions completed were positively correlated with Step 1 scores. The strongest predictor of Step 1 scores was preclinical curriculum grades: each unit increase in a letter grade was associated with a 12-point increase in Step 1 scores.

Institutional differences in USMLE Step 1 and 2 CK performance: Cross-sectional study of 89 US allopathic medical schools

Introduction

The United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge (CK) are important for trainee medical knowledge assessment and licensure, medical school program assessment, and residency program applicant screening. Little is known about how USMLE performance varies between institutions. This observational study attempts to identify institutions with above-predicted USMLE performance, which may indicate educational programs successful at promoting students’ medical knowledge.

Methods

Self-reported institution-level data was tabulated from publicly available US News and World Report and Association of American Medical Colleges publications for 131 US allopathic medical schools from 2012–2014. Bivariate and multiple linear regression were performed. The primary outcome was institutional mean USMLE Step 1 and Step 2 CK scores outside a 95% prediction interval (≥2 standard deviations above or below predicted) based on multiple regression accounting for students’ prior academic performance.

Results

Eighty-nine US medical schools (54 public, 35 private) reported complete USMLE scores over the three-year study period, representing over 39,000 examinees. Institutional mean grade point average (GPA) and Medical College Admission Test score (MCAT) achieved an adjusted R² of 72% for Step 1 (standardized β_MCAT 0.7, β_GPA 0.2) and 41% for Step 2 CK (standardized β_MCAT 0.5, β_GPA 0.3) in multiple regression. Using this regression model, 5 institutions were identified with above-predicted institutional USMLE performance, while 3 institutions had below-predicted performance.

Conclusions

This exploratory study identified several US allopathic medical schools with significant above- or below-predicted USMLE performance. Although limited by self-reported data, the findings raise questions about inter-institutional USMLE performance parity, and thus, educational parity. Additional work is needed to determine the etiology and robustness of the observed performance differences.

The USMLE Step 1 Pass/Fail Reporting Proposal: The APDR Position

BACKGROUND

The National Board of Medical Examiners (NBME) and the United States Medical Licensing Examination (USMLE) has convened a conference of "key stakeholders" on March 11-12, 2019 to consider reporting the results of the USMLE Step 1 as pass/fail.

DISCUSSION

While the original purpose of the USMLE Step 1 was to provide an objective basis for medical licensing, the score is increasingly used in residency applicant screening and selection because it is an objective, nationally recognized metric allowing comparison across medical schools in and outside the United States. Excessive reliance on the Step 1 score in the matching process has led to "Step 1 Culture" that drives medical schools to "teach to the test," increases medical student anxiety, and disadvantages minorities that have been shown to score lower on the USMLE Step 1 examination. The outsize role of the USMLE Step 1 score in resident selection is due to lack of standardization in medical school transcripts, grade inflation, and the lack of class standing in many summative assessments. Furthermore, the numeric score allows initial Electronic Residency Application Service filtering, commonly used by programs to limit the number of residency applications to review.

CONCLUSION

The Association of Program Directors in Radiology (APDR) is concerned that pass/fail reporting of the USMLE Step 1 score would take away an objective measure of medical student's knowledge and the incentive to acquire as much of it as possible. Although the APDR is not in favor of the Step 1 exam being used as a screening tool, in the absence of an equal or better metric for applicant comparison the APDR opposes the change in Step 1 reporting from the numeric score to pass/fail.

Clerkship Curriculum Design and USMLE Step 2 Performance: Exploring the Impact of Self-Regulated Exam Preparation

PURPOSE

This study examined medical students' stress and certification exam preparation practices in a reformed clerkship curriculum that excluded high-stakes knowledge testing from end-of-rotation performance evaluation.

METHOD

Stress and exam preparation practices were assessed via a survey comprising locally developed questions, three subscales of the Motivated Strategies for Learning Questionnaire, and two subscales of the Medical Student Stressor Questionnaire. The association between stress, learning self-regulation, and certification exam scores was evaluated retrospectively using non-parametric tests of association (Spearman's rho).

RESULTS

Forty students responded to the survey and consented to use of academic performance data (57% participation rate). Mean certification exam scores were indistinguishable from historical controls. Exam preparation practices resembled those of pre-clinical students: exam-related worrying and time devoted to studying were high, increasing as the exam drew near; preferred study resources were directly analogous to exam questions; and study involved relatively few generative strategies (e.g., concept mapping). Sustaining effort and creating time and space to study were associated with better exam performance, as was participation in this study.

DISCUSSION

On the surface, the absence of regularly spaced, high-stakes testing from clerkship performance evaluation appears to "do no harm" to students' certification exam scores. Students already performing better academically may excel due in part to effective learning self-regulation strategies. However, a clerkship curriculum that does not scaffold self-regulation via cumulative knowledge assessment could further disadvantage students already earning lower scores. Evaluating the impact of curriculum reforms should continuously examine changes to learners' experience in context.

Study Behaviors and USMLE Step 1 Performance: Implications of a Student Self-Directed Parallel Curriculum

PURPOSE

To determine medical students' study behaviors when preparing for the United States Medical Licensing Examination (USMLE) Step 1, and how these behaviors are associated with Step 1 scores when controlling for likely covariates.

METHOD

The authors distributed a study-behaviors survey in 2014 and 2015 at their institution to two cohorts of medical students who had recently taken Step 1. Demographic and academic data were linked to responses. Descriptive statistics, bivariate correlations, and multiple linear regression analyses were performed.

RESULTS

Of 332 medical students, 274 (82.5%) participated. Most students (n = 211; 77.0%) began studying for Step 1 during their preclinical curriculum, increasing their intensity during a protected study period during which they averaged 11.0 hours studying per day (standard deviation [SD] 2.1) over a period of 35.3 days (SD 6.2). Students used numerous third-party resources, including reading an exam-specific 700-page review book on average 2.1 times (SD 0.8) and completing an average of 3,597 practice multiple-choice questions (SD 1,611). Initiating study prior to the designated study period, increased review book usage, and attempting more practice questions were all associated with higher Step 1 scores, even when controlling for Medical College Admission Test scores, preclinical exam performance, and self-identified score goal (adjusted R = 0.56, P < .001).

CONCLUSIONS

Medical students at one public institution engaged in a self-directed, "parallel" Step 1 curriculum using third-party study resources. Several study behaviors were associated with improved USMLE Step 1 performance, informing both institutional- and student-directed preparation for this high-stakes exam.

Assessing Residents' Competency at Baseline: How Much Does the Medical School Matter?

BACKGROUND

Although there is some consensus about the competencies needed to enter residency, the actual skills of graduating medical students may not meet expectations. In addition, little is known about the association between undergraduate medical education and clinical performance at entry into and during residency.

OBJECTIVE

We explored the association between medical school of origin and clinical performance using a multi-station objective structured clinical examination for incoming residents at the University of Michigan Health System.

METHODS

Prior to assuming clinical duties, all first-year residents at the University of Michigan Health System participate in the Postgraduate Orientation Assessment (POA). This assesses competencies needed during the first months of residency. Performance data for 1795 residents were collected between 2002 and 2012. We estimated POA variance by medical school using linear mixed models.

RESULTS

Medical school predicted the following amounts of variance in performance-data gathering scores: 1.67% (95% confidence interval [CI] 0.36-2.93); assessment scores: 4.93% (95% CI 1.84-6.00); teamwork scores: 0.80% (95% CI 0.00-1.82); communication scores: 2.37% (95% CI 0.66-3.83); and overall POA scores: 4.19% (95% CI 1.59-5.35).

CONCLUSIONS

The results show that residents' medical school of origin is weakly associated with clinical competency, highlighting a potential source of variability in undergraduate medical education. The practical significance of these findings needs further evaluation.

Improving completion rates of students in biomedical PhD programs: an interventional study

BACKGROUND

Analysis of graduation success at the University of Split School of Medicine PhD programs conducted in 2011 revealed that only 11% of students who enrolled and completed their graduate coursework between 1999 and 2011 earned a doctoral degree. In this prospective cohort study we evaluated and compared three PhD programs within the same medical school, where the newest program, called Translational Research in Biomedicine (TRIBE), established in the academic year 2010/11, aimed to increase the graduation rate through an innovative approach.

METHODS

The intervention in the new program was related to three domains: redefined recruitment strategy, strict study regulations, and changes to the curriculum. We compared performance of PhD students between the new and existing programs and analyzed their current status, time to obtain a degree (from enrolment to doctorate), age at doctorate, number of publications on which the thesis was based and the impact factor of journals in which these were published.

RESULTS

These improvement strategies were associated with higher thesis completion rate and reduced time to degree for students enrolled in the TRIBE program. There was no change in the impact factor or number of publications that were the basis for the doctoral theses.

CONCLUSION

Our study describes good practices which proved useful in the design or reform of the PhD training program.

An evaluation of outcomes following the replacement of traditional histology laboratories with self-study modules

Changes in medical school curricula often require educators to develop teaching strategies that decrease contact hours while maintaining effective pedagogical methods. When faced with this challenge, faculty at the University of Cincinnati College of Medicine converted the majority of in-person histology laboratory sessions to self-study modules that utilize multiple audiovisual modalities and a virtual microscope platform. Outcomes related to this shift were investigated through performance on in-house examinations, results of the United States Medical Licensing Examination^® (USMLE^® ) Step 1 Examination, and student feedback. Medical School College Admissions Test^® (MCAT^® ) scores were used as a covariate when comparing in-house examinations. Results revealed no significant change in performance on in-house examinations when the content being assessed was controlled (F(2, 506) = 0.676, P = 0.51). A significant improvement in overall practical examination grade averages was associated with the self-study modules (F(6, 1164) = 10.213, P < 0.01), but gradual changes in examination content may explain this finding. The histology and cell biology portion of USMLE Step 1 Examination remained consistent throughout the time period that was investigated. Student feedback regarding the self-study modules was positive and suggested that features such as instructor narrated videos were an important component of the self-study modules because they helped recreate the experience of in-person laboratory sessions. Positive outcomes from the student perspective and no drop in examination performance suggests that utilizing self-study modules for histology laboratory content may be an option for educators faced with the challenge of reducing contact hours without eliminating content. Anat Sci Educ 10: 276-285. © 2016 American Association of Anatomists.

Undergraduate Rigor Scores: Do They Predict Achievement in Veterinary School?

The relations between potential indicators of undergraduate rigor and subsequent achievement in professional school are not clear; some studies have shown that greater undergraduate selectivity is associated with greater achievement in medical science programs, while others have not. We sought to determine the extent to which indicators of undergraduate rigor were associated with achievement in veterinary school. Participants were graduates from three cohorts. The predictors were undergraduate GPA (UGPA), plus five rigor scores-degree or number of undergraduate credits, number of honors courses, number of withdrawals from or repeats of prerequisite science courses, number of part-time semesters, and ratio of community college credits to total college credits. The outcomes were the veterinary medicine cumulative GPA (CVM GPA), Qualifying Exam scores, and North American Veterinary Licensing Exam scores. Using correlations corrected for range restriction, we regressed each outcome on the five rigor scores and UGPA for each of the three graduating cohorts. In most cases, indicators of undergraduate rigor did not predict subsequent achievement in veterinary school; however, in two comparisons, number of honors courses taken as an undergraduate predicted subsequent achievement. UGPA, as expected, predicted CVM GPA. Admissions committees may want to reevaluate whether they include undergraduate rigor when considering admission to their programs, with the caveat that our findings are specific to our institution and are not generalizable.

Curriculum Review and Revision at the University of Minnesota College of Veterinary Medicine

Curriculum review is an essential part of ongoing curriculum development, and is a mandate of the American Veterinary Medical Association Council on Education (AVMA COE), the accrediting body of all North American schools and colleges of veterinary medicine. This article describes the steps in curriculum review undertaken by the University of Minnesota College of Veterinary Medicine (UMN CVM) in response to this mandate from the COE and to a recommendation from a recent collegiate review that was part of a larger university-level strategic planning effort. The challenges of reviewing and revising the curriculum within a short time frame were met by appointing a dedicated curriculum review board and by engaging students and faculty groups, both as focus groups and as specific faculty work sections within disciplines. Faculty voting on the process was very valuable as it permitted the curriculum review board and faculty groups to move ahead knowing there was a process in place for reassessment if most faculty did not agree with recommendations. Consistent support from the dean of the college and other administrators was vital in helping maintain momentum for curriculum review.

The impact of large scale licensing examinations in highly developed countries: a systematic review

BACKGROUND

To investigate the existing evidence base for the validity of large-scale licensing examinations including their impact.

METHODS

Systematic review against a validity framework exploring: Embase (Ovid Medline); Medline (EBSCO); PubMed; Wiley Online; ScienceDirect; and PsychINFO from 2005 to April 2015. All papers were included when they discussed national or large regional (State level) examinations for clinical professionals, linked to examinations in early careers or near the point of graduation, and where success was required to subsequently be able to practice. Using a standardized data extraction form, two independent reviewers extracted study characteristics, with the rest of the team resolving any disagreement. A validity framework was used as developed by the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education to evaluate each paper's evidence to support or refute the validity of national licensing examinations.

RESULTS

24 published articles provided evidence of validity across the five domains of the validity framework. Most papers (n = 22) provided evidence of national licensing examinations relationships to other variables and their consequential validity. Overall there was evidence that those who do well on earlier or on subsequent examinations also do well on national testing. There is a correlation between NLE performance and some patient outcomes and rates of complaints, but no causal evidence has been established.

CONCLUSIONS

The debate around licensure examinations is strong on opinion but weak on validity evidence. This is especially true of the wider claims that licensure examinations improve patient safety and practitioner competence.

Making progress in the ethical treatment of medical trainees

There is an inherent conflict within clinician educators as we balance the roles of healthcare provider to patients in need of care with that of educator of learners in need of teaching. In this essay we use Beauchamp and Childress' principles of biomedical ethics as a framework to compare the relationship that clinician educators have with their patients and their learners, and suggest that while we typically apply ethical principles when addressing the needs of our patients, these principles are frequently lacking in our interactions with learners. This dichotomy reflects a person-by-situation interaction that may be partly explained by the expectations of the regulatory bodies that define how clinicians should interact with patients and how educators should interact with learners. The result is that we may fall short in applying respect for autonomy, beneficence/nonmaleficence, and justice when addressing the needs of our learners. Fortunately there are ways in which we can incorporate these ethical principles into our interactions with learners while still adhering to accreditation standards and institutional policy. These include flipped classrooms and simulated learning experiences, incorporating aspects of instructional design that have been shown to improve learning outcomes, providing additional resources to learners with greater needs, and organizing training curricula around entrustable professional activities. Although the consistent application of ethical principles with all learners during all learning experiences is likely unachievable, we can, and should, move towards more ethical treatment of our learners.

The Merits and Challenges of Three-Year Medical School Curricula: Time for an Evidence-Based Discussion

The debate about three-year medical school curricula has resurfaced recently, driven by rising education debt burden and a predicted physician shortage. In this Perspective, the authors call for an evidence-based discussion of the merits and challenges of three-year curricula. They examine published evidence that suggests that three-year curricula are viable, including studies on three-year curricula in (1) U.S. medical schools in the 1970s and 1980s, (2) two Canadian medical schools with more than four decades of experience with such curricula, and (3) accelerated family medicine and internal medicine programs. They also briefly describe the new three-year programs that are being implemented at eight U.S. medical schools, including their own. Finally, they offer suggestions regarding how to enhance the discussion between the proponents of and those with concerns about three-year curricula.

Student failures on first-year medical basic science courses and the USMLE step 1: a retrospective study over a 20-year period

Correlates of achievement in the basic science years in medical school and on the Step 1 of the United States Medical Licensing Examination® (USMLE®), (Step 1) in relation to preadmission variables have been the subject of considerable study. Preadmissions variables such as the undergraduate grade point average (uGPA) and Medical College Admission Test® (MCAT®) scores, solely or in combination, have previously been found to be predictors of achievement in the basic science years and/or on the Step 1. The purposes of this retrospective study were to: (1) determine if our statistical analysis confirmed previously published relationships between preadmission variables (MCAT, uGPA, and applicant pool size), and (2) study correlates of the number of failures in five M1 courses with those preadmission variables and failures on Step 1. Statistical analysis confirmed previously published relationships between all preadmission variables. Only one course, Microscopic Anatomy, demonstrated significant correlations with all variables studied including the Step 1 failures. Physiology correlated with three of the four variables studied, but not with the Step 1 failures. Analyses such as these provide a tool by which administrators will be able to identify what courses are or are not responding in appropriate ways to changes in the preadmissions variables that signal student performance on the Step 1.

Do medical student stress, health, or quality of life foretell step 1 scores? A comparison of students in traditional and revised preclinical curricula

THEORY

We explored the theory that measures of medical students' well-being and stress from different types of preclinical curricula are linked with performance on standardized assessment.

HYPOTHESES

Self-reported stress and quality of life among sophomore medical students having different types of preclinical curricula will vary in their relationships to USMLE Step 1 scores.

METHOD

Voluntary surveys in 2010 and 2011 compared self-reported stress, physical and mental health, and quality of life with Step 1 scores for beginning sophomore students in the final year of a traditional, discipline-based curriculum and the 1st year of a revised, systems-based curriculum with changed grading system. Wilcoxon rank sum tests and Spearman rank correlations were used to analyze data, significant at p <.05.

RESULTS

New curriculum students reported worse physical health, subjective feelings, leisure activities, social relationships and morale, and more depressive symptoms and life stress than traditional curriculum students. However, among curriculum-related stressors, few differences emerged; revised curriculum sophomores reported less stress working with real and standardized patients than traditional students. There were no class differences in respondents' Step 1 scores. Among emotional and physical health measures, only feelings of morale correlated negatively with Step 1 performance. Revised curriculum students' Step 1 scores correlated negatively with stress from difficulty of coursework.

CONCLUSIONS

Although revised curriculum students reported worse quality of life, general stress, and health and less stress from patient interactions than traditional students, few measures were associated with performance differences on Step 1. Moreover, curriculum type did not appear to either hinder or help students' Step 1 performance. To identify and help students at risk for academic problems, future assessments of correlates of Step 1 performance should be repeated after the new curriculum is well established, relating them also to performance on other standardized assessments of communication skills, professionalism, and later clinical evaluations in clerkships or internships.

Effects of a curricular revision on learner outcomes in veterinary clinical pathology

A mixed-methods evaluation was conducted to study learner attitudes and knowledge about clinical pathology across a curricular change that instituted a stand-alone clinical pathology course in place of content within a previously integrated pathology course structure. Groups of pre- and post-change students were assessed three times across the two semesters leading up to graduation. At each time, rank-ordered and open-ended response items probed attitudes, and multiple-choice items assessed knowledge. Data about student clinical pathology performance were also collected from clinical pathology instructors and supervising clinicians. Student rank-ordered items were evaluated by factor analysis; resulting factor-scale scores, multiple-choice scores, and rank responses from study cohorts were statistically assessed between groups and within each group over time. Intraclass correlations were calculated for the coding of student open-ended responses, and all coded responses were compared among groups. Analysis revealed that students in the revised curriculum had greater satisfaction with their training and greater confidence in data interpretation compared to students without exposure to an independent clinical pathology course. Although differences in knowledge of clinical pathology were not detected, it was also apparent that the independent clinical pathology course filled a student-perceived curricular need without raising criticisms related to diminished integration with anatomic pathology. Secondary study outcomes included formative feedback for course improvement, evidence of clerkship efficacy, and baseline data for further studies.

Differential student attrition and differential exposure mask effects of problem-based learning in curriculum comparison studies

PURPOSE

Studies that compare the effects of problem-based and conventional medical curricula on student performance generally show no significant differences. However, curriculum comparison studies are at best quasi-experimental, so they are prone to forms of selection bias. The purpose of this study was to reanalyze data from such comparisons by controlling for two potential biases: differential student attrition and differential exposure.

METHOD

The authors reanalyzed 104 previously published comparisons involving a single, problem-based medical school in the Netherlands (Maastricht University's medical school), using student attrition and study duration data from this school and the schools with which it was compared. The authors removed bias by reequalizing the comparison groups in terms of attrition and study duration.

RESULTS

The uncorrected data showed no differences between problem-based and conventional curricula: Mean effect sizes as expressed by Cohen d were 0.02 for medical knowledge and 0.07 for diagnostic reasoning. However, the reanalysis demonstrated medium-level effect sizes favoring the problem-based curriculum. After corrections for attrition and study duration, the mean effect size for knowledge acquisition was 0.31 and for diagnostic reasoning was 0.51.

CONCLUSIONS

Effects of the Maastricht problem-based curriculum were masked by differential attrition and differential exposure in the original studies. Because this school has been involved in many studies included in influential literature reviews published in the past 20 years, the authors' findings have implications for the assessment of the value of problem-based learning put forward by these reviews.

Medical education: past, present and future

This article reviews changes in undergraduate and postgraduate medical education since the Flexner report of 1910. I argue that many of the changes in the twentieth century could be viewed as 'post-Flexnerian', and related to the integration of biomedical science in the preclinical medical curriculum. I then go on to argue that recent changes in the health care systems worldwide will force a critical re-examination of our approach to clinical education-a 'post-Oslerian' era. I suggest that one approach would be to decouple clinical education from clinical care, to some degree, and supplement with curricula designed around careful sequencing of simulated cases.

Commentary: measurement and interpretation challenges in comparing student performance outcomes from different medical schools

This commentary points to several measurement issues that arise in assessing medical student performance outcomes and then discusses the challenge of interpreting between-school differences. A problem often encountered in assessing student learning is creating an instrument that is at the right "pay grade." If it is too easy, ceiling effects compress scores. If it is too difficult, examinee performance can compress about chance values, and morale problems can occur. These issues are discussed in the context of a report by Williams and colleagues that measures medical student performance across five institutions on instruments assessing diagnostic pattern recognition and clinical data interpretation. The author of this commentary observes that, when interpreting between-school differences in assessing student learning, what can seem like small differences can have important consequences.

Tracking development of clinical reasoning ability across five medical schools using a progress test

PURPOSE

Little is known about the acquisition of clinical reasoning skills in medical school, the development of clinical reasoning over the medical curriculum as a whole, and the impact of various curricular methodologies on these skills. This study investigated (1) whether there are differences in clinical reasoning skills between learners at different years of medical school, and (2) whether there are differences in performance between students at schools with various curricular methodologies.

METHOD

Students (n = 2,394) who had completed zero to three years of medical school at five U.S. medical schools participated in a cross-sectional study in 2008. Students took the same diagnostic pattern recognition (DPR) and clinical data interpretation (CDI) tests. Percent correct scores were used to determine performance differences. Data from all schools and students at all levels were aggregated for further analysis.

RESULTS

Student performance increased substantially as a result of each year of training. Gains in DPR and CDI performance during the third year of medical school were not as great as in previous years across the five schools. CDI performance and performance gains were lower than DPR performance and gains. Performance gains attributable to training at each of the participating medical schools were more similar than different.

CONCLUSIONS

Years of training accounted for most of the variation in DPR and CDI performance. As a rule, students at higher training levels performed better on both tests, though the expected larger gains during the third year of medical school did not materialize.

The effect of personal character on the results of clinical performance skill tests

PURPOSE

Even though many studies have indicated that the personality of medical students affects learning style and academic achievement, the effect of personality types on the performance skill tests has not been well known in the medical field due to the rarity of published papers. Thus, the aim of this study was to reveal the effect of personal traits on clinical skill performance tests.

METHODS

Fifty-seven fourth-grade medical students were enrolled in this study. They had all completed clinical performance tests. To assess personality types, we used the Korean version of the Myers-Briggs Type Indicator (MBTI).

RESULTS

Fifty-five of 57 senior medical students responded completely to the MBTI questionnaire. The proportion of four paired MBTI dimensions was Introversion (I)-Extroversion (E) (67.3% vs. 32.7%), Sensing (S)-Intuition (I) (76.4% vs. 23.6%), Thinking (T)-Feeling (F) (61.8% vs. 38.2%), and Judging (J)-Perception (P) (56.4% vs. 43.6%). The dominant personality types were ISTJ (23.6%), ESTJ (14.5%), and ISTP (10.9%). The first objective structured clinical examination (OSCE) test showed higher scores in Extraversion, Judging, and Sensing-Judging types compared to the counterparts (p<0.05), but this effect was not observed in the second OSCE test. On the clinical performance examination, Extraversion, Sensing, and Judging types had a higher score, as measured by standardized patients.

CONCLUSION

Specific personal traits affect the test scores of the clinical performance skill examinations. So, personality measurement might be a useful tool for understanding a student who has difficulty in performance tests. We hope this study will give valuable information to examiners when they instruct and counsel students about clinical performance tests.

A 3-year medical curriculum

The cost of medical education keeps rising while physician shortages loom, student diversity is threatened, and maldistribution continues to affect the physician workforce. The author discusses the rationale for a 3-year undergraduate medical curriculum with reduced costs and proposes a structure with funding for it. Medical educators may consider the issues raised and continue scholarly discussion to promote sustainable, cost effective medical education to address future physician workforce needs.

A descriptive study of medical educators' views of problem-based learning

BACKGROUND

There is a growing amount of literature on the benefits and drawbacks of Problem-Based Learning (PBL) compared to conventional curricula. However, it seems that PBL research studies do not provide information rigorously and formally that can contribute to making evidence-based medical education decisions. The authors performed an investigation aimed at medical education scholars around the question, "What are the views of medical educators concerning the PBL approach?"

METHODS

After framing the question, the method of data collection relied on asking medical educators to report their views on PBL. Two methods were used for collecting data: the questionnaire survey and an online discussion forum.

RESULTS

The descriptive analysis of the study showed that many participants value the PBL approach in the practice and training of doctors. However, some participants hold contrasting views upon the importance of the PBL approach in basic medical education. For example, more than a third of participants (38.5%) had a neutral stance on PBL as a student-oriented educational approach. The same proportion of participants also had a neutral view of the efficiency of traditional learning compared to a PBL tutorial. The open-ended question explored the importance of faculty development in PBL. A few participants had negative perceptions of the epistemological assumptions of PBL. Two themes emerged from the analysis of the forum repliers: the importance of the faculty role and self-managed education.

CONCLUSION

Whilst many participants valued the importance of the PBL approach in the practice and training of doctors and agreed with most of the conventional descriptions of PBL, some participants held contrasting views on the importance of the PBL approach in undergraduate medical education. However there was a strong view concerning the importance of facilitator training. More research is needed to understand the process of PBL better.

An analysis of long-term outcomes of the impact of curriculum: a comparison of the three- and four-year medical school curricula

PURPOSE

To determine the long-term effects of curriculum length on physician competence, the authors compared the performance of graduates from the University of Calgary (U of C; a school with a three-year curriculum) with matched samples from the University of Alberta (U of A) and from other Canadian schools with a four-year curriculum.

METHOD

The authors used data from the College of Physicians and Surgeons of Alberta, Physician Achievement Review (PAR) program to determine curricular outcomes. The authors analyzed PAR program data, comprising reviews from medical colleagues, nonphysician coworkers (e.g., nurses, pharmacists), patients, and the physicians themselves, for 166 physicians from U of C, U of A, and other universities. They compared groups using one-way analysis of covariance (ANCOVA) and multivariate analysis of covariance (MANCOVA), with years since graduation as a covariate, and a Cohen d effect size calculation to assess the magnitude of the differences.

RESULTS

The authors analyzed review data for 498 physicians. The results of ANCOVA showed that no significant differences existed among schools for the self and the patient aggregate mean questionnaire scores. Aggregate mean questionnaire scores from the medical colleague and coworker surveys were significant, albeit with a small effect size. MANCOVA showed small but significant differences among schools on the aggregate mean factor scores for medical colleague, coworker, and patient questionnaires.

CONCLUSIONS

Although differences among schools exist, they are small. They suggest at least equivalent performance for graduates of three- and four-year medical schools who practice in Alberta.

Five myths and the case against a European or national licensing examination

The introduction of a European licensing examination or national examinations, where these do not already exist, offers significant advantages. These are more than offset, however, by the disadvantages and the collateral damage incurred. Five myths about centralizing examinations are explored. Myth 1: The claim that a central examination will ensure that candidates are assessed in important areas of medical practice is unfounded. What tends to be assessed are learning outcomes that can be easily assessed. These are often not the important outcomes related to the overall competence of a doctor. Myth 2: It is claimed that a central examination will lead to improvements in assessment practice. The evidence is that this is not the case and that, in fact, a central examination stifles change and inhibits innovation. Myth 3: A central examination, it is suggested, will meet a need for greater uniformity. There is also an important need to recognize diversity. Myth 4: Central examinations are seen as an indicator that will track the performance of the system. The limitations of the data, however, are usually not recognized and there maybe unfortunate and unintended consequences if the results are used in this way. Myth 5: Finally, a major argument proposed for a European or national examination is that it will lead to safer medical practice and that this will protect the patient from substandard practitioners. There is, in fact, no evidence to support this argument. There is a need for further work and new initiatives on standards and quality improvement in assessment approaches. This can be achieved in a number of ways including monitoring the assessment process and sharing tools and assessment approaches between schools.