Assessing the effects of the 2003 resident duty hours reform on internal medicine board scores

USMLE Performance, Subsequent Standardized Testing, and ABIM Certification Exam Preparation for Internal Medicine Residency Programs: A Narrative Review

Standardized examinations measure progress throughout medical education. Successful completion of the American Board of Internal Medicine Certification Examination (ABIM-CE) benchmarks completion of internal medicine (IM) residency training. Recent declines in initial ABIM-CE pass rates may prompt residency programs to examine strategies to improve learner performance. We synthesized published literature on associations between the United States Medical Licensing Examination (USMLE), in-training examination (ITE), and board preparation to support residents for ABIM-CE. We searched MEDLINE for test performance and preparation strategies for IM board certification during training. Relevant articles published until March 15, 2024, were screened using pre-defined criteria for narrative review, then codified into three domains (USMLE, ITE, curriculum/program strategies). Findings were grouped by theme into considerations for training programs seeking guidance on learning augmentation plans to improve resident performance on ABIM-CE. Themes drawn from articles focused on USMLE include validity in predicting CE performance, noting (1) failing USMLE Step 1 is associated with failing ABIM-CE, (2) any USMLE score < 220 increases failure probability, and (3) a mean USMLE ≥ 250 approximates ~ 100% pass rates on board examination. Inferences from ITE-focused articles support use as a predictive tool; specifically, a score < 35th percentile signals a resident at risk for failing the ABIM-CE while > 70th percentile is predictive of passing. Lastly, inferences from curriculum- and program-focused articles suggest standard contents (conferences) do not correlate with CE passage, while targeted clinical reasoning and remediation plans do. IM residency programs should consider adopting learning augmentation strategies targeted to at-risk residents to support CE passage.

Resident factors associated with American board of internal medicine certification exam failure

INTRODUCTION

Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor.

MATERIALS AND METHODS

We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors.

RESULTS

Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result.

DISCUSSION

Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.

Impact of duty hours on competency-related knowledge acquisition among community hospital residents

Background

The effect of duty hour (DH) restrictions on postgraduate residents' acquisition of clinical competencies is unclear. We evaluated the relationship between DHs and competency-related knowledge acquisition using the General Medicine In-training Examination (GM-ITE).

Methods

We conducted a multicenter, cross-sectional study of community hospital residents among 2019 GM-ITE examinees. Self-reported average DHs per week were classified into five DH categories and the competency domains were classified into four areas: symptomatology and clinical reasoning (CR), physical examination and clinical procedure (PP), medical interview and professionalism (MP), and disease knowledge (DK). The association between these scores and DHs was examined using random-intercept linear models with and without adjustment for confounding factors.

Results

We included 4753 participants in the analyses. Of these, 31% were women, and 49.1% were in the postgraduate year (PGY) 2. Mean CR and MP scores were lower among residents in Category 1 (<50 h) than in residents in Category 3 (≥60 and <70 h; reference group). Mean DK scores were lower among residents in Categories 1 and 2 (≥50 and <60 h) than in the reference group. PGY-2 residents in Categories 1 and 2 had lower CR scores than those in Category 3; however, PGY-1 residents in Category 5 showed higher scores.

Conclusions

The relationship between DHs and each competency area is not strictly linear. The acquisition of knowledge of physical examination and clinical procedures skills in particular may not be related to DHs.

Development of the individualised Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) trial: a protocol summary of a national cluster-randomised trial of resident duty hour policies in internal medicine

INTRODUCTION

Medical trainees' duty hours have received attention globally; restrictions in Europe, New Zealand and some Canadian provinces are much lower than the 80 hours per week enforced in USA. In USA, resident duty hours have been implemented without evidence simultaneously reflecting competing concerns about patient safety and physician education. The objective is to prospectively evaluate the implications of alternative resident duty hour rules for patient safety, trainee education and intern sleep and alertness.

METHODS AND ANALYSIS

63 US internal medicine training programmes were randomly assigned 1:1 to the 2011 Accreditation Council for Graduate Medical Education resident duty hour rules or to rules more flexible in intern shift length and number of hours off between shifts for academic year 2015-2016. The primary outcome is calculated for each programme as the difference in 30-day mortality rate among Medicare beneficiaries with any of several prespecified principal diagnoses in the intervention year minus 30-day mortality in the preintervention year among Medicare beneficiaries with any of several prespecified principal diagnoses. Additional safety outcomes include readmission rates, prolonged length of stay and costs. Measures derived from trainees' and faculty responses to surveys and from time-motion studies of interns compare the educational experiences of residents. Measures derived from wrist actigraphy, subjective ratings and psychomotor vigilance testing compare the sleep and alertness of interns. Differences between duty hour groups in outcomes will be assessed by intention-to-treat analyses.

ETHICS AND DISSEMINATION

The University of Pennsylvania Institutional Review Board (IRB) approved the protocol and served as the IRB of record for 40 programmes that agreed to sign an Institutional Affiliation Agreement. Twenty-three programmes opted for a local review process.

TRIAL REGISTRATION NUMBER

NCT02274818; Pre-results.

Relationships Between the ACGME Resident and Faculty Surveys and Program Pass Rates on the ABIM Internal Medicine Certification Examination

PURPOSE

The Accreditation Council for Graduate Medical Education (ACGME) has surveyed residents since 2003, and faculty since 2012. Surveys are designed to assess program functioning and specify areas for improvement. The purpose of this study was to assess the association of the ACGME's resident and faculty surveys with residency-program-specific performance on the American Board of Internal Medicine (ABIM) certification exam.

METHOD

Data were available from residents and faculty in 375 U.S. ACGME-accredited internal medicine programs from the 2012-2013, 2013-2014, and 2014-2015 academic years. Analysis of variance and correlations were used to examine the relationship between noncompliance with ACGME program requirements as assessed by the resident and faculty surveys, and ABIM program pass rates.

RESULTS

Noncompliance reported on the resident and faculty surveys was highest for programs not meeting the ACGME program requirement of an 80% pass rate on the ABIM certification examination. This relationship was significant for overall noncompliance, both within the resident (P < .001) and faculty (P < .05) surveys, for many areas within the two surveys (correlations ranged between -.07 and -.25, and P values ranged between .20 and < .001), and for the highest levels of noncompliance across areas of the resident (P < .001) and faculty (P < .04) surveys.

CONCLUSIONS

ACGME resident and faculty surveys were significantly associated with ABIM program pass rates, supporting the importance of these surveys within the ACGME's Next Accreditation System.

A 15-year review of the Stanford Internal Medicine Residency Program: predictors of resident satisfaction and dissatisfaction

INTRODUCTION

Satisfaction with training and with educational experiences represents important internal medicine (IM) programmatic goals. Graduates from IM residency programs are uniquely poised to provide insights into their educational and training experiences and to assess whether these experiences were satisfactory and relevant to their current employment.

METHODS

We surveyed former IM residents from the training program held during the years 2000-2015 at the Department of Medicine, Stanford University. The first part of the survey reviewed the IM residency program and the second part sought identifying data regarding gender, race, ethnicity, work, relationships, and financial matters. The primary outcome was satisfaction with the residency experience.

RESULTS

Of the 405 individuals who completed the Stanford IM residency program in the study period, we identified 384 (95%) former residents with a known email address. Two hundred and one (52%) former residents responded to the first part and 185 (48%) answered both the parts of the survey. The mean age of the respondents was 36.9 years; 44% were female and the mean time from IM residency was 6.1 (±4.3) years. Fifty-eight percent reported extreme satisfaction with their IM residency experience. Predictors associated with being less than extremely satisfied included insufficient outpatient experience, insufficient international experience, insufficient clinical research experience, and insufficient time spent with family and peers.

CONCLUSION

The residents expressed an overall high satisfaction rate with their IM training. The survey results provided insights for improving satisfaction with IM residency training that includes diversifying and broadening IM training experiences.

The Relationship Between ACGME Duty Hour Requirements and Performance on the American Board of Emergency Medicine Qualifying Examination

BACKGROUND

In 2003, the Accreditation Council for Graduate Medical Education (ACGME) instituted requirements that limited the number of hours residents could spend on duty, and in 2011, it revised these requirements.

OBJECTIVE

This study explored whether the implementation of the 2003 and 2011 duty hour limits was associated with a change in emergency medicine residents' performance on the American Board of Emergency Medicine (ABEM) Qualifying Examination (QE).

METHODS

Beginning with the 1999 QE and ending with the 2014 QE, candidates for whom all training occurred without duty hour requirements (Group A), candidates under the first set of duty hour requirements (Group C), and candidates under the second set of duty hour requirements (Group E) were compared. Comparisons included mean scores and pass rates.

RESULTS

In Group A, 5690 candidates completed the examination, with a mean score of 82.8 and a 90.2% pass rate. In Group C, 8333 candidates had a mean score of 82.4 and a 90.5% pass rate. In Group E, there were 1269 candidates, with a mean score of 82.5 and an 89.4% pass rate. There was a small but statistically significant decrease in the mean scores (0.04, P < .001) after implementation of the first duty hour requirements, but this difference did not occur after implementation of the 2011 standards. There was no difference among pass rates for any of the study groups (χ² = 1.68, P = .43).

CONCLUSIONS

We did not identify an association between the 2003 and 2011 ACGME duty hour requirements and performance of test takers on the ABEM QE.

The Impact of Resident Duty Hour and Supervision Changes: A Review

In 2003 and again in 2011, the Accreditation Council for Graduate Medical Education (ACGME) instituted increasingly stringent requirements governing resident hours and supervision, with the goals of improving patient safety, resident well-being, and resident education. Although the changes initially stemmed from a catastrophic outcome in a patient treated with psychotropic medications and behavioral restraints, and have been in place over a decade, many psychiatrists are not familiar with these changes and with their potential effects and "side effects" on today's trainees and faculty. The authors review the history leading to these changes, summarize the revised requirements, and review representative literature regarding the impact of the changes. The existing studies of the impact of the new requirements on improving patient safety, resident well-being, and resident education are inconclusive, and most editorials, perspectives, and surveys of faculty and residents reflect a lack of enthusiasm for the changes. They go on to suggest the need for evidence-based outcome studies prior to the institution of further major changes in ACGME requirements.

On resident duty hour restrictions and neurosurgical training: review of the literature

Within neurosurgery, the national mandate of the 2003 duty hour restrictions (DHR) by the Accreditation Council for Graduate Medical Education (ACGME) has been controversial. Ensuring the proper education and psychological well-being of residents while fulfilling the primary purpose of patient care has generated much debate. Most medical disciplines have developed strategies that address service needs while meeting educational goals. Additionally, there are numerous studies from those disciplines; however, they are not specifically relevant to the needs of a neurosurgical residency. The recent implementation of the 2011 DHR specifically aimed at limiting interns to 16-hour duty shifts has proven controversial and challenging across the nation for neurosurgical residencies--again bringing education and service needs into conflict. In this report the current literature on DHR is reviewed, with special attention paid to neurosurgical residencies, discussing resident fatigue, technical training, and patient safety. Where appropriate, other specialty studies have been included. The authors believe that a one-size-fits-all approach to residency training mandated by the ACGME is not appropriate for the training of neurosurgical residents. In the authors' opinion, an arbitrary timeline designed to limit resident fatigue limits patient care and technical training, and has not improved patient safety.

Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam

OBJECTIVES

To evaluate the relationship between the pass rate of the American Board of Internal Medicine (ABIM) certifying exam and the characteristics of residency programs.

METHODS

The study used a retrospective, cross-sectional design with publicly available data from the ABIM and the Fellowship and Residency Electronic Interactive Database. All categorical residency programs with reported pass rates were included. Using univariate and multivariate, linear regression analyses, I analyzed how 69 factors (e.g., location, general information, number of faculty and trainees, work schedule, educational environment) are related to the pass rate.

RESULTS

Of 371 programs, only one region had a significantly different pass rate from the other regions; however, as no other characteristics were reported in this region, I excluded program location from further analysis. In the multivariate analysis, pass rate was significantly associated with four program characteristics: ratio of full-time equivalent paid faculty to positions, percentage of osteopathic doctors, formal mentoring program, and on-site child care (OCC). Numerous factors were not associated at all, including minimum exam scores, salary, vacation days, and average hours per week.

CONCLUSIONS

As shown through the ratio of full-time equivalent paid faculty to positions and whether there was a formal mentoring program, a highly supervised training experience was strongly associated with the pass rate. In contrast, percentage of osteopathic doctors was inversely related to the pass rate. Programs with OCC significantly outperformed programs without OCC. This study suggested that enhancing supervision of training programs and offering parental support may help attract and produce competitive residents.

Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam

OBJECTIVES

To evaluate the relationship between the pass rate of the American Board of Internal Medicine (ABIM) certifying exam and the characteristics of residency programs.

METHODS

The study used a retrospective, cross-sectional design with publicly available data from the ABIM and the Fellowship and Residency Electronic Interactive Database. All categorical residency programs with reported pass rates were included. Using univariate and multivariate, linear regression analyses, I analyzed how 69 factors (e.g., location, general information, number of faculty and trainees, work schedule, educational environment) are related to the pass rate.

RESULTS

Of 371 programs, only one region had a significantly different pass rate from the other regions; however, as no other characteristics were reported in this region, I excluded program location from further analysis. In the multivariate analysis, pass rate was significantly associated with four program characteristics: ratio of full-time equivalent paid faculty to positions, percentage of osteopathic doctors, formal mentoring program, and on-site child care (OCC). Numerous factors were not associated at all, including minimum exam scores, salary, vacation days, and average hours per week.

CONCLUSIONS

As shown through the ratio of full-time equivalent paid faculty to positions and whether there was a formal mentoring program, a highly supervised training experience was strongly associated with the pass rate. In contrast, percentage of osteopathic doctors was inversely related to the pass rate. Programs with OCC significantly outperformed programs without OCC. This study suggested that enhancing supervision of training programs and offering parental support may help attract and produce competitive residents.

How to Further Decrease the Efficiency of Care at a Level I Trauma Center: Implement the Amended Resident Work Hours

Work-hour restrictions were amended in 2011 to limit interns to 16 continuous duty hours, essentially requiring a night float system of 12-hour shifts. We hypothesize that there has been no improvement in outcomes after implementation of the amended work-hour restrictions. Outcomes from trauma admissions were queried from the trauma registry from 2009 to 2011 (PRE) and 2011 to 2013 (POST). The primary outcome was mortality with secondary outcomes intensive care unit length of stay (LOS) and LOS. Patients were stratified based on age, mechanism, gender, blood pressure, heart rate, and injury severity (Injury Severity Score, Glasgow Coma Scale, Base Deficit). Outcomes were then compared from admissions PRE to POST. A total of 9178 patients were included in the study population. The mean age was 42 with most being male (72%) and blunt mechanism (81%). Patient populations were well matched except patients in the POST period were slightly older (43 vs 42 years; P = 0.01). Intensive care unit LOS and LOS were higher in the POST period. After adjusted analysis, admission in the POST period was not a predictor of mortality (odds ratio 0.857; confidence interval 0.655–1.12). The POST period was an independent predictor for LOS (β = 0.74; P = 0.002). This study adds to the mounting evidence that the implementation of the amended limits on work hours leads to furthermore decreased efficiency of care.