Resident factors associated with American board of internal medicine certification exam failure

INTRODUCTION

Performance on the certifying examinations such as the American Board of Internal Medicine Certification Exam (ABIM-CE) is of great interest to residents and their residency programs. Identification of factors associated with certification exam result may allow residency programs to recognize and intervene for residents at risk of failing. Despite this, residency programs have few evidence-based predictors of certification exam outcome. The change to pass-or-fail score reporting of the USA Medical Licensing Exam (USMLE) Step 1 removes one such predictor.

MATERIALS AND METHODS

We performed a retrospective study of residents from a medium-sized internal medicine residency program who graduated from 1998 through 2017. We used univariate tests of associations between ABIM-CE result and various demographic and scholastic factors.

RESULTS

Of 166 graduates, 14 (8.4%) failed the ABIM-CE on the first attempt. Failing the first attempt of the ABIM-CE was associated with older median age on entering residency (29 vs 27 years; P = 0.01); lower percentile rank on the Internal Medicine In-Training Examination (IM-ITE) in each of the first, second, and third years of training (P < 0.001 for all); and lower scores on the USMLE Steps 1, 2 Clinical Knowledge, and 3 (P < 0.05 for all). No association was seen between a variety of other scholastic or demographic factors and first-attempt ABIM-CE result.

DISCUSSION

Although USMLE step 1 has changed to a pass-or-fail reporting structure, there are still other characteristics that allow residency programs to identify residents at risk of ABIM-CE first time failure and who may benefit from intervention.

Performance of ChatGPT in Board Examinations for Specialists in the Japanese Ophthalmology Society

We investigated the potential of ChatGPT in the ophthalmological field in the Japanese language using board examinations for specialists in the Japanese Ophthalmology Society. We tested GPT-3.5 and GPT-4-based ChatGPT on five sets of past board examination problems in July 2023. Japanese text was used as the prompt adopting two strategies: zero- and few-shot prompting. We compared the correct answer rate of ChatGPT with that of actual examinees, and the performance characteristics in 10 subspecialties were assessed. ChatGPT-3.5 and ChatGPT-4 correctly answered 112 (22.4%) and 229 (45.8%) out of 500 questions with simple zero-shot prompting, respectively, and ChatGPT-4 correctly answered 231 (46.2%) questions with few-shot prompting. The correct answer rates of ChatGPT-3.5 were approximately two to three times lower than those of the actual examinees for each examination set (p = 0.001). However, the correct answer rates for ChatGPT-4 were close to approximately 70% of those of the examinees. ChatGPT-4 had the highest correct answer rate (71.4% with zero-shot prompting and 61.9% with few-shot prompting) in "blepharoplasty, orbit, and ocular oncology," and the lowest answer rate (30.0% with zero-shot prompting and 23.3% with few-shot prompting) in "pediatric ophthalmology." We concluded that ChatGPT could be one of the advanced technologies for practical tools in Japanese ophthalmology.

Resources That Improve Medical Board Licensing Examination Performance

Purpose

Examine the factors improving performance on national medical licensing board examinations.

Rationale

Accreditation Council for Graduate Medical Education (ACGME) accredited residency programs report the United States Medical Licensing Examination (USMLE) Step 1 and Comprehensive Osteopathic Licensing Examination-USA (COMLEX-USA) Level 1 scores as the most important criteria in selecting candidates to interview.

Hypotheses

(1) Certain resources are superior for exam preparation.

(2) Certain practice tests better assess exam preparedness.

(3) USMLE performance will correlate with the COMLEX-USA.

Methods

One-hundred and two (102) medical students were surveyed regarding preparation for and performance on COMLEX-USA Level 1 and USMLE Step 1.

Results

USMLE-specific question banks were positively correlated with performance on COMLEX-USA Level 1 and USMLE Step 1 while COMLEX-specific question banks showed no correlation. National Board of Medical Examiners (NBME) Comprehensive Basic Science Self Assessment (CBSSA) and National Board of Osteopathic Medical Examiners (NBOME) Comprehensive Osteopathic Medical Self-Assessment Examination (COMSAE) practice examinations were positively correlated with performance on the USMLE Step 1 and the COMLEX-USA Level 1. Scores on USMLE Step 1 and COMLEX-USA Level 1 were highly correlated. Students who took USMLE Step 1 performed better on COMLEX-USA Level 1 than those who did not.

Conclusion

COMLEX-specific resources may not adequately prepare students for COMLEX-USA Level 1. Students studying for COMLEX-USA Level 1 may benefit by preparing for USMLE Step 1.

A "U-shaped" Curve: Appreciating How Primary Care Residency Intention Relates to the Cost of Board Preparation and Examination

Introduction: The shortage of primary care physicians in the United States has warranted an investigation into how medical education debt and other factors influence medical students’ interests in primary care (PC) residencies. However, sparse research has studied how the cost of board preparation and examination relates to career choice. The objective of this study was to determine if there is an association between the cost of preparing and sitting for board examinations and the intention to enter a PC residency for osteopathic medical students.

Methods: We postulated that students who incurred higher financial costs from preparing and sitting for board examinations would be more likely to be interested in non-primary care (NPC) residencies. Using a non-experimental survey design, this study asked respondents to evaluate the following: “I plan to enter a Primary Care Residency (Family Medicine OR General Internal Medicine OR Pediatrics)” using a Likert scale. Respondents were also asked to select which board examination(s) and pertinent resource(s) they had purchased. Total costs were calculated per student.

Results: A total of 25,852 osteopathic medical students received the survey, of which 1,280 students responded to and completed it, yielding a 4.95% response rate. The distribution of respondents’ intentions to pursue a primary care residency and costs spent yielded a “U” shaped curve. Respondents who Strongly Agreed and Strongly Disagreed to the statement “I plan to enter a Primary Care Residency” spent $5,744 and $5,070 on board-preparation and examination, respectively. No statistically significant differences were found between the cost of preparing and sitting for board examinations and the intention to enter primary care residencies when individuals were grouped by year in school and gender.

Conclusions: Because competitive NPC specialties have relatively higher salaries, we suspected that students who intended to pursue these specialties would have had higher financial costs from board examination and preparation compared to students who intended to pursue PC residencies such as family medicine. Our findings further illustrate these specific educational costs do not correlate with students’ stated intentions to enter primary care residencies. As efforts continue to determine a solution for the primary care physician shortage, it becomes clearer that the focus must also encompass non-financial influences that shape career choice. 

Research during Pediatric Residency Training: A Nationwide Study in Japan

Introduction

Training in scholarship is an essential component of postgraduate education. Previous studies worldwide on the research activities of pediatric residents were questionnaires targeting program directors or surveys conducted in a limited number of institutions; however, no nationwide studies have been conducted. The objective of this study was to describe the research activities of pediatric residents.

Methods

We conducted a nationwide cross-sectional study during 2015 and 2016 in Japan. Study data were collected from each resident's logbook submitted to the board examination office and compared by the type of institution, namely, university, children's, or community hospital.

Results

Of 1,718 eligible participants, 1,500 participated in this study. Overall, 499 (33.3%) residents trained at national/public university hospitals, 371 (24.7%) at private university hospitals, 140 (9.3%) at children's hospitals, and 490 (32.7%) at community hospitals. Although 1,361 (90.7%) residents gave at least one presentation at an academic conference during their residency, only 235 (15.7%) residents published one or more articles in a peer-reviewed academic medical journal. The proportion of residents who gave at least one presentation (p=0.03) and published at least one study (p<0.01) differed significantly among the types of institutions. Residents at community hospitals gave fewer presentations at conferences (odds ratio [OR] 0.56; 95% confidence interval [95% CI] 0.36-0.87) and published fewer peer-reviewed articles (OR 0.53; 95% CI 0.37-0.76) than residents at national/public university hospitals.

Conclusions

This is apparently the first nationwide study demonstrating that the research activities of pediatric residents consisted mostly of presentations at academic conferences, but also that most residents had not published their research. There was a marked variation in residents' academic activities by institution type.

What Is "Preclinical Device Pathology": An Introduction of the Unfamiliar

Medical device pathologists are involved in the preclinical evaluation of medical devices that will be temporarily inserted or permanently and often irreversibly implanted in the human body. The medical device industry is technology based, allowing for rapid device iterations; innovations occur at an accelerated rate compared to the innovations in the pharmaceutical industry. The device pathologist provides the pathology results and is, by training and experience, in an ideal position to help the medical engineer and innovator tackle biomedical problems and to comment on the possible and actual outcomes of preclinical studies. Device pathology expertise is typically a necessity in the prelude for regulatory submission. However, there is a lack of detailed guidelines for a comprehensive preclinical pathology evaluation of the final product after implantation in a test animal. What specifically unites device pathologists is the reliance on gross pathology as the basis for spatial context needed for appropriate histopathologic analyses, the knowledge of detailed protocol instructions, a good understanding of wound healing including the "implant trauma," and interaction with ambitious device innovators. In this article, it is my aim to amalgamate the following articles in this issue with pertinent background information intended to be informative, critical, and stimulating.

(Lack of) Measurable Clinical or Knowledge Gains From Resident Participation in Noon Conference

RATIONALE AND OBJECTIVES

The objective of this study was to determine whether noon conference attendance by diagnostic radiology residents is predictive of measurable performance.

METHODS

This single-center retrospective Health Insurance and Portability and Accountability Act (HIPAA)-compliant cross-sectional study was considered "not regulated" by the institutional review board. All diagnostic radiology residents who began residency training from 2008 to 2012 were included (N = 54). Metrics of clinical performance and knowledge were collected, including junior and senior precall test results, American Board of Radiology scores (z-score transformed), American College of Radiology in-training scores (years 1-3), on-call "great call" and minor and major discrepancy rates, on-call and daytime case volumes, and training rotation scores. Multivariate regression models were constructed to determine if conference attendance, match rank order, or starting year could predict these outcomes. Pearson bivariate correlations were calculated.

RESULTS

Senior precall test results were moderately correlated with American Board of Radiology (r = 0.41) and American College of Radiology (r = 0.38-0.48) test results and mean rotation scores (r = 0.41), indicating moderate internal validity. However, conference attendance, match rank order, and year of training did not correlate with (r = -0.16-0.16) or predict (P > .05) measurable resident knowledge. On multivariate analysis, neither match rank order (P = .14-.96) nor conference attendance (P = .10-.88) predicted measurable clinical efficiency or accuracy. Year started training predicted greater cross-sectional case volume (P < .0001, β = 0.361-0.516) and less faculty-to-resident feedback (P < 0.0001, β = [-0.628]-[-0.733]).

CONCLUSIONS

Residents with lower conference attendance are indistinguishable from those who attend more frequently in a wide range of clinical and knowledge-based performance assessments, suggesting that required attendance may not be necessary to gain certain measurable core competencies.

USMLE and Otolaryngology: Predicting Board Performance

Objective To ascertain what relationship exists between the United States Medical Licensing Examination (USMLE) and the American Board of Otolaryngology Written Qualifying Examination (WQE). Study Design Retrospective, longitudinal study. Setting De-identified database. Subjects Residents entering otolaryngology-head and neck surgery (OHNS) residency between 2007 and 2009 and taking the WQE for the first time between 2012 and 2014. Methods Regression models were used to determine if the USMLE score predicts passage of the WQE on the first attempt, which step score was a better predictor, and whether an increase in the Step 2 score increased the chances of WQE passage. Results There were 611 Step 1 and 402 Step 2 scores. Mean (SD) Step 1 score was 238 (14.4), and the mean (SD) Step 2 score was 243 (16.8). The overall WQE first-time passage rate was 95.3%. Step 1 score was a better predictor than Step 2 for successful passage of the WQE on the first attempt ( P = .0026). An increase in the Step 2 score compared with the Step 1 score did not predict an improved first-time passage rate on the WQE. Conclusions There is an association between USMLE scores and successful first-time passage of the WQE, with Step 1 being a stronger predictor than Step 2. Residents with USMLE scores lower than the average successful OHNS applicant still have a high chance of passing the WQE. USMLE scores alone are of limited usefulness in identifying those applicants at risk of failing the WQE on the first attempt.

Military otolaryngology resident case numbers and board passing rates during the Afghanistan and Iraq wars

OBJECTIVE

During the wars in Iraq and Afghanistan, the US military has continued to train medical residents despite concern that postgraduate medical education at military training facilities has suffered. This study compares the experience of otolaryngology residents at military programs with the experience of their civilian counterparts.

STUDY DESIGN

Retrospective review.

SETTING

Academic military medical centers.

SUBJECTS AND METHODS

Resident caseload data and board examination passing rates were requested from each of the 6 Department of Defense otolaryngology residency programs for 2001 to 2010. The American Board of Otolaryngology and the Accreditation Council for Graduate Medical Education provided the national averages for resident caseload. National board passing rates from 2004 to 2010 were also obtained. Two-sample t tests were used to compare the pooled caseloads from the military programs with the national averages. Board passing rates were compared with a test of proportions.

RESULTS

Data were available for all but one military program. Regarding total cases, only 2001 and 2003 showed a significant difference (P < .05), with military residents completing more cases in those years. For individual case categories, the military averages were higher in Otology (299.6 vs 261.2, P = .033) and Plastics/Reconstruction (248.1 vs 149.2, P = .003). Only the Head & Neck category significantly favored the national average over the military (278.3 and 226.0, P = .039). The first-time board passing rates were identical between the groups (93%).

CONCLUSION

Our results suggest that the military otolaryngology residency programs are equal in terms of caseload and board passing rates compared with civilian programs over this time period.

A nomogram to predict the probability of passing the American Board of Internal Medicine examination

BACKGROUND

Although the American Board of Internal Medicine (ABIM) certification is valued as a reflection of physicians' experience, education, and expertise, limited methods exist to predict performance in the examination.

PURPOSE

The objective of this study was to develop and validate a predictive tool based on variables common to all residency programs, regarding the probability of an internal medicine graduate passing the ABIM certification examination.

METHODS

The development cohort was obtained from the files of the Cleveland Clinic internal medicine residents who began training between 2004 and 2008. A multivariable logistic regression model was built to predict the ABIM passing rate. The model was represented as a nomogram, which was internally validated with bootstrap resamples. The external validation was done retrospectively on a cohort of residents who graduated from two other independent internal medicine residency programs between 2007 and 2011.

RESULTS

Of the 194 Cleveland Clinic graduates used for the nomogram development, 175 (90.2%) successfully passed the ABIM certification examination. The final nomogram included four predictors: In-Training Examination (ITE) scores in postgraduate year (PGY) 1, 2, and 3, and the number of months of overnight calls in the last 6 months of residency. The nomogram achieved a concordance index (CI) of 0.98 after correcting for over-fitting bias and allowed for the determination of an estimated probability of passing the ABIM exam. Of the 126 graduates from two other residency programs used for external validation, 116 (92.1%) passed the ABIM examination. The nomogram CI in the external validation cohort was 0.94, suggesting outstanding discrimination.

CONCLUSIONS

A simple user-friendly predictive tool, based on readily available data, was developed to predict the probability of passing the ABIM exam for internal medicine residents. This may guide program directors' decision-making related to program curriculum and advice given to individual residents regarding board preparation.

A nomogram to predict the probability of passing the American Board of Internal Medicine examination

BACKGROUND

Although the American Board of Internal Medicine (ABIM) certification is valued as a reflection of physicians' experience, education, and expertise, limited methods exist to predict performance in the examination.

PURPOSE

The objective of this study was to develop and validate a predictive tool based on variables common to all residency programs, regarding the probability of an internal medicine graduate passing the ABIM certification examination.

METHODS

The development cohort was obtained from the files of the Cleveland Clinic internal medicine residents who began training between 2004 and 2008. A multivariable logistic regression model was built to predict the ABIM passing rate. The model was represented as a nomogram, which was internally validated with bootstrap resamples. The external validation was done retrospectively on a cohort of residents who graduated from two other independent internal medicine residency programs between 2007 and 2011.

RESULTS

Of the 194 Cleveland Clinic graduates used for the nomogram development, 175 (90.2%) successfully passed the ABIM certification examination. The final nomogram included four predictors: In-Training Examination (ITE) scores in postgraduate year (PGY) 1, 2, and 3, and the number of months of overnight calls in the last 6 months of residency. The nomogram achieved a concordance index (CI) of 0.98 after correcting for over-fitting bias and allowed for the determination of an estimated probability of passing the ABIM exam. Of the 126 graduates from two other residency programs used for external validation, 116 (92.1%) passed the ABIM examination. The nomogram CI in the external validation cohort was 0.94, suggesting outstanding discrimination.

CONCLUSIONS

A simple user-friendly predictive tool, based on readily available data, was developed to predict the probability of passing the ABIM exam for internal medicine residents. This may guide program directors' decision-making related to program curriculum and advice given to individual residents regarding board preparation.