The predictive validity of the internal medicine in-training examination

On the Predictive Power of the In-Training Examination for the Certification of Anesthesiologist Assistants

OBJECTIVES

In-training examinations (ITEs) are a popular teaching tool for certification programs. This study examines the relationship between examinees' performance on the National Commission for Certification of Anesthesiologist Assistants (NCCAA) ITE and the high-stakes NCCAA Certification Examination.

METHODS

We utilized a mixed-methods approach in our study. Before estimating the models for the predictive validity study, a series of interviews with program directors were conducted to discuss the role of the ITE in students' education. Multiple linear regression analysis was then used to assess the strength of the relationship between the ITE and Certification Examination scores, while considering the percentage of program examinees completed in their anesthesiologist assistant program between their ITE and Certification Examination attempts. Logistic regression analysis was used to estimate the probability of passing the Certification Examination as a function of ITE score.

RESULTS

Interviews with program directors confirmed that the ITE provided a valuable testing experience for students and highlighted the areas where students need to focus. Moreover, both the ITE score and the percentage of the program between exams were deemed statistically significant predictors for Certification Examination scores. The logistic regression model indicated that higher scores on the ITE implied a higher probability of passing the Certification Examination.

CONCLUSION

This research demonstrated the high predictive validity of the ITE examination scores in predicting the Certification Examination outcomes. Together with the proportion of the program covered between exams, the variables explain a significant amount of variability in Certification Examination scores. The ITE feedback helped students assess their preparedness and better focus their studies for the high-stakes certification examination for the profession.

Online Infectious Diseases Subspecialty Supplementary Curriculum for Medical Students and Residents: Moving Beyond "You Get What You Get"

OBJECTIVES

Students and residents rotating through infectious diseases (ID) electives are instructed primarily by participation in rounds and clinics, with teaching focused on diseases encountered. This "you get what you get" approach allows learners to apply knowledge directly to patient care, however, may miss topics encountered in standardized testing. This multisite study investigates the use of asynchronous web-based learning modules and its impact on student and resident knowledge.

METHODS

Students and residents rotating through an ID elective were assigned to their standard elective (old) or asked to complete asynchronous web-based learning modules in addition to the standard curriculum (new). Learners submitted pre- and post-tests and scores were tabulated. In the following academic year, learners at the host site were provided the learning modules and a post-elective survey.

RESULTS

Nine learners (100%) completed the pre-test and 5 (55.6%) completed the post-test in the standard (old) curriculum group, while 15 (100%) completed the pre-test and post-test in the new curriculum group. The mean percentage change in accuracy was 9% and 5.3% in the old and new curricula, respectively. Most (94%) survey respondents recommended continued use of the curriculum and expansion to other subspecialty electives.

CONCLUSIONS

Subspecialty electives have multiple purposes including assisting learners in exploring careers, providing a well-rounded medical experience, or preparing learners for content in standardized tests. Consistent curricula are not always provided with electives to supplement the clinical experience. Our web curriculum was well-received with perceived knowledge gain, though with very small pre-post-test groups a score improvement could not be determined. An asynchronous online curriculum for learners in ID was feasible and well-received among faculty, and learners felt their knowledge was enhanced. Content areas supplemented those encountered during the ID elective. While an improvement in post-test scores was not demonstrated, learners and faculty felt modules were beneficial.

Association Between Resident Race and Ethnicity and Clinical Performance Assessment Scores in Graduate Medical Education

PURPOSE

To assess the association between internal medicine (IM) residents' race/ethnicity and clinical performance assessments.

METHOD

The authors conducted a cross-sectional analysis of clinical performance assessment scores at 6 U.S. IM residency programs from 2016 to 2017. Residents underrepresented in medicine (URiM) were identified using self-reported race/ethnicity. Standardized scores were calculated for Accreditation Council for Graduate Medical Education core competencies. Cross-classified mixed-effects regression assessed the association between race/ethnicity and competency scores, adjusting for rotation time of year and setting; resident gender, postgraduate year, and IM In-Training Examination percentile rank; and faculty gender, rank, and specialty.

RESULTS

Data included 3,600 evaluations by 605 faculty of 703 residents, including 94 (13.4%) URiM residents. Resident race/ethnicity was associated with competency scores, with lower scores for URiM residents (difference in adjusted standardized scores between URiM and non-URiM residents, mean [standard error]) in medical knowledge (-0.123 [0.05], P = .021), systems-based practice (-0.179 [0.05], P = .005), practice-based learning and improvement (-0.112 [0.05], P = .032), professionalism (-0.116 [0.06], P = .036), and interpersonal and communication skills (-0.113 [0.06], P = .044). Translating this to a 1 to 5 scale in 0.5 increments, URiM resident ratings were 0.07 to 0.12 points lower than non-URiM resident ratings in these 5 competencies. The interaction with faculty gender was notable in professionalism (difference between URiM and non-URiM for men faculty -0.199 [0.06] vs women faculty -0.014 [0.07], P = .01) with men more than women faculty rating URiM residents lower than non-URiM residents. Using the 1 to 5 scale, men faculty rated URiM residents 0.13 points lower than non-URiM residents in professionalism.

CONCLUSIONS

Resident race/ethnicity was associated with assessment scores to the disadvantage of URiM residents. This may reflect bias in faculty assessment, effects of a noninclusive learning environment, or structural inequities in assessment.

Association of Gender and Race/Ethnicity with Internal Medicine In-Training Examination Performance in Graduate Medical Education

BACKGROUND

Disparities in objective assessments in graduate medical education such as the In-Training Examination (ITE) that disadvantage women and those self-identifying with race/ethnicities underrepresented in medicine (URiM) are of concern.

OBJECTIVE

Examine ITE trends longitudinally across post-graduate year (PGY) with gender and race/ethnicity.

DESIGN

Longitudinal analysis of resident ITE metrics at 7 internal medicine residency programs, 2014-2019. ITE trends across PGY of women and URiM residents compared to non-URiM men assessed via ANOVA. Those with ITE scores associated with less than 90% probability of passing the American Board of Internal Medicine certification exam (ABIM-CE) were identified and odds of being identified as at-risk between groups were assessed with chi square.

PARTICIPANTS

A total of 689 IM residents, including 330 women and URiM residents (48%).

MAIN MEASURES

ITE score KEY RESULTS: There was a significant difference in ITE score across PGY for women and URiM residents compared to non-URiM men (F_{(2, 1321)} 4.46, p=0.011). Adjusting for program, calendar year, and baseline ITE, women and URiM residents had smaller ITE score gains (adjusted mean change in score between PGY1 and PGY3 (se), non-URiM men 13.1 (0.25) vs women and URiM residents 11.4 (0.28), p<0.001). Women and URiM residents had greater odds of being at potential risk for not passing the ABIM-CE (OR 1.75, 95% CI 1.10 to 2.78) with greatest odds in PGY3 (OR 3.13, 95% CI 1.54 to 6.37).

CONCLUSION

Differences in ITE over training were associated with resident gender and race/ethnicity. Women and URiM residents had smaller ITE score gains across PGY translating into greater odds of potentially being seen as at-risk for not passing the ABIM-CE. Differences in ITE over training may reflect differences in experiences of women and URiM residents during training and may lead to further disparities.

Does an endocrinology subspecialty residency rotation enhance resident endocrine clinical knowledge?

BACKGROUND

Internal Medicine (IM) programs offer elective subspecialty rotations in which residents may enroll to supplement the experience and knowledge obtained during general inpatient and outpatient rotations. Objective evidence that these rotations provide enhanced subspecialty specific knowledge is lacking. The purpose of this study was to determine whether exposure to an endocrinology subspecialty rotation enhanced a resident's endocrinology-specific knowledge beyond that otherwise acquired during IM residency.

METHODS

Data were collected on internal medicine resident scores on the American College of Physicians Internal Medicine In-Training Examinations (IM-ITE) for calendar years 2012 through 2018 along with enrollment data as to whether residents had completed an endocrinology subspecialty rotation prior to sitting for a given IM-ITE. Three hundred and six internal medicine residents in the University of Minnesota Internal Medicine residency program with 664 scores total on the IM-ITE for calendar years 2012 through 2018. Percentage of correct answers on the overall and endocrine subspecialty content areas on the IM-ITE for each exam were determined and the association between prior exposure to an endocrinology subspecialty rotation and percentage of correct answers in the endocrinology content area was analyzed using generalized linear mixed-effects models.

RESULTS

Two hundred and thirty-three residents (76%) completed an endocrinology subspecialty rotation at some point during their residency; 121 (40%) residents had at least one IM-ITE both before and after exposure to an endocrine subspecialty rotation. Exposure to an endocrinology subspecialty rotation exhibited a positive association with the expected IM-ITE percent correct on the endocrinology content area (5.5% predicted absolute increase). Advancing year of residency was associated with a predicted increase in overall IM-ITE score but did not improve the predictive model for endocrine subspecialty score.

CONCLUSIONS

Completion of an endocrinology subspecialty elective was associated with an increase in resident endocrine specific knowledge as assessed by the IM-ITE. These findings support the value of subspecialty rotations in enhancing a resident's subspecialty specific medical knowledge.

Novel Formative Approach of the ESAP-ITE Provides Strong Predictive Value for ABIM Certification Outcomes

Background

The Endocrine Self-Assessment Program In-Training Examination (ESAP-ITE) has the novel formative approach of allowing open access to all questions and answers after secure examination administration is complete, resulting in the creation of an entirely new in-training examination annually.

Objective

To determine whether scores on the novel ESAP-ITE predict pass/fail outcomes on the American Board of Internal Medicine Endocrinology, Diabetes, and Metabolism Certification Examination (ABIM-ECE).

Methods

All endocrine fellows-in-training who took the ESAP-ITE between 2016 and 2019 and then subsequently attempted the ABIM-ECE within the same calendar year were included (n = 982). Primary analyses used the ESAP-ITE score from the final year of fellowship training. Covariates included sex, age on date of ABIM-ECE, medical school country, fellowship program region, pass/fail outcomes on the ABIM Internal Medicine Certification Examination, and ESAP-ITE score. All variables were analyzed using multivariable logistic regression.

Results

ESAP-ITE score (P < 0.001), ABIM Internal Medicine Certification Examination outcome (P < 0.001), and age (P = 0.005) were each significant predictors of passing the ABIM-ECE on the first attempt. ESAP-ITE score was the strongest predictor of passing the ABIM-ECE, and this relationship was such that a score of 75% correct yielded a 97% probability of passing the ABIM-ECE, whereas a score of 50% correct generated only a 70% probability of doing so. Sex, fellowship program region, and medical school country were not significant predictors of ABIM-ECE outcomes.

Conclusions

In addition to serving as an important learning instrument for endocrine fellowship programs, ESAP-ITE is a robust predictive tool for pass/fail outcomes on the ABIM-ECE.

Validation of the General Medicine in-Training Examination Using the Professional and Linguistic Assessments Board Examination Among Postgraduate Residents in Japan

Purpose

In Japan, the General Medicine In-training Examination (GM-ITE) was developed by a non-profit organization in 2012. The GM-ITE aimed to assess the general clinical knowledge among residents and to improve the training programs; however, it has not been sufficiently validated and is not used for high-stake decision-making. This study examined the association between GM-ITE and another test measure, the Professional and Linguistic Assessments Board (PLAB) 1 examination.

Methods

Ninety-seven residents who completed the GM-ITE in fiscal year 2019 were recruited and took the PLAB 1 examination in Japanese. The association between two tests was assessed using the Pearson product-moment statistics. The discrimination indexes were also assessed for each question.

Results

A total of 91 residents at 17 teaching hospitals were finally included in the analysis, of whom 69 (75.8%) were women and 59 (64.8%) were postgraduate second year residents. All the participants were affiliated with community hospitals. Positive correlations were demonstrated between the GM-ITE and the PLAB scores (r = 0.58, p < 0.001). The correlations between the PLAB score and the scores in GM-ITE categories were as follows: symptomatology/clinical reasoning (r = 0.54, p < 0.001), physical examination/procedure (r = 0.38, p < 0.001), medical interview/professionalism (r = 0.25, p < 0.001), and disease knowledge (r = 0.36, p < 0.001). The mean discrimination index of each question of the GM-ITE (mean ± SD; 0.23 ± 0.15) was higher than that of the PLAB (0.16 ± 0.16; p = 0.004).

Conclusion

This study demonstrates incremental validity evidence of the GM-ITE to assess the clinical knowledge acquisition. The results indicate that GM-ITE can be widely used to improve resident education in Japan.

Curriculum Transition From Lecture-Based to Team-Based Learning is Associated With Improved Performance on Internal Medicine In-Training Examination

BACKGROUND

Team-based learning (TBL) is an alternative to traditional lectures in graduate medical education, but evidence is scarce regarding its impact on knowledge acquisition and standardized testing performance.

OBJECTIVE

We examined the association between resident performance on the Internal Medicine In-Training Examination (IM-ITE) and these 2 educational methods.

METHODS

In 2013, the internal medicine residency program at Albany Medical College transitioned from a lecture-based curriculum to TBL. Residents enrolled in academic years 2011-2012 and 2012-2013 comprised the lecture cohort, and those enrolled in 2015-2016 and 2016-2017 the TBL cohort. Covariates included the type of medical school attended, gender, and United States Medical Licensing Examination Step 2 Clinical Knowledge scores. We performed univariate analysis and multivariable regression to determine the association between covariates and ITE scores.

RESULTS

Of 120 residents, 60 were in the lecture cohort and 60 in the TBL cohort. The IM-ITE percent correct scores were higher with TBL than lecture (PGY-1 61.0% vs 55.0%, P < .001; PGY-2 69.0% vs 59.7%, P < .001; PGY-3 73.2% vs 61.7%, P < .001). In a multivariable regression analysis of 3 PGYs combined, the transition from lecture to TBL resulted in an increase in IM-ITE Z-score of 0.415 (P < .001), equivalent to 0.415 SD, when including the effects of all covariates.

CONCLUSIONS

Compared to a lecture-based curriculum, TBL was associated with improved resident medical knowledge acquisition as evidenced by higher IM-ITE scores.

Do the selection criteria of internal medicine residency program predict resident performance?

BACKGROUND

Well-performing physician reflects the success of the residency program in selecting the best candidates for training. This study aimed to evaluate the selection criteria, mainly the United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) results and applicants' status as international or locally trained applicants, used by the medical education department and the internal medicine residency program in Hamad Medical Corporation in Qatar to predict the residents' performance during their training.

METHODS

A retrospective chart review was performed for three batches of graduates who started residency training in 2011, 2012, and 2013. Each group completed 4 years of training. The USMLE Step 2 CK status of the applicant, in-training exam (ITE) scores, formative evaluation scores, Arab Board written and clinical exams pass rate, and other indicators were analyzed. Statistical analysis included chi squares and independent t-test to identify associations. Multivariable analyses were conducted using logistic and linear regressions to test for adjusted associations.

RESULTS

The study included 118 (81 international/37 locally trained applicants) internal medicine residents. The ITE score correlated positively with the USMLE Step 2 CK score (r = 0.621, r = 0.587, r = 0.576, r = 0.571, p < 0.001) over the 4 years of training and among the international compared with locally trained applicants (p < 0.001). The rate of passing part 1 and 2 written exam of the Arab Board was higher in international than in local applicants, whereas clinical Arab Board exam and formative evaluation were not associated with any criteria.

CONCLUSIONS

Higher USMLE Step 2 CK score correlated with better performance on ITE but not with other performance indicators, whereas international applicants did better in both ITE and Arab Board written exam than local applicants. These variables may provide reasonable predictors of well-performing physicians.

A Systematic Review of the Relationship Between In-Training Examination Scores and Specialty Board Examination Scores

BACKGROUND

In-training examinations (ITEs) are intended for low-stakes, formative assessment of residents' knowledge, but are increasingly used for high-stake purposes, such as to predict board examination failures.

OBJECTIVE

The aim of this review was to investigate the relationship between performance on ITEs and board examination performance across medical specialties.

METHODS

A search of the literature for studies assessing the strength of the relationship between ITE and board examination performance from January 2000 to March 2019 was completed. Results were categorized based on the type of statistical analysis used to determine the relationship between ITE performance and board examination performance.

RESULTS

Of 1407 articles initially identified, 89 articles underwent full-text review, and 32 articles were included in this review. There was a moderate-strong relationship between ITE and board examination performance, and ITE scores significantly predict board examination scores for the majority of studies. Performing well on an ITE predicts a passing outcome for the board examination, but there is less evidence that performing poorly on an ITE will result in failing the associated specialty board examination.

CONCLUSIONS

There is a moderate to strong correlation between ITE performance and subsequent performance on board examinations. That the predictive value for passing the board examination is stronger than the predictive value for failing calls into question the "common wisdom" that ITE scores can be used to identify "at risk" residents. The graduate medical education community should continue to exercise caution and restraint in using ITE scores for moderate to high-stakes decisions.

Design and implementation of an academic enrichment program to improve performance on the internal medicine in-training exam

The internal medicine In-Training Exam (ITE) is administered at residency training programs to assess medical knowledge. Our internal medicine residency program witnessed a performance decline on the ITE between 2011 and 2014. The goal of this quality improvement project was to improve medical knowledge among residents as measured by an improvement in performance on the ITE, through the design and implementation of an Academic Enrichment Program (AEP). The AEP was designed in 2014-2015, and entailed a multipronged approach, including strengthening and tailoring of the didactic curriculum, establishment of a minimum conference attendance rate, and adoption of the New England Journal of Medicine Knowledge-Plus Internal Medicine Board Review platform. Residents performing below a pre-specified percentile rank cutoff on the previous year's ITE in any of the 12 content areas were required to complete a pre-specified percentage of the question bank in that specific topic. We examined a total of 164 residents enrolled in our program under the categorical training track. The mean (± SEM) ITE percentile for the 12 content areas increased significantly from calendar years 2011-2014 to 2015-2018, reflecting implementation of the AEP (p < 0.001). In brief, compared to the AEP-unexposed graduating classes of residents, the AEP-exposed graduating classes of residents displayed a significant improvement in the mean ITE percentile rank. This quality improvement project was carried out at a single institution. The implementation of a structured academic enrichment program significantly improves performance on the ITE.

Correlations Between the USMLE Step Examinations, American College of Physicians In-Training Examination, and ABIM Internal Medicine Certification Examination

PURPOSE

To assess the correlations between United States Medical Licensing Examination (USMLE) performance, American College of Physicians Internal Medicine In-Training Examination (IM-ITE) performance, American Board of Internal Medicine Internal Medicine Certification Exam (IM-CE) performance, and other medical knowledge and demographic variables.

METHOD

The study included 9,676 postgraduate year (PGY)-1, 11,424 PGY-2, and 10,239 PGY-3 internal medicine (IM) residents from any Accreditation Council for Graduate Medical Education-accredited IM residency program who took the IM-ITE (2014 or 2015) and the IM-CE (2015-2018). USMLE scores, IM-ITE percent correct scores, and IM-CE scores were analyzed using multiple linear regression, and IM-CE pass/fail status was analyzed using multiple logistic regression, controlling for USMLE Step 1, Step 2 Clinical Knowledge, and Step 3 scores; averaged medical knowledge milestones; age at IM-ITE; gender; and medical school location (United States or Canada vs international).

RESULTS

All variables were significant predictors of passing the IM-CE with IM-ITE scores having the strongest association and USMLE Step scores being the next strongest predictors. Prediction curves for the probability of passing the IM-CE based solely on IM-ITE score for each PGY show that residents must score higher on the IM-ITE with each subsequent administration to maintain the same estimated probability of passing the IM-CE.

CONCLUSIONS

The findings from this study should support residents and program directors in their efforts to more precisely identify and evaluate knowledge gaps for both personal learning and program improvement. While no individual USMLE Step score was as strongly predictive of IM-CE score as IM-ITE score, the combined relative contribution of all 3 USMLE Step scores was of a magnitude similar to that of IM-ITE score.

Characteristics of internal medicine residents who successfully match into cardiology fellowships

BACKGROUND

The unique traits of residents who matriculate into subspecialty fellowships are poorly understood. We sought to identify characteristics of internal medicine (IM) residents who match into cardiovascular (CV) fellowships.

METHODS

We conducted a retrospective cohort study of 8 classes of IM residents who matriculated into residency from 2007 to 2014. The primary outcome was successful match to a CV fellowship within 1 year of completing IM residency. Independent variables included residents' licensing exam scores, research publications, medical school reputation, Alpha Omega Alpha (AOA) membership, declaration of intent to pursue CV in the residency application personal statement, clinical evaluation scores, mini-clinical evaluation exercise scores, in-training examination (ITE) performance, and exposure to CV during residency.

RESULTS

Of the 339 included residents (59% male; mean age 27) from 120 medical schools, 73 (22%) matched to CV fellowship. At the time of residency application, 104 (31%) had ≥1 publication, 38 (11%) declared intention to pursue CV in their residency application personal statement, and 104 (31%) were members of AOA. Prior to fellowship application, 111 (33%) completed a CV elective rotation. At the completion of residency training, 108 (32%) had ≥3 publications. In an adjusted logistic regression analysis, declaration of intention to pursue CV (OR 6.4, 99% CI 1.7-23.4; p < 0.001), completion of a CV elective (OR 7.3, 99% CI 2.8-19.0; p < 0.001), score on the CV portion of the PGY-2 ITE (OR 1.05, 99% CI 1.02-1.08; p < 0.001), and publication of ≥3 manuscripts (OR 4.7, 99% CI 1.1-20.5; p = 0.007) were positively associated with matching to a CV fellowship. Overall PGY-2 ITE score was negatively associated (OR 0.93, 99% CI 0.90-0.97; p < 0.001) with matching to a CV fellowship.

CONCLUSIONS

Residents' matriculation into CV fellowships was associated with declaration of CV career intent, completion of a CV elective rotation, CV medical knowledge, and research publications during residency. These findings may be useful when advising residents about pursuing careers in CV. They may also help residents understand factors associated with a successful match to a CV fellowship. The negative association between matching into CV fellowship and overall ITE score may indicate excessive subspecialty focus during IM residency.

Association of Gender With Learner Assessment in Graduate Medical Education

IMPORTANCE

Gender bias may affect assessment in competency-based medical education.

OBJECTIVE

To evaluate the association of gender with assessment of internal medicine residents.

DESIGN, SETTING, AND PARTICIPANTS

This multisite, retrospective, cross-sectional study included 6 internal medicine residency programs in the United States. Data were collected from July 1, 2016, to June 30, 2017, and analyzed from June 7 to November 6, 2019.

EXPOSURES

Faculty assessments of resident performance during general medicine inpatient rotations.

MAIN OUTCOMES AND MEASURES

Standardized scores were calculated based on rating distributions for the Accreditation Council for Graduate Medical Education's core competencies and internal medicine Milestones at each site. Standardized scores are expressed as SDs from the mean. The interaction of gender and postgraduate year (PGY) with standardized scores was assessed, adjusting for site, time of year, resident In-Training Examination percentile rank, and faculty rank and specialty.

RESULTS

Data included 3600 evaluations for 703 residents (387 male [55.0%]) by 605 faculty (318 male [52.6%]). Interaction between resident gender and PGY was significant in 6 core competencies. In PGY2, female residents scored significantly higher than male residents in 4 of 6 competencies, including patient care (mean standardized score [SE], 0.10 [0.04] vs 0.22 [0.05]; P = .04), systems-based practice (mean standardized score [SE], -0.06 [0.05] vs 0.13 [0.05]; P = .003), professionalism (mean standardized score [SE], -0.04 [0.06] vs 0.21 [0.06]; P = .001), and interpersonal and communication skills (mean standardized score [SE], 0.06 [0.05] vs 0.32 [0.06]; P < .001). In PGY3, male residents scored significantly higher than female patients in 5 of 6 competencies, including patient care (mean standardized score [SE], 0.47 [0.05] vs 0.32 [0.05]; P = .03), medical knowledge (mean standardized score [SE], 0.47 [0.05] vs 0.24 [0.06]; P = .003), systems-based practice (mean standardized score [SE], 0.30 [0.05] vs 0.12 [0.06]; P = .02), practice-based learning (mean standardized score [SE], 0.39 [0.05] vs 0.16 [0.06]; P = .004), and professionalism (mean standardized score [SE], 0.35 [0.05] vs 0.18 [0.06]; P = .03). There was a significant increase in male residents' competency scores between PGY2 and PGY3 (range of difference in mean adjusted standardized scores between PGY2 and PGY3, 0.208-0.391; P ≤ .002) that was not seen in female residents' scores (range of difference in mean adjusted standardized scores between PGY2 and PGY3, -0.117 to 0.101; P ≥ .14). There was a significant increase in male residents' scores between PGY2 and PGY3 cohorts in 6 competencies with female faculty and in 4 competencies with male faculty. There was no significant change in female residents' competency scores between PGY2 to PGY3 cohorts with male or female faculty. Interaction between faculty-resident gender dyad and PGY was significant in the patient care competency (β estimate [SE] for female vs male dyad in PGY1 vs PGY3, 0.184 [0.158]; β estimate [SE] for female vs male dyad in PGY2 vs PGY3, 0.457 [0.181]; P = .04).

CONCLUSIONS AND RELEVANCE

In this study, resident gender was associated with differences in faculty assessments of resident performance, and differences were linked to PGY. In contrast to male residents' scores, female residents' scores displayed a peak-and-plateau pattern whereby assessment scores peaked in PGY2. Notably, the peak-and-plateau pattern was seen in assessments by male and female faculty. Further study of factors that influence gender-based differences in assessment is needed.

Do USMLE steps, and ITE score predict the American Board of Internal Medicine Certifying Exam results?

BACKGROUND

To evaluate if United States Medical Licensing Examination (USMLE) Step 1, USMLE Step 2 CK, USMLE Step 3, and residency third-year in-service training exam (ITE) scores predict the results of American Board of Internal Medicine Certifying Exam (ABIM-CE).

METHODS

We performed a retrospective review of USMLE Step 1, USMLE Step 2 CK, USMLE Step 3, third-year residency ITE scores and ABIM-CE results of IM residents at our residency program from 2004 through 2017. Statistical analysis was perfrormed using Pearson correlation coefficient, and logistic regression to assess the relationship between USMLE Step 1, USMLE Step 2CK, USMLE Step 3, 3rd year ITE scores and ABIM-CE results. We used Multivariate logistic regression to predict pass or fail results in ABIM-CE based on USMLE and third-year ITE test scores controlling for other covariates.

RESULTS

Among 114 Internal Medicine MD residents included in the study, 92% (n = 105) passed the ABIM-CE. The OR of passing ABIM-CE was 2.70 (95% CI = 1.38-5.29), 2.31 (95% CI = 1.33-4.01), and 1.63 (95% CI = 0.81-3.29) with a ten-point increase in USMLE Step 1, USMLE Step 2 CK and USMLE Step 3 scores respectively. The OR of ABIM-CE passing chance was 2.96 (95% CI = 0.95-9.20), with a ten-point increase in the average score of the above three exams. A 5 % increase in ITE percentage raised the likelihood of passing ABIM-CE (OR 2.92, 95% CI 1.15-7.38). All residents who failed ABIM-CE had Step 1 scores < 220. Among 31 residents with Step 2 CK score < 220, 20% (n = 6) failed ABIM. Similarly, 9% of residents with USMLE Step 3 score < 220 failed ABIM-CE; all residents who failed had scored < 220. The probability curve predicted that the chance of passing ABIM- CE was around 80% with USMLE scores greater than 200 and increased to almost 100% with USMLE scores of 250 or more.

CONCLUSION

USMLE Step 1, USMLE Step 2 CK, and third-year ITE scores can predict the chances of passing ABIM-CE. The third-year ITE score has a higher preditive value compared to USMLE Step 1 and USMLE Step 2 scores. USMLE Step 1 scores more predictive of ABIM-CE results compared to USMLE Step 2CK scores. Thus, residency programs can identify internal medicine residents at risk of failing ABIM-CE and formulate interventions at an early stage during residency training. Measures such as enrolling them in question banks or board review courses can be helpful in improving their chances of passing ABIM-CE.

Junior otolaryngology resident in-service exams predict written board exam passage

OBJECTIVE

The purpose of this study was to assess the association between the Otolaryngology Training Examination (OTE) taken during residency and the passage rate of first-time examinees on the American Board of Otolaryngology-Head and Neck Surgery Written Qualifying Examination (WQE).

METHODS

Retrospective cohort study using a de-identified database containing information on examinees who took the WQE in 2007 through 2014, and examinees who took the OTE exam 2005 through 2014. A total of 2,214 otolaryngology residents took the WQE for the first time in 2007 through 2014 after taking the OTE during residency training. Data were analyzed using one-way frequencies and table analyses. Logistic regression was used to model the relationship between the pass/fail WQE result and the OTE stanine. Data transformations were used to analyze WQE passage as a function of OTE scores.

RESULTS

There is a significant relationship between OTE score and passage of the WQE on the first attempt, evident even for junior residents. The probability of passing the WQE on the first attempt is 97% if the resident scores in the top six stanines of the OTE during residency training compared to 71% to 79% if the examinee scores in the bottom three stanines.

CONCLUSION

There is a significant relationship between performance on the OTE and passing the WQE on the first attempt. Because this relationship is evident even in the first years of residency, it allows early identification of those with a higher chance of failing the WQE so that extra efforts can be undertaken to prepare for the WQE.

LEVEL OF EVIDENCE

3 Laryngoscope, 129:124-128, 2019.

Performance on the Nephrology In-Training Examination and ABIM Nephrology Certification Examination Outcomes

BACKGROUND AND OBJECTIVES

Medical specialty and subspecialty fellowship programs administer subject-specific in-training examinations to provide feedback about level of medical knowledge to fellows preparing for subsequent board certification. This study evaluated the association between the American Society of Nephrology In-Training Examination and the American Board of Internal Medicine Nephrology Certification Examination in terms of scores and passing status.

DESIGN, SETTING, PARTICIPANTS, & MEASUREMENTS

The study included 1684 nephrology fellows who completed the American Society of Nephrology In-Training Examination in their second year of fellowship training between 2009 and 2014. Regression analysis examined the association between In-Training Examination and first-time Nephrology Certification Examination scores as well as passing status relative to other standardized assessments.

RESULTS

This cohort included primarily men (62%) and international medical school graduates (62%), and fellows had an average age of 32 years old at the time of first completing the Nephrology Certification Examination. An overwhelming majority (89%) passed the Nephrology Certification on their first attempt. In-Training Examination scores showed the strongest association with first-time Nephrology Certification Examination scores, accounting for approximately 50% of the total explained variance in the model. Each SD increase in In-Training Examination scores was associated with a difference of 30 U (95% confidence interval, 27 to 33) in certification performance. In-Training Examination scores also were significantly associated with passing status on the Nephrology Certification Examination on the first attempt (odds ratio, 3.46 per SD difference in the In-Training Examination; 95% confidence interval, 2.68 to 4.54). An In-Training Examination threshold of 375, approximately 1 SD below the mean, yielded a positive predictive value of 0.92 and a negative predictive value of 0.50.

CONCLUSIONS

American Society of Nephrology In-Training Examination performance is significantly associated with American Board of Internal Medicine Nephrology Certification Examination score and passing status.

Exploring examinee behaviours as validity evidence for multiple-choice question examinations

CONTEXT

Clinical-vignette multiple choice question (MCQ) examinations are used widely in medical education. Standardised MCQ examinations are used by licensure and certification bodies to award credentials that are meant to assure stakeholders as to the quality of physicians. Such uses are based on the interpretation of MCQ examination performance as giving meaningful information about the quality of clinical reasoning. There are several assumptions foundational to these interpretations and uses of standardised MCQ examinations. This study explores the implicit assumption that cognitive processes elicited by clinical-vignette MCQ items are like the processes thought to occur with 'real-world' clinical reasoning as theorised by dual-process theory.

METHODS

Fourteen participants (three medical students, five residents and six staff physicians) completed three sets of five timed MCQ items (total 15) from the Medical Knowledge Self-Assessment Program (MKSAP). Upon answering a set of MCQs, each participant completed a retrospective think aloud (TA) protocol. Using constant comparative analysis (CCA) methods sensitised by dual-process theory, we performed a qualitative thematic analysis.

RESULTS

Examinee behaviours fell into three categories: clinical reasoning behaviours, test-taking behaviours and reactions to the MCQ. Consistent with dual-process theory, statements about clinical reasoning behaviours were divided into two sub-categories: analytical reasoning and non-analytical reasoning. Each of these categories included several themes.

CONCLUSIONS

Our study provides some validity evidence that test-takers' descriptions of their cognitive processes during completion of high-quality clinical-vignette MCQs align with processes expected in real-world clinical reasoning. This supports one of the assumptions important for interpretations of MCQ examination scores as meaningful measures of clinical reasoning. Our observations also suggest that MCQs elicit other cognitive processes, including certain test-taking behaviours, that seem 'inauthentic' to real-world clinical reasoning. Further research is needed to explore if similar themes arise in other contexts (e.g. simulated patient encounters) and how observed behaviours relate to performance on MCQ-based assessments.

Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance

The Accreditation Council for Graduate Medical Education's Next Accreditation System requires training programs to demonstrate that fellows are achieving competence in medical knowledge (MK), as part of a global assessment of clinical competency. Passing American Board of Internal Medicine (ABIM) certification examinations is recognized as a metric of MK competency. This study examines several in-training MK assessment approaches and their ability to predict performance on the ABIM Hematology or Medical Oncology Certification Examinations. Results of a Hematology In-Service Examination (ISE) and an Oncology In-Training Examination (ITE), program director (PD) ratings, demographic variables, United States Medical Licensing Examination (USMLE), and ABIM Internal Medicine (IM) Certification Examination were compared. Stepwise multiple regression and logistic regression analyses evaluated these assessment approaches as predictors of performance on the Hematology or Medical Oncology Certification Examinations. Hematology ISE scores were the strongest predictor of Hematology Certification Examination scores (β = 0.41) (passing odds ratio [OR], 1.012; 95 % confidence interval [CI], 1.008-1.015), and the Oncology ITE scores were the strongest predictor of Medical Oncology Certification Examination scores (β = 0.45) (passing OR, 1.013; 95 % CI, 1.011-1.016). PD rating of MK was the weakest predictor of Medical Oncology Certification Examination scores (β = 0.07) and was not significantly predictive of Hematology Certification Examination scores. Hematology and Oncology ITEs are better predictors of certification examination performance than PD ratings of MK, reinforcing the effectiveness of ITEs for competency-based assessment of MK.

Pulmonary and Critical Care In-Service Training Examination Score as a Predictor of Board Certification Examination Performance

RATIONALE

Most trainees in combined pulmonary and critical care medicine fellowship programs complete in-service training examinations (ITEs) that test knowledge in both disciplines. Whether ITE scores predict performance on the American Board of Internal Medicine Pulmonary Disease Certification Examination and Critical Care Medicine Certification Examination is unknown.

OBJECTIVES

To determine whether pulmonary and critical care medicine ITE scores predict performance on subspecialty board certification examinations independently of trainee demographics, program director competency ratings, fellowship program characteristics, and prior medical knowledge assessments.

METHODS

First- and second-year fellows who were enrolled in the study between 2008 and 2012 completed a questionnaire encompassing demographics and fellowship training characteristics. These data and ITE scores were matched to fellows' subsequent scores on subspecialty certification examinations, program director ratings, and previous scores on their American Board of Internal Medicine Internal Medicine Certification Examination. Multiple linear regression and logistic regression were used to identify independent predictors of subspecialty certification examination scores and likelihood of passing the examinations, respectively.

MEASUREMENTS AND MAIN RESULTS

Of eligible fellows, 82.4% enrolled in the study. The ITE score for second-year fellows was matched to their certification examination scores, which yielded 1,484 physicians for pulmonary disease and 1,331 for critical care medicine. Second-year fellows' ITE scores (β = 0.24, P < 0.001) and Internal Medicine Certification Examination scores (β = 0.49, P < 0.001) were the strongest predictors of Pulmonary Disease Certification Examination scores, and were the only significant predictors of passing the examination (ITE odds ratio, 1.12 [95% confidence interval, 1.07-1.16]; Internal Medicine Certification Examination odds ratio, 1.01 [95% confidence interval, 1.01-1.02]). Similar results were obtained for predicting Critical Care Medicine Certification Examination scores and for passing the examination. The predictive value of ITE scores among first-year fellows on the subspecialty certification examinations was comparable to second-year fellows' ITE scores.

CONCLUSIONS

The Pulmonary and Critical Care Medicine ITE score is an independent, and stronger, predictor of subspecialty certification examination performance than fellow demographics, program director competency ratings, and fellowship characteristics. These findings support the use of the ITE to identify the learning needs of fellows as they work toward subspecialty board certification.

The Predictive Validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA Examinations With Regard to Outcomes on American Board of Family Medicine Examinations

PURPOSE

To examine the predictive validity of the National Board of Osteopathic Medical Examiners' Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series with regard to the American Board of Family Medicine's (ABFM's) In-Training Examination (ITE) and Maintenance of Certification for Family Physicians (MC-FP) Examination.

METHOD

A repeated-measures design was employed, using test scores across seven levels of training for 1,023 DOs who took the MC-FP for the first time between April 2012 and November 2014 and for whom the ABFM had ITE scores for each of their residency years. Pearson and disattenuated correlations were calculated; Fisher r to z transformation was performed; and sensitivity, specificity, and positive and negative predictive values for the COMLEX-USA Level 2-Cognitive Evaluation (CE) with regard to the MC-FP were computed.

RESULTS

The Pearson and disattenuated correlations ranged from 0.55 to 0.69 and from 0.61 to 0.80, respectively. For MC-FP scores, only the correlation increase from the COMLEX-USA Level 2-CE to Level 3 was statistically significant (for Pearson correlations: z = 2.41, P = .008; for disattenuated correlations: z = 3.16, P < .001). The sensitivity, specificity, and positive and negative predictive values of the COMLEX-USA Level 2-CE with the MC-FP were 0.90, 0.39, 0.96, and 0.19, respectively.

CONCLUSIONS

Evidence was found that the COMLEX-USA can assist family medicine residency program directors in predicting later resident performance on the ABFM's ITE and MC-FP, which is becoming increasingly important as graduate medical education accreditation moves toward a single aligned model.

Does the Physical Medicine and Rehabilitation Self-Assessment Examination for Residents Predict the Chances of Passing the Part 1 Board Certification Examination?

BACKGROUND

Each year, residents in accredited United States Physical Medicine and Rehabilitation (PMR) residency programs can take the American Academy of Physical Medicine and Rehabilitation (AAPM&R) Self-Assessment Examination for Residents (SAE-R). This 150-question, multiple-choice examination is intended for self-assessment of physiatric knowledge, but its predictive value for performance on the part 1 American Board of Physical Medicine and Rehabilitation Certification Examination (ABPMR-CE) is unknown.

OBJECTIVE

To investigate the predictive value of the SAE-R in relation to the part 1 ABPMR-CE.

DESIGN

Retrospective study.

METHODS

Data were analyzed from first time takers of the part 1 ABPMR-CE during a 5-year period from 2010 through 2014 who took at least 1 SAE-R in the third or fourth postgraduate year (PGY) of residency.

MAIN OUTCOME MEASUREMENTS

Raw scores from the SAE-R were compared with scaled scores on the part 1 examination. Regression models analyzed the predictive value of the SAE-R total score for each PGY level.

RESULTS

SAE-R raw scores increased an average of 5.5 points between the PGY 3 and PGY 4 year. PGY3 SAE-R raw scores accounted for 24.8% and PGY4 SAE-R scores for 27.1% of the variance in part 1 ABPMR-CE scores (P < .0001). Residents who obtained a raw score greater than 80 (53% correct) on the SAE-R had an 80% or greater chance of passing the ABPMR-CE. Scores greater than 90 (60% correct) on the SAE-R were associated with a 95% chance of passing the ABPMR-CE.

CONCLUSION

The SAE-R scores provide some information regarding the likelihood of passing the part 1 certification examination. This study supports the SAE-R as a means of providing PMR residents with feedback regarding their level of knowledge.

Association of Characteristics, Deficits, and Outcomes of Residents Placed on Probation at One Institution, 2002-2012

PURPOSE

To describe the population of residents placed on probation, identify learner characteristics associated with being placed on probation, and describe immediate and long-term career outcomes for those placed on probation as compared with matched controls.

METHOD

The authors collected data for residents at the University of Colorado School of Medicine placed on probation from July 2002 to June 2012, including postgraduate year placed on probation, deficits identified, mandated evaluation for physical and mental health, duration of probation, disability accommodations requested, and number of additional training months required. They were retrospectively compared with 102 controls matched for specialty, matriculation, and postgraduate year. Variables assessed included demographics, academic performance, license status, specialty, state board certification, and board citations.

RESULTS

Of 3,091 residents, 3.3% were placed on probation (88 residents; 14 fellows). Compared with controls, those on probation were more likely to be international medical graduates, married, not Caucasian, older (all P < .001), male (P = .01), to have transferred from another graduate medical education training program, and to have taken time off between medical school and residency (all P < .001). Among those currently in practice, 53 (63.9%) were board certified compared with 93 (100%) of the controls. Placement on probation was associated with failure to graduate and lack of board certification. All 7 graduates cited by state medical boards were in the probation group.

CONCLUSIONS

Further research is needed to understand these associations and to determine whether changes in curricula or remediation programs may alter these outcomes.

The transition to competency-based pediatric training in the United Arab Emirates

Although competency-based medical education has become the standard for physician training in the West, many developing countries have not yet adopted competency-based training. In 2009 in the United Arab Emirates, the government regulatory and operational authorities for healthcare in Abu Dhabi mandated a wide-scale reform of the emirate's postgraduate residency programs to the competency-based framework of the newly formed Accreditation Council for Graduate Medical Education-International (ACGME-I). This article briefly describes the rationale for competency-based medical education and provides an overview of the transition from traditional, time-based residency training to competency-based postgraduate medical education for the Pediatrics residency programs in Abu Dhabi. We will provide data on the initial impact of this transition on resident performance and patient outcomes in a Pediatrics residency program in an academic medical center in the United Arab Emirates.

Concurrent Validity Between a Shared Curriculum, the Internal Medicine In-Training Examination, and the American Board of Internal Medicine Certifying Examination

BACKGROUND

A core objective of residency education is to facilitate learning, and programs need more curricula and assessment tools with demonstrated validity evidence.

OBJECTIVE

We sought to demonstrate concurrent validity between performance on a widely shared, ambulatory curriculum (the Johns Hopkins Internal Medicine Curriculum), the Internal Medicine In-Training Examination (IM-ITE), and the American Board of Internal Medicine Certifying Examination (ABIM-CE).

METHODS

A cohort study of 443 postgraduate year (PGY)-3 residents at 22 academic and community hospital internal medicine residency programs using the curriculum through the Johns Hopkins Internet Learning Center (ILC). Total and percentile rank scores on ILC didactic modules were compared with total and percentile rank scores on the IM-ITE and total scores on the ABIM-CE.

RESULTS

The average score on didactic modules was 80.1%; the percentile rank was 53.8. The average IM-ITE score was 64.1% with a percentile rank of 54.8. The average score on the ABIM-CE was 464. Scores on the didactic modules, IM-ITE, and ABIM-CE correlated with each other (P < .05). Residents completing greater numbers of didactic modules, regardless of scores, had higher IM-ITE total and percentile rank scores (P < .05). Resident performance on modules covering back pain, hypertension, preoperative evaluation, and upper respiratory tract infection was associated with IM-ITE percentile rank.

CONCLUSIONS

Performance on a widely shared ambulatory curriculum is associated with performance on the IM-ITE and the ABIM-CE.

The relationship between performance on the Infectious Diseases In-Training and Certification Examinations

BACKGROUND

The Infectious Diseases Society of America In-Training Examination (IDSA ITE) is a feedback tool used to help fellows track their knowledge acquisition during fellowship training. We determined whether the scores on the IDSA ITE and from other major medical knowledge assessments predict performance on the American Board of Internal Medicine (ABIM) Infectious Disease Certification Examination.

METHODS

The sample was 1021 second-year fellows who took the IDSA ITE and ABIM Infectious Disease Certification Examination from 2008 to 2012. Multiple regression analysis was used to determine if ABIM Infectious Disease Certification Examination scores were predicted by IDSA ITE scores, prior United States Medical Licensing Examination (USMLE) scores, ABIM Internal Medicine Certification Examination scores, fellowship director ratings of medical knowledge, and demographic variables. Logistic regression was used to evaluate if these same assessments predicted a passing outcome on the certification examination.

RESULTS

IDSA ITE scores were the strongest predictor of ABIM Infectious Disease Certification Examination scores (β = .319), followed by prior ABIM Internal Medicine Certification Examination scores (β = .258), USMLE Step 1 scores (β = .202), USMLE Step 3 scores (β = .130), and fellowship directors' medical knowledge ratings (β = .063). IDSA ITE scores were also a significant predictor of passing the Infectious Disease Certification Examination (odds ratio, 1.017 [95% confidence interval, 1.013-1.021]).

CONCLUSIONS

The significant relationship between the IDSA ITE score and performance on the ABIM Infectious Disease Certification Examination supports the use of the ITE as a valid feedback tool in fellowship training.

Assessing achievement in nephrology training: using clinic chart audits to quantitatively screen competency

BACKGROUND

Entrustable professional activities (EPAs) are complex tasks representing vital physician functions in multiple competencies, used to demonstrate trainee development along milestones. Managing a nephrology outpatient clinic has been proposed as an EPA for nephrology fellowship training.

STUDY DESIGN

Retrospective cohort study of nephrology fellow outpatient clinic performance using a previously validated chart audit tool.

SETTING & PARTICIPANTS

Outpatient encounter chart audits for training years 2008-2009 through 2012-2013, corresponding to participation in the Nephrology In-Training Examination (ITE). A median of 7 auditors (attending nephrologists) audited a mean of 1,686±408 (SD) charts per year. 18 fellows were audited; 12, in both of their training years.

PREDICTORS

Proportion of chart audit and quality indicator deficiencies.

OUTCOMES

Longitudinal deficiency and ITE performance.

MEASUREMENTS & RESULTS

Among fellows audited in both their training years, chart audit deficiencies were fewer in the second versus the first year (5.4%±2.0% vs 17.3%±7.0%; P<0.001) and declined between the first and second halves of the first year (22.2%±6.4% vs 12.3%±9.5%; P=0.002). Most deficiencies were omission errors, regardless of training year. Quality indicator deficiencies for hypertension and chronic kidney disease-associated anemia recognition and management were fewer during the second year (P<0.001). Yearly audit deficiencies ≥5% were associated with an ITE score less than the 25th percentile for second-year fellows (P=0.03), with no significant association for first-year fellows. Auditor-reported deficiencies declined between the first and second halves of the year (17.0% vs 11.1%; P<0.001), with a stable positive/neutral comment rate (17.3% vs 17.8%; P=0.6), suggesting that the decline was not due to auditor fatigue.

LIMITATIONS

Retrospective design and small trainee numbers.

CONCLUSIONS

Managing a nephrology outpatient clinic is an EPA. The chart audit tool was used to assess longitudinal fellow performance in managing a nephrology outpatient clinic. Failure to progress may be quantitatively identified and remediated. The tool identifies deficiencies in all 6 competencies, not just medical knowledge, the primary focus of the ITE and the nephrology subspecialty board examination.

Recent trends in internal medicine education: a brief update

This perspective attempts to bring graduate medical offices, residency programs and medical students interested in categorical internal medicine (CIM) a brief update on the American Board of Internal Medicine (ABIM), Accreditation Council for Graduate Medical Education (ACGME) and the National Registry for Medical Programs (NRMP) changes for the past 3-5 years in the United States (US). The US model for certification and recertification may serve as a homogenous example for other countries. This model will be described so that there is an understanding of the importance of such changes in the American system and its effect on resident education. This is critical knowledge for both teachers and learners in internal medicine in preparation for a lifetime career and requirements for certification/credentialing for both programs and their residents/fellows. Data from the review indicate a small increase in the number of applicants but a concordant decrease in ABIM initial certification exams. Programs should well be aware of the new focus on outcomes via the Next Accreditation System (NAS) being put forth by the ACGME.

Relationship of electronic medical knowledge resource use and practice characteristics with Internal Medicine Maintenance of Certification Examination scores

BACKGROUND

Maintenance of certification examination performance is associated with quality of care. We aimed to examine relationships between electronic medical knowledge resource use, practice characteristics and examination scores among physicians recertifying in internal medicine.

METHODS

We conducted a cross-sectional study of 3,958 United States physicians who took the Internal Medicine Maintenance of Certification Examination (IM-MOCE) between January 1, 2006 and December 31, 2008, and who held individual licenses to one or both of two large electronic knowledge resource programs. We examined associations between physicians' IM-MOCE scores and their days of electronic resource use, practice type (private practice, residency teaching clinic, inpatient, nursing home), practice model (single or multi-specialty), sex, age, and medical school location.

RESULTS

In the 365 days prior to the IM-MOCE, physicians used electronic resources on a mean (SD, range) of 20.3 (36.5, 0-265) days. In multivariate analyses, the number of days of resource use was independently associated with increased IM-MOCE scores (0.07-point increase per day of use, p = 0.02). Increased age was associated with decreased IM-MOCE scores (1.8-point decrease per year of age, p < 0.001). Relative to physicians working in private practice settings, physicians working in residency teaching clinics and hospital inpatient practices had higher IM-MOCE scores by 29.1 and 20.0 points, respectively (both p < 0.001).

CONCLUSIONS

Frequent use of electronic resources was associated with modestly enhanced IM-MOCE performance. Physicians involved in residency education clinics and hospital inpatient practices had higher IM-MOCE scores than physicians working in private practice settings.

Predictive value of the korean academy of family medicine in-training examination for certifying examination

Background

In-training examination (ITE) is a cognitive examination similar to the written test, but it is different from the Clinical Practice Examination of the Korean Academy of Family Medicine (KAFM) Certification Examination (CE). The objective of this is to estimate the positive predictive value of the KAFM-ITE for identifying residents at risk for poor performance on the three types of KAFM-CE.

Methods

372 residents who completed the KAFM-CE in 2011 were included. We compared the mean KAFM-CE scores with ITE experience. We evaluated the correlation and the positive predictive value (PPV) of ITE for the multiple choice question (MCQ) scores of 1st written test & 2nd slide examination, the total clinical practice examination scores, and the total sum of 2nd test.

Results

275 out of 372 residents completed ITE. Those who completed ITE had significantly higher MCQ scores of 1st written test than those who did not. The correlation of ITE scores with 1st written MCQ (0.627) was found to be the highest among the other kinds of CE. The PPV of the ITE score for 1st written MCQ scores was 0.672. The PPV of the ITE score ranged from 0.376 to 0.502.

Conclusion

The score of the KAFM ITE has acceptable positive predictive value that could be used as a part of comprehensive evaluation system for residents in cognitive field.

Otolaryngology resident in-service examination scores predict passage of the written board examination

OBJECTIVE

The purpose of this study was to assess the association between the Otolaryngology Training Examination (OTE) and the passage rate of first-time examinees on the American Board of Otolaryngology (ABOto) Written Qualifying Examination (WQE).

STUDY DESIGN

Historical cohort study.

SETTING

De-identified database containing information on examinees who took the WQE in 2007 to 2011 and the OTE in 2005 to 2011.

SUBJECTS AND METHODS

One thousand three hundred nine otolaryngology residents took the WQE for the first time in 2007 to 2011 and the OTE in their final and penultimate years of training in 2005 to 2011. Data were analyzed using 1-way frequencies and table analysis. Logistic regression was used to model the relationship between the pass/fail WQE result and the OTE percentile score in the final and penultimate years of training. Data transformation was used to analyze WQE passage as a function of OTE scores by deciles and quartiles.

RESULTS

There is a significant relationship between OTE score and passage of the WQE on the first attempt. The probability of passing the WQE on the first attempt is 97% if the resident scores in the upper 3 quartiles of the OTE in the final and penultimate years of training compared with 70% if the examinee scores in the bottom quartile both years.

CONCLUSION

Residents who score in the bottom quartile in both their final and penultimate years of training are at significantly higher risk of failing the WQE on the first attempt. Remediation efforts should be focused on these individuals.

A multiple choice testing program coupled with a year-long elective experience is associated with improved performance on the internal medicine in-training examination

BACKGROUND

The Internal Medicine In-Training Exam (IM-ITE) assesses the content knowledge of internal medicine trainees. Many programs use the IM-ITE to counsel residents, to create individual remediation plans, and to make fundamental programmatic and curricular modifications.

OBJECTIVE

To assess the association between a multiple-choice testing program administered during 12 consecutive months of ambulatory and inpatient elective experience and IM-ITE percentile scores in third post-graduate year (PGY-3) categorical residents.

DESIGN

Retrospective cohort study.

PARTICIPANTS

One hundred and four categorical internal medicine residents. Forty-five residents in the 2008 and 2009 classes participated in the study group, and the 59 residents in the three classes that preceded the use of the testing program, 2005-2007, served as controls.

INTERVENTION

A comprehensive, elective rotation specific, multiple-choice testing program and a separate board review program, both administered during a continuous long-block elective experience during the twelve months between the second post-graduate year (PGY-2) and PGY-3 in-training examinations.

MEASURES

We analyzed the change in median individual percent correct and percentile scores between the PGY-1 and PGY-2 IM-ITE and between the PGY-2 and PGY-3 IM-ITE in both control and study cohorts. For our main outcome measure, we compared the change in median individual percentile rank between the control and study cohorts between the PGY-2 and the PGY-3 IM-ITE testing opportunities.

RESULTS

After experiencing the educational intervention, the study group demonstrated a significant increase in median individual IM-ITE percentile score between PGY-2 and PGY-3 examinations of 8.5 percentile points (p < 0.01). This is significantly better than the increase of 1.0 percentile point seen in the control group between its PGY-2 and PGY-3 examination (p < 0.01).

CONCLUSION

A comprehensive multiple-choice testing program aimed at PGY-2 residents during a 12-month continuous long-block elective experience is associated with improved PGY-3 IM-ITE performance.

Associations between quality indicators of internal medicine residency training programs

BACKGROUND

Several residency program characteristics have been suggested as measures of program quality, but associations between these measures are unknown. We set out to determine associations between these potential measures of program quality.

METHODS

Survey of internal medicine residency programs that shared an online ambulatory curriculum on hospital type, faculty size, number of trainees, proportion of international medical graduate (IMG) trainees, Internal Medicine In-Training Examination (IM-ITE) scores, three-year American Board of Internal Medicine Certifying Examination (ABIM-CE) first-try pass rates, Residency Review Committee-Internal Medicine (RRC-IM) certification length, program director clinical duties, and use of pharmaceutical funding to support education. Associations assessed using Chi-square, Spearman rank correlation, univariate and multivariable linear regression.

RESULTS

Fifty one of 67 programs responded (response rate 76.1%), including 29 (56.9%) community teaching and 17 (33.3%) university hospitals, with a mean of 68 trainees and 101 faculty. Forty four percent of trainees were IMGs. The average post-graduate year (PGY)-2 IM-ITE raw score was 63.1, which was 66.8 for PGY3s. Average 3-year ABIM-CE pass rate was 95.8%; average RRC-IM certification was 4.3 years. ABIM-CE results, IM-ITE results, and length of RRC-IM certification were strongly associated with each other (p<0.05). PGY3 IM-ITE scores were higher in programs with more IMGs and in programs that accepted pharmaceutical support (p<0.05). RRC-IM certification was shorter in programs with higher numbers of IMGs. In multivariable analysis, a higher proportion of IMGs was associated with 1.17 years shorter RRC accreditation.

CONCLUSIONS

Associations between quality indicators are complex, but suggest that the presence of IMGs is associated with better performance on standardized tests but decreased duration of RRC-IM certification.

The ASN in-training examination and the ABIM certifying examination: time for a new testing paradigm

In 2009, a new computer-based in-training examination (ITE) was created by the American Society of Nephrology (ASN) in conjunction with the National Board of Medical Examiners to assess the medical knowledge of fellows training in nephrology. The ITE, patterned after the American Board of Internal Medicine certifying examination for nephrology (ABIM-CE), allowed fellows and training program directors (TPDs) to compare their performance with programs in the nation. However, despite using case vignettes to simulate real-life clinical experiences, the nation's graduating fellows answered a mean of only 69% of the questions correctly. Although unpublicized, the ABIM-CE passing grade can be estimated to be approximately 65%, resulting in the current passing rate of 94%. The ABIM states that they are testing 85% synthesis and clinical judgment, rather than recall memory. So are we to conclude from these two exams that the medical knowledge of our graduating nephrology fellows allows them to properly manage only about two-thirds of their patients? Rather, the construction of the test questions and their relevance to clinical practice should be reassessed. The time has come for a new testing paradigm, one that would require candidates passing the ABIM-CE to score 85% to 90% correct answers to more relevant "core" examination questions that assess the actual medical knowledge needed to practice competently in our subspecialty. If the ABIM-CE was revised, the ASN ITE, with no specific passing grade, could then test core and "factual" recall knowledge with feedback of questions to TPDs to assess and help fellows in training.

The in-training examination: an analysis of its predictive value on performance on the general pediatrics certification examination

OBJECTIVE

This study investigates the predictive validity of the In-Training Examination (ITE). Although studies have confirmed the predictive validity of ITEs in other medical specialties, no study has been done for general pediatrics.

STUDY DESIGN

Each year, residents in accredited pediatric training programs take the ITE as a self-assessment instrument. The ITE is similar to the American Board of Pediatrics General Pediatrics Certifying Examination. First-time takers of the certifying examination over a 5-year period who took at least 1 ITE examination were included in the sample. Regression models analyzed the predictive value of the ITE.

RESULTS

The predictive power of the ITE in the first training year is minimal. However, the predictive power of the ITE increases each year, providing the greatest power in the third year of training.

CONCLUSIONS

Even though ITE scores provide information regarding the likelihood of passing the certification examination, the data should be used with caution, particularly in the first training year. Other factors also must be considered when predicting performance on the certification examination. This study continues to support the ITE as an assessment tool for program directors, as well as a means of providing residents with feedback regarding their acquisition of pediatric knowledge.

Associations between United States Medical Licensing Examination (USMLE) and Internal Medicine In-Training Examination (IM-ITE) scores

BACKGROUND

Little is known about the associations of previous standardized examination scores with scores on subsequent standardized examinations used to assess medical knowledge in internal medicine residencies.

OBJECTIVE

To examine associations of previous standardized test scores on subsequent standardized test scores.

DESIGN

Retrospective cohort study.

PARTICIPANTS

One hundred ninety-five internal medicine residents.

METHODS

Bivariate associations of United States Medical Licensing Examination (USMLE) Steps and Internal Medicine In-Training Examination (IM-ITE) scores were determined. Random effects analysis adjusting for repeated administrations of the IM-ITE and other variables known or hypothesized to affect IM-ITE score allowed for discrimination of associations of individual USMLE Step scores on IM-ITE scores.

RESULTS

In bivariate associations, USMLE scores explained 17% to 27% of the variance in IME-ITE scores, and previous IM-ITE scores explained 66% of the variance in subsequent IM-ITE scores. Regression coefficients (95% CI) for adjusted associations of each USMLE Step with IM-ITE scores were USMLE-1 0.19 (0.12, 0.27), USMLE-2 0.23 (0.17, 0.30), and USMLE-3 0.19 (0.09, 0.29).

CONCLUSIONS

No single USMLE Step is more strongly associated with IM-ITE scores than the others. Because previous IM-ITE scores are strongly associated with subsequent IM-ITE scores, appropriate modeling, such as random effects methods, should be used to account for previous IM-ITE administrations in studies for which IM-ITE score is an outcome.