Correlation between radiology resident rotation performance and examination scores

Do Residency Selection Factors Predict Radiology Resident Performance?

RATIONALE AND OBJECTIVES

The purpose of our study is to determine what information in medical student residency applications predicts radiology residency success as defined by objective clinical performance data.

MATERIALS AND METHODS

We performed a retrospective cohort study of residents who entered our institution's residency program through the National Resident Matching Program as postgraduate year 2 residents and completed the program over the past 2 years. Medical school grades, selection to Alpha Omega Alpha (AOA) Honor Society, United States Medical Licensing Examination (USMLE) scores, publication in peer-reviewed journals, and whether the applicant was from a peer institution were the variables examined. Clinical performance was determined by calculating each resident's cumulative major discordance rate for on-call cases the resident read and gave a preliminary interpretation. A major discordance was defined as a difference between the preliminary resident and the final attending interpretations that could immediately impact the care of the patient. A multivariate logistic regression was performed to determine significant variables.

RESULTS

Twenty-seven residents provided preliminary reports on call for 67,145 studies. The mean major discordance rate was 1.08% (range 0.34%-2.54%). Higher USMLE Step 1 scores, publication before residency, and election to AOA Honor Society were all statistically significant predictors of lower major discordance rates (P values 0.01, 0.01,  and <0.001, respectively).

CONCLUSIONS

Overall resident performance was excellent. There are predictors that help select the better performing residents, namely higher USMLE Step 1 scores, one to two publications during medical school, and election to AOA in the junior year of medical school.

Does Residency Selection Criteria Predict Performance in Orthopaedic Surgery Residency?

BACKGROUND

More than 1000 candidates applied for orthopaedic residency positions in 2014, and the competition is intense; approximately one-third of the candidates failed to secure a position in the match. However, the criteria used in the selection process often are subjective and studies have differed in terms of which criteria predict either objective measures or subjective ratings of resident performance by faculty.

QUESTIONS/PURPOSES

Do preresidency selection factors serve as predictors of success in residency? Specifically, we asked which preresidency selection factors are associated or correlated with (1) objective measures of resident knowledge and performance; and (2) subjective ratings by faculty.

METHODS

Charts of 60 orthopaedic residents from our institution were reviewed. Preresidency selection criteria examined included United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, Medical College Admission Test (MCAT) scores, number of clinical clerkship honors, number of letters of recommendation, number of away rotations, Alpha Omega Alpha (AOA) honor medical society membership, fourth-year subinternship at our institution, and number of publications. Resident performance was assessed using objective measures including American Board of Orthopaedic Surgery (ABOS) Part I scores and Orthopaedics In-Training Exam (OITE) scores and subjective ratings by faculty including global evaluation scores and faculty rankings of residents. We tested associations between preresidency criteria and the subsequent objective and subjective metrics using linear correlation analysis and Mann-Whitney tests when appropriate.

RESULTS

Objective measures of resident performance namely, ABOS Part I scores, had a moderate linear correlation with the USMLE Step 2 scores (r = 0.55, p < 0.001) and number of clinical honors received in medical school (r = 0.45, p < 0.001). OITE scores had a weak linear correlation with the number of clinical honors (r = 0.35, p = 0.009) and USMLE Step 2 scores (r = 0.29, p = 0.02). With regards to subjective outcomes, AOA membership was associated with higher scores on the global evaluation (p = 0.005). AOA membership also correlated with higher global evaluation scores (r = 0.60, p = 0.005) with the strongest correlation existing between AOA membership and the "interpersonal and communication skills" subsection of the global evaluations.

CONCLUSIONS

We found that USMLE Step 2, number of honors in medical school clerkships, and AOA membership demonstrated the strongest correlations with resident performance. Our goal in analyzing these data was to provide residency programs at large a sense of which criteria may be "high yield" in ranking applicants by analyzing data from within our own pool of residents. Similar studies across a broader scope of programs are warranted to confirm applicability of our findings. The continually emerging complexities of the field of orthopaedic surgery lend increasing importance to future work on the appropriate selection and training of orthopaedic residents.

Does Delay in Taking the American Board of Physical Medicine and Rehabilitation Certification Examinations Affect Passing Rates?

OBJECTIVES

The objective of this study was to examine whether a delay in initially taking the Part I or Part II American Board of Physical Medicine and Rehabilitation certification examinations influences the score or passing rates of candidates and whether enrollment in a subspecialty fellowship influences performance on the Part II certification examination.

DESIGN

This was a retrospective review of first-time candidates taking the American Board of Physical Medicine and Rehabilitation initial certification examinations from 2010 to 2014.

RESULTS

Passing rates declined as candidates delayed the examination past the time of initial eligibility. Passing rates for Part I were as follows (mean [95% confidence interval]): no delay, 91% (89%-94%); 1-yr delay, 68% (56%-82%); and more that 2 yrs' delay, 59% (55%-68%). For Part II, passing rates were as follows: no delay, 90% (85%-94%); 1-yr delay, 83% (72%-94%); and more than 2 yrs' delay, 68% (50%-83%). Those completing an accredited fellowship performed better (92% pass rate) on the Part II examination than did those who did not (81% pass rate), but they also had higher final year evaluations from their residency program directors.

CONCLUSIONS

It seems preferable to take both Part I and Part II examinations as soon as one is eligible. Enrollment in an accredited fellowship may be associated with a higher chance of passing the examination.

Student Evaluations, Outcomes, and National Licensure Examinations in Radiology Education: A Narrative Review of the Literature

OBJECTIVE

The purpose of this study was to examine literature on radiological student evaluation and outcome assessments including national board examinations.

METHODS

A review of the literature was performed using relevant key words. Articles were retrieved through December 2012 using PubMed, ScienceDirect, ERIC, Proquest, and ICL databases along with a manual review of references.

RESULTS

Of the 4716 unique abstracts reviewed by the author, 54 were found to be relevant to the purpose of this study. Student grade point average correlates with board scores in the nursing, chiropractic, and medical professions. Scores on the chiropractic college admission test and undergraduate grade point average correlate with success in professional college. There is a correlation between board scores and college attended. Board preparation programs do not appear to affect board examination scores.

CONCLUSION

Although evaluations can be effective teaching tools, they are not used by many radiology programs. Some programs have inadequate evaluations and do not allow students to review their evaluations. There are no definitive links between mastery of radiology and specific evaluations, outcomes, or pre-professional/clinical grades. Studies suggest that board examination scores reflect long-term mastery of knowledge rather than short-term memorization of facts.

Can American Board of Physical Medicine and Rehabilitation Part 2 board examination scores be predicted from rotation evaluations or mock oral examinations?

OBJECTIVE

The aim of this study was to determine whether either Physical Medicine and Rehabilitation residency performance on core competency evaluations or on practice mock oral examinations is correlated to performance on future American Board of Physical Medicine and Rehabilitation Part 2 board-certifying examination.

DESIGN

This is a retrospective cohort study of residents who took part 2 of the American Board of Physical Medicine and Rehabilitation certification examination between 1995 and 2011 (N = 31 or 38 or 67).

RESULTS

The postgraduate year 4 mock oral examination average achieved significance in correlation analysis (Spearman ρ, 0.0391; P = 0.030). Patient care and a composite average of the other core competencies evaluations were also significantly correlated with performance on part 2 of the board-certifying examination (Spearman ρ, 0.329; P = 0.044). The only independent variable that was uniquely predictive was postgraduate year 4 mock oral examinations (χ = 7.09; P = 0.029). More specifically, when controlling for rotation performances, residents with higher mock oral examination scores were 9.6 times (Exp B = 9.6; 95% confidence interval, 1.2-80; P = 0.036) more likely than those one grade lower to achieve the upper half on oral board examinations vs. either of the lower 2 quartiles.

CONCLUSIONS

The postgraduate year 4 mock oral examinations and the core competency evaluations composite are each predictive of performance on American Board of Physical Medicine and Rehabilitation part 2 examination. Further research into this area, with a larger sample size and with multiple institutions, would be helpful to allow for a better measurement of these evaluation tools' effectiveness.

The relationship between faculty performance assessment and results on the in-training examination for residents in an emergency medicine training program

BACKGROUND

Medical knowledge (MK) in residents is commonly assessed by the in-training examination (ITE) and faculty evaluations of resident performance.

OBJECTIVE

We assessed the reliability of clinical evaluations of residents by faculty and the relationship between faculty assessments of resident performance and ITE scores.

METHODS

We conducted a cross-sectional, observational study at an academic emergency department with a postgraduate year (PGY)-1 to PGY-3 emergency medicine residency program, comparing summative, quarterly, faculty evaluation data for MK and overall clinical competency (OC) with annual ITE scores, accounting for PGY level. We also assessed the reliability of faculty evaluations using a random effects, intraclass correlation analysis.

RESULTS

We analyzed data for 59 emergency medicine residents during a 6-year period. Faculty evaluations of MK and OC were highly reliable (κ  =  0.99) and remained reliable after stratification by year of training (mean κ  =  0.68-0.84). Assessments of resident performance (MK and OC) and the ITE increased with PGY level. The MK and OC results had high correlations with PGY level, and ITE scores correlated moderately with PGY. The OC and MK results had a moderate correlation with ITE score. When residents were grouped by PGY level, there was no significant correlation between MK as assessed by the faculty and the ITE score.

CONCLUSIONS

Resident clinical performance and ITE scores both increase with resident PGY level, but ITE scores do not predict resident clinical performance compared with peers at their PGY level.