Evaluating factors in the selection of residents

Exploring Potential Schedule-Related and Gender Biases in Ophthalmology Residency Interview Scores

Purpose Prior studies have revealed grading discrepancies in evaluation of personal statements and letters of recommendation based on candidate's race and gender. Fatigue and the end-of-day phenomenon can negatively impact task performance but have not been studied in the residency selection process. Our primary objective is to determine whether factors related to interview time and day as well as candidate's and interviewer's gender have a significant effect on residency interview scores.

Methods Seven years of ophthalmology residency candidate evaluation scores from 2013 to 2019 were collected at a single academic institution, standardized by interviewer into relative percentiles (0–100 point grading scale), and grouped into the following categories for comparisons: different interview days (Day 1 vs. Day 2), morning versus afternoon (AM vs. PM), interview session (Day 1 AM/PM vs. Day 2 AM/PM), before and after breaks (morning break, lunch break, and afternoon break), residency candidate's gender, and interviewer's gender.

Results Candidates in the morning sessions were found to have higher scores than afternoon sessions (52.75 vs. 49.28, p < 0.001). Interview scores in the early morning, late morning, and early afternoon were higher than late afternoon scores (54.47, 53.01, 52.15 vs. 46.74, p < 0.001). Across all interview years, there were no differences in scores received before and after morning breaks (51.71 vs. 52.83, p = 0.49), lunch breaks (53.01 vs. 52.15, p = 0.58), and afternoon breaks (50.35 vs. 48.30, p = 0.21). No differences were found in scores received by female versus male candidates (51.55 vs. 50.49, p = 0.21) or scores given by female versus male interviewers (51.31 vs. 50.84, p = 0.58).

Conclusion Afternoon residency candidate interview scores, especially late afternoon, were significantly lower than morning scores, suggesting the need to further study the effects of interviewer's fatigue in the residency interview process. The interview day, presence of break times, candidate's gender, and interviewer's gender had no significant effects on interview score.

Use of the Interview in Resident Candidate Selection: A Review of the Literature

BACKGROUND

Although the resident candidate interview is costly and time-consuming for both applicants and programs, it is considered critically important for resident selection. Noncognitive attributes, including communication skills and professionalism, can be assessed by the personal interview.

OBJECTIVE

We conducted a review of the literature on the residency interview to identify the interview characteristics used for resident selection and to ascertain to what extent the interview yields information that predicts future performance.

METHODS

We searched PubMed and Scopus using the following search terms: residency, internship, interview, selection, and performance. We extracted information on characteristics of the interview process, including type of interview format, measures taken to minimize bias by interviewers, and testing of other clinical/surgical skills.

RESULTS

We identified 104 studies that pertained to the resident selection interview, with highly varied interview formats and assessment tools. A positive correlation was demonstrated between a medical school academic record and the interview, especially for unblinded interview formats. A total of 34 studies attempted to correlate interview score with performance in residency, with mixed results. We also identified a number of studies that included personality testing, clinical skills testing, or surgical skills testing.

CONCLUSIONS

Our review identified a wide variety of approaches to the selection interview and a range of factors that have been studied to assess its effectiveness. More research needs to be done not only to address and ascertain appropriate interview formats that predict positive performance in residency, but also to determine interview factors that can predict both residents' "success" and program attrition.

Reliability of a structured interview for admission to an emergency medicine residency program

BACKGROUND

Interviews are most important in resident selection. Structured interviews are more reliable than unstructured ones.

PURPOSE

We sought to measure the interrater reliability of a newly designed structured interview during the selection process to an Emergency Medicine residency program.

METHODS

The critical incident technique was used to extract the desired dimensions of performance. The interview tool consisted of 7 clinical scenarios and 1 global rating. Three trained interviewers marked each candidate on all scenarios without discussing candidates' responses. Interitem consistency and estimates of variance were computed.

RESULTS

Twenty-eight candidates were interviewed. The generalizability coefficient was 0.67. Removing the central tendency ratings increased the coefficient to 0.74. Coefficients of interitem consistency ranged from 0.64 to 0.74.

CONCLUSIONS

The structured interview tool provided good although suboptimal interrater reliability. Increasing the number of scenarios improves reliability as does applying differential weights to the rating scale anchors. The latter would also facilitate the identification of those candidates with extreme ratings.

Correlating selection criteria with subsequent performance as residents

The objective of this study was to determine which criteria in the residency application had the highest correlation with subsequent performance of orthopaedic residents. Data collected from the application files of 58 residents included scores on standardized tests, number of honors grades in the basic and clinical years of medical school, election to Alpha Omega Alpha, numbers of research projects and publications, and numbers of extracurricular activities. Measures of performance included scores on the Orthopaedic In-Training Examination and American Board of Orthopaedic Surgery Part I Examination, and faculty evaluations of overall, cognitive, affective, and psychomotor performance. The number of honors grades on clinical rotations was the strongest predictor of performance, whereas election to Alpha Omega Alpha was second. The only other significant correlation was between the number of fine motor activities and psychomotor performance. None of the predictor variables had a significant correlation with Orthopaedic In-Training Examination or American Board of Orthopaedic Surgery Examination scores. Consistency between faculty rankings in each of the four categories was supported by regression analysis. From the results of this study, it appears that academic performance in clinical clerkships in medical school is the most predictive of resident performance. Range restriction in the data available for orthopaedic residency applicants, however, likely precludes the development of a reliable model to assist in the selection of orthopaedic residents.

Evaluation of a Resident Selection Process in Plastic Surgery

Methods of residency selection continue to be a contentious issue, with little being known about the reliability of the various components of the selection process. By determining the reliability and validity of application reviews and interview judgements, the process of selecting candidates for a residency program can be made more systematic and more efficient. The present study examined the selection process used at the University of Toronto Division of Plastic Surgery for the 1997 selection year. Analysis revealed the initial review of the 51 applications to have high inter-rater reliability, with a panel of three judges producing a reliability of greater than 0.80, and high test-retest reliability (r=0.75). A second review of the 14 applications from candidates selected for interview was somewhat less reliable (probably due to the restriction of range in the scores) and required at least six raters to obtain a reliability of 0.80. Inter-rater reliability of the scores from two independent interviews was quite high, with four raters providing a reliability of at least 0.80 in both interviews. Test-retest reliability between the two interviews was 0.77. Finally, the overall ‘gut’ rankings of the 11-member selection committee were quite reliable and were predicted by a combination of application scores and interview scores (multiple R of 0.96). At least within this program, each component of the selection process appeared to be quite reliable and final ‘gut’ decisions were well founded in a reasonable balance of application scores and interview scores.