Improving reliability of a residency interview process

Impact of virtual vs. in-person interviews among neurosurgery residency applicants

BACKGROUND

The interview is considered a key factor in selecting residents in various medical and surgical specialties. However, the reliability of the interview process in selecting neurosurgery training program applicants remains largely under-investigated.

OBJECTIVE

To investigate the reliability of the interview process for neurosurgery residency applicants and to evaluate the impact of virtual interviews on this process.

METHODS

We analyzed the records of neurosurgery residency applicant interviews at our institution between 2016 and 2021. An average of 20 neurosurgery faculty members (clinical and research) interviewed each applicant and graded them 1 (best) to 4 (worst). Intraclass correlation coefficient (ICC) and Levene's test were used to assess the inter-rater and intra-rater reliability, respectively.

RESULTS

214 neurosurgery residency applicants were interviewed at a single institution between 2016 and 2021. The mean applicant rating each year ranged from 1.77 to 1.92. Inter-rater agreement was relatively poor in each year, (ICC < 0.5, P < 0.05). Among 60% of the raters, variability of scores significantly changed from year to year, (p < 0.05). When comparing the scores submitted during the virtual interview process (2021) with the scores submitted in the previous years (2016-2020), 2 interviewers (10%) had less variability using the virtual process.

CONCLUSION

Our analysis found that the current interview process for neurosurgery residency applicants' selection suffers from poor inter- and intra-rater reliability. Virtual interviews may be part of a cost-effective strategy to improve the reliability of the interview process. Further validation is needed, as well as identification of novel strategies to maximize the reliability of the selection process.

Moving beyond Cronbach's Alpha and Inter-Rater Reliability: A Primer on Generalizability Theory for Pharmacy Education

BACKGROUND

When available, empirical evidence should help guide decision-making. Following each administration of a learning assessment, data becomes available for analysis. For learning assessments, Kane's Framework for Validation can helpfully categorize evidence by inference (i.e., scoring, generalization, extrapolation, implications). Especially for test-scores used within a high-stakes setting, generalization evidence is critical. While reporting Cronbach's alpha, inter-rater reliability, and other reliability coefficients for a single measurement error are somewhat common in pharmacy education, dealing with multiple concurrent sources of measurement error within complex learning assessments is not. Performance-based assessments (e.g., OSCEs) that use raters, are inherently complex learning assessments.

PRIMER

Generalizability Theory (G-Theory) can account for multiple sources of measurement error. G-Theory is a powerful tool that can provide a composite reliability (i.e., generalization evidence) for more complex learning assessments, including performance-based assessments. It can also help educators explore ways to make a learning assessment more rigorous if needed, as well as suggest ways to better allocate resources (e.g., staffing, space, fiscal). A brief review of G-Theory is discussed herein focused on pharmacy education.

MOVING FORWARD

G-Theory has been common and useful in medical education, though has been used rarely in pharmacy education. Given the similarities in assessment methods among health-professions, G-Theory should prove helpful in pharmacy education as well. Within this Journal and accompanying this Idea Paper, there are multiple reports that demonstrate use of G-Theory in pharmacy education.

Implementation of a modified multiple mini-interview method to assess non-cognitive qualities during resident candidate interviews

BACKGROUND AND PURPOSE

Conventional onsite interview methods often make comparing applicants difficult. Literature has noted conventional interviews leave room for bias and high interrater variability, making non-cognitive attributes difficult to ascertain. In 2016, the residency committee of a small, multi-site, academic-based postgraduate year one residency program implemented a modified multiple mini-interview (MMI) approach as a component of the residency interview process to better qualify candidate attributes.

EDUCATIONAL ACTIVITY AND SETTING

A modified MMI was developed to address the non-cognitive attributes, ethical reasoning, communication, and professionalism. Scenarios, scripts, questions, and rubrics were developed by residency committee members. The author of the case was assigned to role play that scenario with candidates while other committee members silently observed. Candidates and residency committee members were surveyed to explore their perception of the MMI as a component of the residency interview process.

FINDINGS

Thirty-one candidates have been interviewed since the incorporation of the modified MMI. Of those, 20 completed the post-interview survey. The majority of resident candidates (55%) completing the survey felt they were able to portray strengths and abilities more effectively vs. a conventional interview. Of the five residency committee members, all (100%) completed the survey and all (100%) perceived implementation of the modified MMI provided increased confidence in determining candidate ranking.

SUMMARY

Implementation of a modified MMI approach to an onsite residency interview process assisted residency committee members in assessing non-cognitive attributes and contributed to greater confidence in determining resident candidate ranking.

Validation of learning assessments: A primer

The Accreditation Council for Pharmacy Education's Standards 2016 has placed greater emphasis on validating educational assessments. In this paper, we describe validity, reliability, and validation principles, drawing attention to the conceptual change that highlights one validity with multiple evidence sources; to this end, we recommend abandoning historical (confusing) terminology associated with the term validity. Further, we describe and apply Kane's framework (scoring, generalization, extrapolation, and implications) for the process of validation, with its inferences and conclusions from varied uses of assessment instruments by different colleges and schools of pharmacy. We then offer five practical recommendations that can improve reporting of validation evidence in pharmacy education literature. We describe application of these recommendations, including examples of validation evidence in the context of pharmacy education. After reading this article, the reader should be able to understand the current concept of validation, and use a framework as they validate and communicate their own institution's learning assessments.

Assessment of SOAP note evaluation tools in colleges and schools of pharmacy

INTRODUCTION

To describe current methods used to assess SOAP notes in colleges and schools of pharmacy.

METHODS

Members of the American Association of Colleges of Pharmacy Laboratory Instructors Special Interest Group were invited to share assessment tools for SOAP notes. Content of submissions was evaluated to characterize overall qualities and how the tools assessed subjective, objective, assessment, and plan information.

RESULTS

Thirty-nine assessment tools from 25 schools were evaluated. Twenty-nine (74%) of the tools were rubrics and ten (26%) were checklists. All rubrics included analytic scoring elements, while two (7%) were mixed with holistic and analytic scoring elements. A majority of the rubrics (35%) used a four-item rating scale. Substantial variability existed in how tools evaluated subjective and objective sections. All tools included problem identification in the assessment section. Other assessment items included goals (82%) and rationale (69%). Seventy-seven percent assessed drug therapy; however, only 33% assessed non-drug therapy. Other plan items included education (59%) and follow-up (90%).

DISCUSSION AND CONCLUSIONS

There is a great deal of variation in the specific elements used to evaluate SOAP notes in colleges and schools of pharmacy. Improved consistency in assessment methods to evaluate SOAP notes may better prepare students to produce standardized documentation when entering practice.

Resident characteristics to evaluate during recruitment and interview: a Delphi study

BACKGROUND AND OBJECTIVES

The recruitment and interview process for medical residency programmes is a time- and resource-intensive effort. There is very little research to guide programmes when evaluating residency candidates. This study represents one step in identifying candidate characteristics to assess during the recruitment and interview process.

METHODS

Sixteen expert interviewers from 14 family medicine residency programmes in North Carolina participated in a three-round Delphi study to build consensus around a ranked list of successful resident candidate characteristics. An interrater reliability analysis produced average pair-wise agreement and Krippendorff's Alpha coefficients.

RESULTS

Clinical skills, medical knowledge, interpersonal and communication skills, critical thinking, and professional and ethical behaviour were the highest ranked characteristics. Average pair-wise agreement for rounds two and three were 6.30 and 11.04%, respectively.

CONCLUSIONS

Residency programmes may benefit from using an empirically studied list of characteristics to evaluate candidate applications and interviews. Future research should include national surveys of expert interviewers from a variety of residency programmes and a longitudinal study to correlate interview evaluations using the ranked list with measures of residency success.

Educational testing and validity of conclusions in the scholarship of teaching and learning

Validity and its integral evidence of reliability are fundamentals for educational and psychological measurement, and standards of educational testing. Herein, we describe these standards of educational testing, along with their subtypes including internal consistency, inter-rater reliability, and inter-rater agreement. Next, related issues of measurement error and effect size are discussed. This article concludes with a call for future authors to improve reporting of psychometrics and practical significance with educational testing in the pharmacy education literature. By increasing the scientific rigor of educational research and reporting, the overall quality and meaningfulness of SoTL will be improved.