A Comparison of Remote vs In-Person Proctored In-Training Examination Administration for Internal Medicine

PURPOSE

In response to COVID-19, the American College of Physicians provided residents the option to complete the 2020 Internal Medicine In-Training Examination (IM-ITE) via in-person and remote proctoring. This study evaluated the extent to which scores obtained from both testing modalities were comparable.

METHOD

Data were analyzed from residents from all U.S.-based Accreditation Council for Graduate Medical Education-accredited IM residency programs and participating Canadian and international programs who completed the IM-ITE in 2020. The final sample contained 27,115 IM residents: 9,205 postgraduate year (PGY) 1, 9,332 PGY-2, and 8,578 PGY-3. Testing modality, gender, PGY, time spent on assessment, and native language were used to predict percent-correct scores in a multilevel regression model. This model included all main effects and all 2-way interactions between testing modality and each resident-level demographic variable, allowing those effects to be controlled for.

RESULTS

Of 27,115 residents studied, 11,354 (42%) tested remotely and 15,761 (58%) in person. Across the parameters of interest (main effect of testing modality and 2-way interactions), the only statistically significant effects were the interaction effects between testing mode (interaction effect: -0.61; 95% confidence interval (CI): -1.01, -0.21) and PGY (interaction effect: -0.54; 95% CI: -0.95, -0.13) (P = .002). Differences between in-person and remote predicted scores were slightly larger for PGY-1 than for PGY-2 and PGY-3 residents (controlling for the other predictors in the model), but the magnitude of these differences across residency training was well under a single percentage point. Because these statistically significant effects were deemed educationally nonsignificant, the study concluded that performance did not substantively differ across in-person and remote examinees.

CONCLUSIONS

Residents taking the 2020 IM-ITE performed similarly across in-person and remote proctoring. This study provides evidence of score comparability across the 2 testing modalities and supports continued use of remote proctoring for the IM-ITE.

Uncovering the Complexity of Item Position Effects in a Low-Stakes Testing Context

Previous researchers have only either adopted an item or examinee perspective to position effects, where they focused on exploring the relationships among position effects and item or examinee variables separately. Unlike previous researchers, we adopted an integrated perspective to position effects, where we focused on exploring the relationships among position effects, item variables, and examinee variables simultaneously. We evaluated the degree to which position effects on two separate low-stakes tests administered to two different samples were moderated by different item (item length, number of response options, mental taxation, and graphic) and examinee (effort, change in effort, and gender) variables. Items exhibited significant negative linear position effects on both tests, with the magnitude of the position effects varying from item to item. Longer items were more prone to position effects than shorter items; however, the level of mental taxation required to answer the item, the presence of a graphic, and the number of response options were not related to position effects. Examinee effort levels, change in effort patterns, and gender did not moderate the relationships among position effects and item features.

Revisiting Retake Policy: Analyzing the Success Rates of Examinees With Multiple Attempts on the United States Medical Licensing Examination

PURPOSE

A testing program's decision making on retake policy considers a number of factors, including fairness to examinees, examination security, examination purpose, and classification accuracy. For high-stakes licensure and certification examinations charged with protection of the public, this includes balancing fairness issues inherent in the potential for false negatives with the public protection need to minimize false positives. Since 2012, the United States Medical Licensing Examination (USMLE) has maintained a policy of ≤ 6 attempts on any examination component. The purpose of this study was to empirically examine the appropriateness of existing USMLE retake policy.

METHOD

A sample of over 300,000 examinees who were administered each Step examination between 2008 and 2018 was examined to address how pass rates (per attempt and cumulatively) differed for examinees and examinee subgroups via graphic depictions of pass rates, the impact of restricting the number of attempts on examinees and examinee subgroups via logistic regression analyses, and the impact of restricting the number of attempts on classification error via classification error analyses.

RESULTS

Repeaters displayed consistently lower pass rates on subsequent attempts. Regardless of Step, most passing examinees did so by their fourth attempt. Models predicting the log odds of examinees taking ≥ 5 attempts showed a significant effect of gender, race, and medical school status, although the associated sample sizes are small. Misclassification analyses showed that, as attempts increased, the percentage of false passers increased and percentage of false failers decreased.

CONCLUSIONS

These considerations led to the USMLE's decision to reduce its attempt limit from 6 to 4 on any Step or Step component effective July 2021. This change balances the competing interests of fairness and examination access to the examinee population, while simultaneously maintaining a minimum standard consistent with the mission of the USMLE program to protect the public.

Does the Spiritual Values/Religion Subscale of the Self-Description Questionnaire III Function Differentially Across Heterosexual and Non-Heterosexual Young Adults? A Measurement Invariance Study

We evaluated the dimensionality and measurement invariance of the Spiritual Values/Religion (SVR) subscale from the Self-Description Questionnaire III across heterosexual and non-heterosexual young adults. We found a one-factor model provided adequate fit to the data for each group, with the SVR items exhibiting configural, metric, and scalar invariance across the two groups. Given that we established measurement invariance, we examined the latent mean difference on the construct and found the heterosexual group reported significantly higher levels of spiritual value/religion than the non-heterosexual group. Our results provided empirical support for the theorized factor structure of the SVR items and the use of the SVR subscale across heterosexual and non-heterosexual young adults, making the scale a viable option for researchers studying religiosity in these specific subpopulations.

Relations among religiosity, age of self-identification as gay, lesbian, or bisexual, and alcohol use among college students

Research indicates that lesbian, gay, and bisexual (LGB) youth are at increased risk for heavy drinking relative to their heterosexual peers. One possible contributing factor is that religiosity fails to provide the significant protection for LGB youth that it provides in general population samples. Although prior studies provide some support for this hypothesis, there is little research on the reasons that religiosity may fail to protect against heavy drinking among LGB youth. The current study addressed this question by examining relations among religiosity, age of self-identification as LGB, and alcohol use in a sample of 162 young adults self-identified as lesbian, gay, bisexual, or questioning. Age of self-identification was conceptualized as an indicator of stress or internal conflict between religious beliefs and self-identification. We hypothesized that higher levels of religiosity would be associated with a later age of identification as LGB, which, in turn, would contribute to heavier drinking. Path analyses conducted in Mplus provided support for this hypothesis, with higher levels of religiosity indirectly contributing to increased alcohol use through later age of self-identification. Whereas additional studies directly assessing stress associated with the self-identification process are needed, the results of this study suggest that religiosity may serve as a risk, rather than protective, factor among LGB youth. The development of religious support groups specifically for LGB youth may help these individuals reconcile their religious beliefs with their emerging sexual identities, thereby allowing them to derive some of the same benefits that heterosexual youth derive from religious beliefs and practices. (PsycInfo Database Record (c) 2020 APA, all rights reserved).