The Predictive Validity of the National Board of Osteopathic Medical Examiners' COMLEX-USA Examinations With Regard to Outcomes on American Board of Family Medicine Examinations

A validity study of COMLEX-USA Level 3 with the new test design

CONTEXT

The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores.

OBJECTIVES

The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework.

METHODS

Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE).

RESULTS

All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure.

CONCLUSIONS

This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series.

Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

PURPOSE

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency.

METHODS

The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics.

RESULTS

Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ± 63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively.

CONCLUSION

USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Examining concurrent validity between COMLEX-USA Level 2-Cognitive Evaluation and COMLEX-USA Level 2-Performance Evaluation

CONTEXT

The Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) is a three level examination used as a pathway to licensure for students in osteopathic medical education programs. COMLEX-USA Level 2 includes a written assessment of Fundamental Clinical Sciences for Osteopathic Medical Practice (Level 2-Cognitive Evaluation [L2-CE]) delivered in a computer based format and separate performance evaluation (Level 2-Performance Evaluation [L2-PE]) administered through live encounters with standardized patients. L2-PE was designed to augment L2-CE. It is expected that the two examinations measure related yet distinct constructs.

OBJECTIVES

To explore the concurrent validity of L2-CE with L2-PE.

METHODS

First attempt test scores were obtained from the National Board of Osteopathic Medical Examiners database for 6,639 candidates who took L2-CE between June 2019 and May 2020 and matched to the students' L2-PE scores. The sample represented all colleges of osteopathic medicine and 97.5% of candidates who took L2-CE during the complete 2019-2020 test cycle. We calculated disattenuated correlations between the total score for L2-CE, the L2-CE scores for the seven competency domains (CD1 through CD7), and the L2-PE scores for the Humanistic Domain (HM) and Biomedical/Biomechanical Domain (BM). All scores were on continuous scales.

RESULTS

Pearson correlations ranged from 0.10 to 0.88 and were all statically significant (p<0.01). L2-CE total score was most strongly correlated with CD2 (0.88) and CD3 (0.85). Pearson correlations between the L2-CE competency domain subscores ranged from 0.17 to 0.70, and correlations which included either HM or BM ranged from 0.10 to 0.34 with the strongest of those correlations being between BM and L2-CE total score (0.34) as well as between HM and BM (0.28).The largest increase between corresponding Pearson and disattenuated correlations was for pairs of scores with lower reliabilities such as CD5 and CD6, which had a Pearson correlation of 0.17 and a disattenuated correlation of 0.68. The smallest increase in correlations was observed in pairs of scores with larger reliabilities such as L2-CE total score and HM, which had a Pearson correlation of 0.23 and a disattenuated correlation of 0.28. The reliability of L2-CE was 0.87, 0.81 for HM, and 0.73 for BM. The reliabilities for the L2-CE competency domain scores ranged from 0.22 to 0.74. The small to moderate correlations between the L2-CE total score and the two L2-PE support the expectation that these examinations measure related but distinct constructs. The correlations between L2-PE and L2-CE competency domain subscores reflect the distribution of items defined by the L2-PE blueprint, providing evidence that the examinations are performing as designed.

CONCLUSIONS

This study provides evidence supporting the validity of the blueprints for constructing COMLEX-USA Levels 2-CE and 2-PE examinations in concert with the purpose and nature of the examinations.

2019 United States Osteopathic Medical Regulatory Summit: Consensus, Recommendations, and Next Steps in Defining Osteopathic Distinctiveness

Background

Osteopathic distinctiveness is a result of professional education, identity formation, training, credentialing, and qualifications. With the advancement of a single graduate medical education (GME) accreditation system and the continued growth of the osteopathic medical profession, osteopathic distinctiveness and professional identity are seen as lacking clarity and pose a challenge.

Summit

To achieve consensus on a succinct definition of osteopathic distinctiveness and to identify steps to more clearly define and advance that distinctiveness, particularly in professional self-regulation, a representative group of osteopathic medical students, residents, physicians, and members of the licensing, GME, and undergraduate medical education (UME) communities convened the 2019 United States Osteopathic Medical Regulatory Summit in February 2019. Key features of osteopathic distinctiveness were discussed. Growth in the profession; changes in health care delivery, technology, and demographics within the profession and patient communities; and associated challenges and opportunities for osteopathic medical practice and patients were considered.

Consensus

Osteopathic medicine is a distinctive practice that brings unique, added value to patients, the public, and the health care community at large. A universal definition and common understanding of that distinctiveness is lacking. Efforts to unify messaging that defines osteopathic distinctiveness, to align the distinctive elements of osteopathic medical education and professional self-regulation across a continuum, and to advance research on care and educational program outcomes are critical to the future of the osteopathic medical profession.

Recommendations

(1) Convene a task force of groups represented at the Summit to develop a succinct and consistent message defining osteopathic distinctiveness. (2) Demonstrate uniqueness of the profession through research demonstrating efficacy of care and patient outcomes, adding to the public good. (3) Harmonize GME and UME by beginning to align entrustable professional activities with UME milestones. (4) Convene representatives from osteopathic specialty colleges and certification boards to define curricular elements across GME, certification, and osteopathic continuous certification. (5) Build on the Project in Osteopathic Medical Education and Empathy study.

An Investigation of the Relationship Between COMLEX-USA Licensure Examination Performance and State Licensing Board Disciplinary Actions

PURPOSE

Passing the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) serves as a licensing requirement, yet there is limited understanding between this high-stakes exam and performance outcomes. This study examined the relationship between COMLEX-USA scores and disciplinary actions received by osteopathic physicians.

METHOD

Data for osteopathic physicians (N = 26,383) who graduated from medical school between 2004 and 2013 were analyzed using multinomial logistic regression to assess the relationship between COMLEX-USA scores and placement into one of 3 disciplinary action categories relative to no action received, controlling for years in practice and gender.

RESULTS

Less than 1% of physicians in this study (n = 187) had a disciplinary action(s). Controlling for all COMLEX-USA levels, years in practice, and gender, higher Level 3 scores were associated with significant decreased odds for all action categories: revoked licensed (odds ratio [OR] = 0.51, 95% confidence interval [CI] 0.36, 0.72; P < .001), imposed limitations to practice (OR = 0.59, 95% CI 0.41, 0.84; P < .01), and other action imposed (OR = 0.48, 95% CI 0.33, 0.69; P < .001), relative to not receiving an action. In these same models, higher Level 2 Performance Evaluation Biomedical/Biomechanical Domain scores decreased the odds for an action that revoked a license (OR = 0.75, 95% CI 0.58, 0.98; P < .05) and imposed limitations to practice (OR = 0.64, 95% CI 0.49, 0.84; P < .001).

CONCLUSIONS

These findings provide evidence that the COMLEX-USA delivers useful information regarding the likelihood of a practitioner receiving state board disciplinary actions.

Success Predictors For Third-Year Osteopathic Medical Students on National Standardized Examinations: A Family Medicine Clerkship Course Study

CONTEXT

Medical education institutions often use community-based sites and preceptors during students' third and fourth years for clinical training. However, differences in the sites, preceptors, assessment methods, and students may result in variations in clinical training, potentially affecting educational outcomes. During clerkships at A.T. Still University School of Osteopathic Medicine in Arizona, all students are evaluated by several methods for each required clerkship course. Required assessments include the clinical preceptor's evaluation, online coursework specific to each clerkship, patient log documentation, and the Comprehensive Osteopathic Medical Achievement Test (COMAT) relevant to that clerkship.

OBJECTIVE

To evaluate which methods of student assessment in a family medicine clerkship course were most predictive of the future success of students on national standardized examinations.

METHODS

Third-year osteopathic medical students from a single class who had completed the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)-USA Level 2-Cognitive Evaluation (CE) and 2-Performance Evaluation (PE) and the COMAT were included in the study. Scores on the examinations were used as success benchmarks. Analysis of 4 categories of predictor variables-clerkship site, previous student performance, preceptor evaluation, and clerkship coursework assessment (ie, assignment scores and log numbers)-was used to predict success on the national standardized examinations.

RESULTS

Ninety-nine of 105 students were eligible for inclusion. No associations were found between examination scores and clerkship site or log numbers (all P≥.10). Correlations were found for previous student performance (ie, grade point average for first-year and second-year coursework) and all examinations except COMLEX-USA Level 2-PE (r=0.56-0.74, all P<.001), and between total score in family medicine clerkship coursework and COMLEX-USA Level 2-CE and COMAT scores (r=0.28-0.39, all P≤.006). Correlations were also found between preceptor evaluation (total score and subscore on medical knowledge) and all assessed national standardized examinations (r=0.20-0.34, all P<.049).

CONCLUSION

Our results suggest that analysis of predictor variables in clerkship courses can reasonably predict success on national standardized examinations and may be useful for early identification of struggling students who may need additional support to perform well on the examinations.

Association Between Performance on COMLEX-USA and the American College of Osteopathic Family Physicians In-Service Examination

BACKGROUND

The primary goal of residency programs is to select and educate qualified candidates to become competent physicians. Program directors often use performance on licensure examinations to evaluate the ability of candidates during the resident application process. The American College of Osteopathic Family Physicians (ACOFP) administers an in-service examination (ISE) to residents annually. There are few prior studies of the relationship between the Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series and formative assessments of residents in training.

OBJECTIVE

We explored the relationship between performance on COMLEX-USA and the ACOFP in-service examination to offer support on the use of licensing examinations in resident selection.

METHODS

In 2016, performance data from the COMLEX-USA and the ISE were matched for 3 resident cohorts (2011-2013, inclusive; N = 1384). Correlations were calculated to examine the relationship between COMLEX-USA and ISE scores. Multiple linear regression models were used to determine if performance on COMLEX-USA significantly predicted third-year ISE (ISE-3) scores.

RESULTS

Findings indicated that correlations among performance on COMLEX-USA and ISE were statistically significant (all P < .001), and there was strong intercorrelation between COMLEX-USA Level 3 and ISE-1 performance (r = 0.57, P < .001). Performance on the COMLEX-USA Levels 1 and 2-Cognitive Examination significantly predicted performance on the ISE-3 (F(2,1381) = 228.8, P < .001).

CONCLUSIONS

The results support using COMLEX-USA as a part of resident selection in family medicine. Additionally, program directors may use performance on COMLEX-USA to predict success on the ISE-3.

Enhancing COMLEX-USA: Evidence-Based Redesign of the Osteopathic Medical Licensure Examination Program

To ensure the Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) remains relevant and current in meeting the needs of the state licensing boards and other constituents, the National Board of Osteopathic Medical Examiners (NBOME) has developed a new blueprint for an enhanced, competency-based examination program to be implemented with the COMLEX-USA Level 3 examination in late 2018. This article summarizes the evidence-based design processes on which the new blueprint is built, how it differs from the previous blueprint, and the evidence supporting its validity for the primary and intended purpose of COMLEX-USA — osteopathic physician licensure. It concludes with the changes being implemented by the NBOME to ensure COMLEX-USA remains current and meets the needs of its stakeholders, the state licensing boards.