Standard setting made easy: validating the Equal Z-score (EZ) method for setting cut-score for clinical examinations

Equal Z standard-setting method to estimate the minimum number of panelists for a medical school’s objective structured clinical examination in Taiwan: a simulation study

PURPOSE

Undertaking a standard-setting exercise is a common method for setting pass/fail cut scores for high-stakes examinations. The recently introduced equal Z standard-setting method (EZ method) has been found to be a valid and effective alternative for the commonly used Angoff and Hofstee methods and their variants. The current study aims to estimate the minimum number of panelists required for obtaining acceptable and reliable cut scores using the EZ method.

METHODS

The primary data were extracted from 31 panelists who used the EZ method for setting cut scores for a 12-station of medical school’s final objective structured clinical examination (OSCE) in Taiwan. For this study, a new data set composed of 1,000 random samples of different panel sizes, ranging from 5 to 25 panelists, was established and analyzed. Analysis of variance was performed to measure the differences in the cut scores set by the sampled groups, across all sizes within each station.

RESULTS

On average, a panel of 10 experts or more yielded cut scores with confidence more than or equal to 90% and 15 experts yielded cut scores with confidence more than or equal to 95%. No significant differences in cut scores associated with panel size were identified for panels of 5 or more experts.

CONCLUSION

The EZ method was found to be valid and feasible. Less than an hour was required for 12 panelists to assess 12 OSCE stations. Calculating the cut scores required only basic statistical skills.

The Impact of COVID-19 Pandemic on the Learning Outcomes of Medical Students in Taiwan: A Two-Year Prospective Cohort Study of OSCE Performance

Background/Aims: To avoid the negative impacts of the COVID-19 pandemic on clinical clerkship, supplemental teachings such as digital materials in the scenario-based distal simulations were implemented. This study utilized the OSCE (objective-structured clinical examination) to evaluate the impact of COVID-19 pandemic on the learning outcome of medical students from the regular group (class of 2020) and pandemic-impacted group (class of 2021). Methods: All medical students serially took, firstly, the mock-OSCE, secondly, the mock-OSCE, and the national OSCE. Then, the serial OSCE scores were compared between groups. Results: Although with similar scores in the first mock OSCE, the regular group (n = 78) had a higher average score in the national OSCE than the pandemic-impacted group (n = 80) (872.18 vs. 834.96, p = 0.003). In terms of improvement, the performances of the regular group were also better than the pandemic-impacted group between the second mock OSCE and the national OSCE (79.10 vs. 38.14, p = 0.014), and between the second mock OSCE and the national OSCE (125.11 vs. 77.52, p = 0.003). While separating distinct genres, the regular group had more of a score increment in standardized patient-based stations between the second mock OSCE and the national OSCE (regular vs. pandemic-impacted: 57.03 vs. 18.95, p = 0.003), as well as between the first mock OSCE and the national OSCE (75.97 vs. 26.36, p < 0.001), but there was no significant difference among the skill-based stations. In particular, the scores of the emergency medicine associated station in the national OSCE of the pandemic-impacted group was lower. Conclusions: Our study implies that the pandemic significantly hampered the learning outcomes of final year medical students in their clinical participation. Especially facing the COVID-19 pandemic, more supplemental teachings are needed to compensate the decreasing emergency medicine exposure.