Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment

Can routine EPA-based assessments predict OSCE performances of undergraduate medical students?

BACKGROUND

Objective structured clinical examination (OSCE) is used worldwide. This study aims to explore potential alternatives to the OSCE by using entrustable professional activities (EPA)-based assessments in the workplace.

METHODS

This study enrolled 265 six-year undergraduate medical students (UGY) from 2021 to 2023. During their rotations, students were assessed using 13 EPAs, with the grading methods modified to facilitate application. Before graduation, they participated in two mock OSCEs and a National OSCE. We used generalized estimating equations to analyze the associations between the EPA assessments and the OSCE scores, adjusting for age and sex, and developed a prediction model. EPA8 and EPA9, which represent advanced abilities that were not significant in the regression models, were removed from the prediction model.

RESULTS

Most EPAs were significantly correlated with OSCE scores across the three cohorts. The prediction model for forecasting passing in the three OSCEs demonstrated fair predictive capacity (area under curve = 0.82, 0.66, and 0.71 for students graduated in 2021, 2022, and 2023, respectively all p < 0.05).

CONCLUSIONS

The workplace-based assessments (EPA) showed a high correlation with competency-based assessments in simulated settings (OSCE). EPAs may serve as alternative tools to formal OSCE for medical students.

Implementation and faculty perception of outpatient medical student workplace-based assessments

BACKGROUND

There is growing interest in use of entrustable professional activity (EPA)-grounded workplace-based assessments (WBAs) to assess medical students through direct observation in the clinical setting. However, there has been very little reflection on how these tools are received by the faculty using them to deliver feedback. Faculty acceptance of WBAs is fundamentally important to sustained utilisation in the clinical setting, and understanding faculty perceptions of the WBA as an adjunct for giving targeted feedback is necessary to guide future faculty development in this area.

APPROACH

Use of a formative EPA-grounded WBA was implemented in the ambulatory setting during the paediatrics clerkship following performance-driven training and frame-of-reference training with faculty. Surveys and semi-structured interviews with faculty members explored how faculty perceived the tool and its impact on feedback delivery.

EVALUATION

Faculty reported providing more specific, task-oriented feedback following implementation of the WBA, as well as greater timeliness of feedback and greater satisfaction with opportunities to provide feedback, although these later two findings did not reach significance. Themes from the interviews reflected the benefits of WBAs, persistent barriers to the provision of feedback and suggestions for improvement of the WBA.

IMPLICATIONS

EPA-grounded WBAs are feasible to implement in the outpatient primary care setting and improve feedback delivery around core EPAs. The WBAs positively impacted the way faculty conceptualise feedback and provide learners with more actionable, behaviour-based feedback. Findings will inform modifications to the WBA and future faculty development and training to allow for sustainable WBA utilisation in the core clerkship.