Understanding underperformance in a high-stakes clinical-based simulation assessment in physiotherapy: a descriptive analysis

BACKGROUND

High-stakes assessments are often used as a 'gate-keeper' activity for entry into the health professions by ensuring that the minimum core competency thresholds of the profession are met. The aim of the study was to explore if common areas of underperformance existed in international candidates assessed with a high-stakes clinical-based simulation assessment for entry into the physiotherapy profession in Australia.

METHODS

A retrospective mixed methods analysis of the clinical assessments completed by international candidates over a one-month period in 2021 that were deemed as not meeting competency. The clinical assessments were completed in one of the three practice areas: cardiorespiratory, musculoskeletal, or neurological rehabilitation. Each assessment was scored by two independent assessors, who discussed the performance and then completed a moderated assessment form. The assessment form used to score competency included seven domains such as initial assessment, effective treatment, communication skills, and risk management.

RESULTS

Fifty-one clinical assessments graded as not competent were analysed. Across the practice areas, a high failure rate was found in domains related to interpreting assessment findings and developing a treatment plan. This trend was also observed in the qualitative data, suggesting candidates struggled to meet competency in areas of planning and prioritisation, interpretation and implementation of the information gathered, and selection and evaluation of effective treatment.

CONCLUSION

These findings align with published data on the underperformance of Australian physiotherapy students in clinical placement settings, suggesting these issues are not specific to high stakes assessment of overseas physiotherapists, and that education needs to focus on improving these skills within the profession at all levels. With the identified areas of underperformance aligning with the ability to use higher order thinking and skills integral to clinical reasoning, improvements in the education and implementation of clinical reasoning may be a place to start.

Simulation-based clinical assessment identifies threshold competence to practise physiotherapy in Australia: a crossover trial

BACKGROUND

Although evidence exists for the efficacy of high-fidelity simulation as an educational tool, there is limited evidence for its application in high-stakes professional threshold competency assessment. An alternative model of simulation-based assessment was developed by the Australian Physiotherapy Council (APC), using purpose-written standardised patients, mapped to the appropriate threshold level. The aim of this two-phase study was to investigate whether simulation-based clinical assessments resulted in equivalent outcomes to standard, real-life assessments for overseas-trained physiotherapists seeking registration to practice in Australia.

METHODS

A randomised crossover trial comparing simulation-based assessment to real-life assessment was completed. Participants were internationally trained physiotherapists applying for registration to practice in Australia, voluntarily recruited from the Australian Physiotherapy Council (APC) assessment waiting list: study 1 n = 25, study 2 n = 144. Study 1 participants completed usual APC real-life assessments in 3 practice areas, completed on different days at APC partner healthcare facilities. Participants also underwent 3 practice area-matched simulation-based assessments, completed on the same day at purpose-designed simulation facilities. Study 2 participants completed 3 simulation-based assessments and 1 real-life assessment that was randomly allocated for order and practice area. Assessment of competency followed the standard APC procedure of 90-minute examinations using The Moderated Assessment Form (MAF).

RESULTS

The overall pass rate was higher for real-life assessments in both studies: study 1, 50% versus 42.7%; study 2, 55.6% versus 44.4%. Chi-square analysis showed a high to moderate level of exact matching of pass/fail grades across all assessments: study 1, 73.4% (p < 0.001); study 2, 58.3% (p = 0.027). Binary logistic regression showed that the best predictors of real-life pass/fail grade were simulation-based MAF pass/fail grade (study 1, OR 7.86 p < 0.001; study 2, OR 2.037, p = 0.038) and simulation-based total MAF score (study 1, OR 1.464 p < 0.001; study 2, OR 1.234, p = 0.001).

CONCLUSION

Simulation-based assessment is a significant predictor of clinical performance and can be used to successfully identify high stakes threshold competence to practice physiotherapy in Australia.

Simulation for Maintenance of Certification

Maintenance of certification (MOC) is a process through which practitioners are able to show continuing competence in their areas of expertise. Simulation plays an increasingly important role in the assessment of students and residents, as well as in the initial practice certification for health care professionals. The use of simulation as an assessment tool in MOC has been sluggish to be universally accepted. This article discusses the role of simulation in health care education, how simulation might be effectively applied in the MOC process, and the future role of simulation in the MOC process.