Validation of the Omni: A Novel, Multimodality, and Longitudinal Surgical Skills Assessment

A scoping review of assessment methods of competence of general surgical trainees

BACKGROUND

Only rigorous evaluation of competence will result in the production of safe surgeons that are able to provide the best care for patients. The development of competency-based assessment should ultimately be evidence driven.

OBJECTIVES

Explore the volume of existing evidence pertaining to the different objective assessment methods reported in the literature.

ELIGIBILITY CRITERIA

Studies describing objective assessment of postgraduate general surgical trainees within the last 20 years.

SOURCES OF EVIDENCE

PubMed, Ovid Medline and Web of Sciences.

CHARTING METHODS

A data chart proforma was designed and data were extracted into tables. Basic numerical analysis of extracted data and narrative synthesis of charted data.

RESULTS

A total of 343 papers were reviewed. 26 were eligible for inclusion. 92% of articles were published from 2008 onwards. 50% have been published in the last five years. The articles originated from 6 different countries, predominantly the United Kingdom (42%), followed by the United States of America (38%). In addition, a small number were published from Canada (8%), Japan (4%), Germany (4%) and Australia (4%). UK publications were predominantly between 2008 and 2014 while the USA had a later predominance between 2015 and 2018. 42% were based on quantitative methodology, 27% had a qualitative approach while 31% had mixed analysis. There were sixteen assessment methods presented. The most common type of assessment was Objective Structured Assessments (27%), which included Objective Structured Assessment of Technical Skill (OSATS) (23%) and Objective Structured Assessment of Non-Technical Skill (4%). Procedure Based Assessment (PBA) (23%) and Entrustability Scales (23%) were also prevalent.

CONCLUSIONS

This scoping review has identified a range of different assessment methods. The assessment methods with a higher volume and level of supporting evidence were OSATS, PBAs and Entrustability Scales. There was a lower volume and level of supporting evidence found within this review for the remaining assessment methods.

Postgraduate Assessment Approaches Across Surgical Specialties: A Systematic Review of the Published Evidence

PURPOSE

Ensuring competence for surgical trainees requires holistic assessment of the qualities and competencies necessary to practice safely and effectively. To determine the next steps toward achieving this aim, the authors conducted a systematic review to summarize and appraise the available evidence related to any assessment approach to postgraduate surgical training and to identify the dominant themes for assessment approaches across different specialties or countries.

METHOD

Medline and Embase were searched on January 10, 2019, without language or time restrictions. Any peer-reviewed study that described an assessment framework (in practice or novel) throughout postgraduate surgical training globally was included. An iterative review and thematic analysis were performed on full-text articles to determine assessment themes. Studies were then grouped by assessment themes. A tailored quality assessment of the studies included in the final analysis was conducted. Assessment themes and validity were compared across surgical specialties and countries.

RESULTS

From an initial 7,059 articles, 91 studies (evaluating 6,563 surgical trainees) were included in the final analysis. Ten defined assessment themes were extracted. Ten studies (11.0%) were deemed low risk of bias based on the quality assessment tool used and thus were determined to be high quality. Minor differences in assessment themes were observed between specialties and countries. Assessment themes neglected by individual surgical specialties and assessment themes that need validated assessment tools were identified.

CONCLUSIONS

This review highlights the low quality of evidence and fragmented efforts to develop and optimize surgical assessments. The minor differences observed demonstrate a common approach, globally and across specialties, related to surgical assessments. A paradigm shift in assessment approaches, which will require national and international collaboration, is required to optimize design and validation so that a comprehensive assessment of surgical competence can be implemented.