Assessing Operative Skill in the Competency-based Education Era: Lessons From the UK and Ireland

INTRODUCTION

Decisions regarding the operative competence of surgical residents in the United Kingdom and Ireland are informed by operative workplace-based assessments (WBAs) and operative number targets for index procedures. This review seeks to outline the validity evidence of these assessment methods.

METHODS

A review of the MEDLINE (Pubmed), EMBASE and Cochrane Library databases was undertaken in accordance with the Joanna Briggs Institute Protocol for Scoping Reviews (2020). Articles were included if they provided evidence of the validity of procedure-based assessments, direct observation of procedural skills, or indicative operative number targets. The educational impact of each article was evaluated using a modified Kirkpatrick model.

RESULTS

Twenty-eight articles outlining validity evidence of WBAs and operative number targets were synthesised by narrative review. Five studies documented users' views on current assessment methods (Kirkpatrick level 1). Two articles recorded changes in attitudes towards current operative assessments (level 2a). Ten studies documented the ability of current assessments to record improvements in operative competence (level 2b). Ten studies measured a change in behaviour as a result of the introduction of these assessments (level 3). One article studied the ability of operative assessments to predict clinical outcomes (level 4b).

CONCLUSIONS

Operative WBAs are reliable. Scores achieved correlate with both time spent in training and recorded operative experience. Trainers and residents have concerns regarding the subjectivity of these assessments and the opportunistic nature in which they are used. Operative number targets are not criterion-referenced, lack validity evidence, and may be set too low to ensure operative competence.

Use of work-based assessments in postgraduate medical education

Work-based assessments are ubiquitous in postgraduate medical training in the UK. This article discusses the variety of these assessments and explores barriers to their use and solutions for improving the educational value of these tools for adult learners. The focus should be on feedback and learning rather than assessment, and this may promote discussion of more challenging scenarios where the opportunity for learning is greater. Mobile devices may help reduce the administrative, geographical and time constraints of completing work-based assessments.

Adaptation of "Objective Structured Assessment of Technical Skills" for Adenotonsillectomy into Turkish: A Validity and Reliability Study

Objective

To adapt an objective surgical assessment tool (OSATS) for adenotonsillectomy/tonsillectomy to Turkish.

Methods

A prospective longitudinal validation study was planned. Adenoidectomy items were added to the tonsillectomy OSATS. Data were collected from 80 real-time observed surgical performances. To examine interrater reliability, 25 performances were simultaneously scored by two evaluators. In total, 105 test forms were filled, and reliability and validity studies were conducted.

Results

Regarding reliability, Cronbach's alpha values were found to be >0.95, indicating very high internal consistency. For interrater reliability, a statistically significant positive correlation was found between the two evaluators' task-specific checklist (TSC) (p=0.0001; r=0.969) and global rating checklist (GRC) (p=0.0001; r=0.957). Between the TSC and GRC, there was a statistically significant positive correlation (p=0.0001; r=0.951). A model was developed using the generalizability theory, and the highest variances belonged to surgical processes. Regarding validity, the trainers' scores were significantly higher than the trainees' scores (p=0.0001). As surgical experience increased, both the TSC and the GRC scores significantly increased (p=0.0001). A trainee's first-ever 34 adenotonsillectomy performances were scored. A scatter plot of the obtained data showed that the trainee had gained more scores as the number of operations increased.

Conclusion

The results indicate that adenotonsillectomy OSATS is a reliable and valid tool for the assessment of residents' surgical skills. This is the first OSATS developed for adenotonsillectomy in the Turkish literature. It is ready to be used in the operating room to provide constructive feedback and monitor the development of trainees' surgical skills in adenotonsillectomy.

Objective Assessment of Technical Skills in Otorhinolaryngology-Head and Neck Surgery Residents: A Systematic Review

Objectives The primary goal is the indexation of validated methods used to assess surgical competency in otorhinolaryngology-head and neck surgery (ORL-HNS) residents. Secondary goals include assessment of the reliability and validity of these tools, as well as the documentation of specific procedures in ORL-HNS involved. Data Sources MEDBASE, OVID, Medline, CINAHL, and EBM, as well as the printed references, available through the Université de Montréal library. Review Methods The PRISMA method was used to review digital and printed databases. Publications were reviewed by 2 independent reviewers, and selected articles were fully analyzed to classify evaluation methods and categorize them by procedure and subspecialty of ORL-HNS involved. Reliability and validity were assessed and scored for each assessment tool. Results Through the review of 30 studies, 5 evaluation methods were described and validated to assess the surgical competency of ORL-HNS residents. The evaluation method most often described was the combined Global Rating Scale and Task-Specific Checklist tool. Reliability and validity for this tool were overall high; however, considerable data were unavailable. Eleven distinctive surgical procedures were studied, encompassing many subspecialties of ORL-HNS: facial plastics, general ear-nose-throat, laryngology, otology, pediatrics, and rhinology. Conclusions Although assessment tools have been developed for an array of surgical procedures, involving most ORL-HNS subspecialties, the use of combined checklists has been repeatedly validated in the literature and shown to be easily applicable in practice. It has been applied to many ORL-HNS procedures but not in oncologic surgery to date.

Operative Experience vs. Competence: A Curriculum Concordance and Learning Curve Analysis

OBJECTIVE

Certification of completion of training in general surgery requires proof of competence of index operations by means of 3, level-4 consultant-validated procedural-based assessments. The aim of this study was to examine the relationship between index operative experience and competence.

DESIGN

Higher surgical trainee procedural-based assessments were compared with e-logbooks to determine the relationship between index operative experience and achievement of a third level 4 competence (L4C) related to the indicative procedures of emergency laparotomy (EL, target 100), Hartmann procedure (5), appendicectomy (80), segmental colectomy (20), laparoscopic cholecystectomy (50), and inguinal hernia (80).

SETTING

All trainees are from a single UK Deanery.

PARTICIPANTS

Consecutive 69 national training number higher surgical trainees were appointed to a single UK Deanery between 2007 and 2014.

RESULTS

EL L4C was achieved at a median of 76 (15-136) cases, Hartmann procedure L4C at 17 (7-27) cases (p = 0.009 vs. EL), appendicectomy L4C at 107 (20-206) cases, segmental colectomy L4C at 52 (15-131) cases, laparoscopic cholecystectomy L4C at 72 (40-197) cases, and inguinal hernia L4C at 64 (17-132) cases.

CONCLUSIONS

The learning curve and caseload required to demonstrate L4C related to specific procedure varied over 4-fold, from 0.76 to 3.4 times the national indicative target number guidance. Certification of completion of training operative logbook number targets should be reconsidered to better reflect the competencies demanded by the curriculum.