Influencing the national training agenda

Assessing Operative Skill in the Competency-based Education Era: Lessons From the UK and Ireland

INTRODUCTION

Decisions regarding the operative competence of surgical residents in the United Kingdom and Ireland are informed by operative workplace-based assessments (WBAs) and operative number targets for index procedures. This review seeks to outline the validity evidence of these assessment methods.

METHODS

A review of the MEDLINE (Pubmed), EMBASE and Cochrane Library databases was undertaken in accordance with the Joanna Briggs Institute Protocol for Scoping Reviews (2020). Articles were included if they provided evidence of the validity of procedure-based assessments, direct observation of procedural skills, or indicative operative number targets. The educational impact of each article was evaluated using a modified Kirkpatrick model.

RESULTS

Twenty-eight articles outlining validity evidence of WBAs and operative number targets were synthesised by narrative review. Five studies documented users' views on current assessment methods (Kirkpatrick level 1). Two articles recorded changes in attitudes towards current operative assessments (level 2a). Ten studies documented the ability of current assessments to record improvements in operative competence (level 2b). Ten studies measured a change in behaviour as a result of the introduction of these assessments (level 3). One article studied the ability of operative assessments to predict clinical outcomes (level 4b).

CONCLUSIONS

Operative WBAs are reliable. Scores achieved correlate with both time spent in training and recorded operative experience. Trainers and residents have concerns regarding the subjectivity of these assessments and the opportunistic nature in which they are used. Operative number targets are not criterion-referenced, lack validity evidence, and may be set too low to ensure operative competence.

Differences in progression by surgical specialty: a national cohort study

The aim of surgical training across the 10 surgical specialties is to produce competent day 1 consultants. Progression through training in the UK is assessed by the Annual Review of Competency Progression (ARCP).

OBJECTIVE

This study aimed to examine variation in ARCP outcomes within surgical training and identify differences in outcomes between specialties.

DESIGN

A national cohort study using data from the UK Medical Education Database was performed. ARCP outcome was the primary outcome measure. Multilevel ordinal regression analyses were performed, with ARCP outcomes nested within trainees.

PARTICIPANTS

Higher surgical trainees (ST3-ST8) from nine UK surgical specialties were included (vascular surgery was excluded due to insufficient data). All surgical trainees across the UK with an ARCP outcome between 2010 and 2017 were included.

RESULTS

Eight thousand two hundred and twenty trainees with an ARCP outcome awarded between 2010 and 2017 were included, comprising 31 788 ARCP outcomes. There was substantial variation in the proportion of non-standard outcomes recorded across specialties with general surgery trainees having the highest proportion of non-standard outcomes (22.5%) and urology trainees the fewest (12.4%). After adjustment, general surgery trainees were 1.3 times more likely to receive a non-standard ARCP outcome compared with trainees in trauma and orthopaedics (T&O) (OR 1.33, 95% CI 1.21 to 1.45, p=0.001). Urology trainees were 36% less likely to receive a non-standard outcome compared with T&O trainees (OR 0.64, 95% CI 0.54 to 0.75, p<0.001). Female trainees and older age were associated with non-standard outcomes (OR 1.11, 95% CI 1.02 to 1.22, p=0.020; OR 1.04, 95% CI 1.03 to 1.05, p<0.001).

CONCLUSION

There is wide variation in the training outcome assessments across surgical specialties. General surgery has higher rates of non-standard outcomes compared with other surgical specialties. Across all specialties, female sex and older age were associated with non-standard outcomes.

An international comparison of competency-based orthopaedic curricula and minimum operative experience - Review article

INTRODUCTION

Orthopaedic training has undergone considerable changes in the last few decades. Although structured training pathways exist in most countries, the requirements for completion of training are remarkably different. This review aims to assess key differences among orthopaedic curricula in selected high-income countries with well-established orthopaedic training programmes, focusing on their criteria for assessing technical competence prior to completion of training.

METHODS

Current orthopaedic training curricula published by the relevant accrediting bodies in the UK, USA, Canada, Australia, Germany and the European Union were reviewed. Data extracted included specified training duration, minimum or desirable operative experience requirements, methods and timing of in-training assessments.

RESULTS

The overall training duration ranged between 9 and 10 years in the UK and Australia, compared to 5-6 years in all other countries. While operative logbook was an essential component of formative and end-of-training reviews in all countries, minimum indicative numbers in index operations were a requirement only in the UK (minimum total required; 1800, index operations; 365) and USA (minimum total required; 1000, index operations; 455). On average, USA residents performed 1700 procedures compared to German residents performing 730 procedures before completion of training.

CONCLUSION

There is a lack of robust data describing the operative experiences of orthopaedic trainees outside of the UK and USA. UK training is the longest among countries compared in this review and also sets the highest standards in minimum operative experience requirements. Based on the evidence available, surgeons exiting training and entering independent practice in the above countries are not trained to the same minimum standard.

Changing Autonomy in Operative Experience Through UK General Surgery Training: A National Cohort Study

Supplemental Digital Content is available in the text

Objectives:

To determine the operative experience of UK general surgery trainees and assess the changing procedural supervision and acquisition of competency assessments through the course of training.

Background:

Competency assessment is changing with concepts of trainee autonomy decisions (termed entrustment decisions) being introduced to surgical training.

Methods:

Data from the Intercollegiate Surgical Curriculum Programme and the eLogbook databases for all UK General Surgery trainees registered from August 1, 2007 who had completed training were used. Total and index procedures (IP) were counted and variation by year of training assessed. Recorded supervision codes and competency assessment outcomes for IPs were assessed by year of training.

Results:

We identified 311 trainees with complete data. Appendicectomy was the most frequently undertaken IP during first year of training [mean procedures (mp) = 26] and emergency laparotomy during final year of training (mp = 27). The proportion of all IPs recorded as unsupervised increased through training (P < 0.05) and varied between IPs with 91.2% of appendicectomies (mp = 20), 40.6% of emergency laparotomies (mp = 27), and 17.4% of segmental colectomies (mp = 15) recorded as unsupervised during the final year of training. Acquisition of competency assessments increased through training and varied by IP.

Conclusions:

The changing autonomy of trainees through the course of an entire training scheme, alongside formal competency assessments, may provide evidence of changing entrustment decisions made by trainers for different key procedures. Other countries utilizing electronic logbooks could adopt similar techniques to further understanding of competency attainment amongst their surgical trainees.

Does Orthopaedic Training Compromise the Outcome in Knee Joint Arthroplasty?

OBJECTIVE

This study investigates knee joint arthroplasty and compares the outcomes between attending (consultant) orthopedic surgeons and resident (trainee) surgeons.

DESIGN

Retrospective review and comparison of knee joint arthroplasty outcomes between 4 surgeon groups (attending, supervised senior and junior residents, and unsupervised senior residents). Measured outcomes were implant survival (revision rate) and patient reported functional outcomes, measured by Oxford knee score (OKS).

SETTING

New Zealand arthroplasty service.

PARTICIPANTS

Seventeen years of knee joint arthroplasty data from the New Zealand Joint Registry (NZJR) was reviewed.

RESULTS

The New Zealand Joint Registry (NZJR) data showed 79,671 total knee arthroplasties (TKA) and 8854 unicompartmental knee arthroplasties (UKA) performed between 1999 and 2016. Attending surgeons performed 90% and 97% of TKA and UKA, respectively. The number and proportion of resident performed knee joint arthroplasty has decreased. Faster operation times was observed in the attending surgeon group. Attending surgeon revision rate was 0.49 and 1.19/100 component years for TKA and UKA, respectively, this was not significantly increased in resident surgeon groups. Postoperative OKS was 37.7 and 39.7 for attending surgeon performed TKA and UKA, respectively. Mean OKS were less than 2 points worse in resident groups (resident range: 36.3-36.9) compared to attending colleagues for TKA, but for UKA scores were up to 11 points worse (resident range: 28.9-38.8).

CONCLUSIONS

New Zealand has a high rate of attending surgeon performed TKA and UKA. Revision rates were not increased in resident surgeon groups. Postoperative function was not reduced by a clinically significant amount in TKA in any of the resident surgeon groups but was reduced in supervised junior resident and unsupervised senior resident surgeon groups for UKA.

What are the essential features of a successful surgical registry? a systematic review

OBJECTIVE

The regulation of surgical implants is vital to patient safety, and there is an international drive to establish registries for all implants. Hearing loss is an area of unmet need, and industry is targeting this field with a growing range of surgically implanted hearing devices. Currently, there is no comprehensive UK registry capturing data on these devices; in its absence, it is difficult to monitor safety, practices and effectiveness. A solution is developing a national registry of all auditory implants. However, developing and maintaining a registry faces considerable challenges. In this systematic review, we aimed to identify the essential features of a successful surgical registry.

METHODS

A systematic literature review was performed adhering to Preferred Reporting Items for Systematic Review and Meta-Analysis recommendations. A comprehensive search of the Medline and Embase databases was conducted in November 2016 using the Ovid Portal. Inclusion criteria were: publications describing the design, development, critical analysis or current status of a national surgical registry. All registry names identified in the screening process were noted and searched in the grey literature. Available national registry reports were reviewed from registry websites. Data were extracted using a data extraction table developed by thematic analysis. Extracted data were synthesised into a structured narrative.

RESULTS

Sixty-nine publications were included. The fundamentals to successful registry development include: steering committee to lead and oversee the registry; clear registry objectives; planning for initial and long-term funding; strategic national collaborations among key stakeholders; dedicated registry management team; consensus meetings to agree registry dataset; established data processing systems; anticipating challenges; and implementing strategies to increase data completion. Patient involvement and awareness of legal factors should occur throughout the development process.

CONCLUSIONS

This systematic review provides robust knowledge that can be used to inform the successful development of any UK surgical registry. It also provides a methodological framework for international surgical registry development.

Learning Curves in Arthroplasty in Orthopedic Trainees

The NHS is adapting to a changing environment, in which economical constraints have forced theatres to maximise efficiency. An environment in which working hours and surgical exposure has been reduced and outcomes are being published. Litigation is high, and patients are living longer with higher demands. We ask, will traditional methods of apprentiship type training suffice in producing competent arthroplasty surgeons when hands on experience is falling. We review learning curves and assessment tools available to accurately assess competency and support trainee orthopaedic surgeons in their acquisition of surgical proficiency.

Procedure-based assessments in trauma and orthopaedic training--The trainees' perspective

INTRODUCTION

The study aimed to gain an understanding of the attitudes of trauma and orthopaedic (T&O) trainees regarding procedure-based assessments (PBAs) and identify factors that influence any perceived educational benefit.

METHODS AND MATERIALS

A questionnaire was emailed to all T&O trainees in the UK via an established e-mail communication tool after an initial pilot exercise. The data were analysed using the online survey software.

RESULTS

Of the 616 trainees included 53% found PBAs useful as a learning tool for delivery of feedback. Trainees agreed that there were barriers to the successful use of PBAs (61%). Completing the PBA at the time of the procedure (p < 0.001) and the trainer delivering quality feedback with PBAs (p < 0.001) significantly increased the number of trainees perceiving an improvement in their practice. Completing higher numbers of PBAs did not have this effect (p = 0.26). There was wide geographical variation in the use of PBAs by trainees.

CONCLUSIONS

This is the first nationwide study offering a deeper insight into factors influencing T&O trainees' perceptions of the educational benefit gained from using PBAs. This study informs the debate on how to improve the effective use of PBAs in T&O training, and generally, of workplace-based assessments in surgical training.

Expert practical operative skills teaching in Trauma and Orthopaedics at a nominal cost

BACKGROUND AND PURPOSE

The AO Foundation Operative Fracture Management course is the gold standard in training courses currently available for trainees at ST3 level. We have devised a low cost operative skills course comprising instructional lectures, demonstrations and practical dry bone workshops. To assess the quality of teaching, candidates' feedback was analysed in two cohorts for the running of the course over two consecutive years: 2008 and 2009.

METHODS

Trainees were given short instructional lectures by consultant surgeons followed by workshops, with a trainer to candidate ratio of 1:4. A trauma inventory was provided by Stryker Trauma UK, ensuring a nominal fee for each candidate (£50). Feedback was anonymously collected according to a Likert scale and analysed using non-parametric methods appropriate for ranked data.

MAIN FINDINGS

Twenty one of 22 (95%) candidates gave feedback in 2008 and 18 out of 18 candidates (100%) in 2009. The teaching provided was highly rated consistently for both years, apart from an informal session on theatre tips and tricks in 2008. This was not repeated in 2009 to allow more practical time. Only one session, an intramedullary nailing lecture, had a significant difference in scores between the 2 years (p = 0.044) because of improved scores in 2009.

CONCLUSIONS

Due to changes in training, trainees have reduced exposure in theatre and this has implications for the early stages of acquiring practical operative skills. As an adjunct to the AO course, practical skills teaching by consultants in the format of a low cost skills workshop outside of a theatre environment can be achieved with support from a trauma implant supplier.

Analysing the operative experience of basic surgical trainees in Ireland using a web-based logbook

BACKGROUND

There is concern about the adequacy of operative exposure in surgical training programmes, in the context of changing work practices. We aimed to quantify the operative exposure of all trainees on the National Basic Surgical Training (BST) programme in Ireland and compare the results with arbitrary training targets.

METHODS

Retrospective analysis of data obtained from a web-based logbook (http://www.elogbook.org) for all general surgery and orthopaedic training posts between July 2007 and June 2009.

RESULTS

104 trainees recorded 23,918 operations between two 6-month general surgery posts. The most common general surgery operation performed was simple skin excision with trainees performing an average of 19.7 (± 9.9) over the 2-year training programme. Trainees most frequently assisted with cholecystectomy with an average of 16.0 (± 11.0) per trainee. Comparison of trainee operative experience to arbitrary training targets found that 2-38% of trainees achieved the targets for 9 emergency index operations and 24-90% of trainees achieved the targets for 8 index elective operations. 72 trainees also completed a 6-month post in orthopaedics and recorded 7,551 operations. The most common orthopaedic operation that trainees performed was removal of metal, with an average of 2.90 (± 3.27) per trainee. The most common orthopaedic operation that trainees assisted with was total hip replacement, with an average of 10.46 (± 6.21) per trainee.

CONCLUSIONS

A centralised web-based logbook provides valuable data to analyse training programme performance. Analysis of logbooks raises concerns about operative experience at junior trainee level. The provision of adequate operative exposure for trainees should be a key performance indicator for training programmes.

Skill sets and competencies for the modern military surgeon: lessons from UK military operations in Southern Afghanistan

INTRODUCTION

British military forces remain heavily committed on combat operations overseas. UK military operations in Afghanistan (Operation HERRICK) are currently supported by a surgical facility at Camp Bastion, in Helmand Province, in the south of the country. There have been no large published series of surgical workload on Operation HERRICK. The aim of this study is to evaluate this information in order to determine the appropriate skill set for the modern military surgical team.

METHOD

A retrospective analysis of operating theatre records between 1st May 2006 and 1st May 2008 was performed. Data was collated on a monthly basis and included patient demographics, operation type and time of operation.

RESULTS

During the study period 1668 cases required 2210 procedures. Thirty-two per cent were coalition forces (ISAF), 27% were Afghan security forces (ANSF) and 39% were civilians. Paediatric casualties accounted for 14.7% of all cases. Ninety-three per cent of cases were secondary to battle injury and of these 51.3% were emergencies. The breakdown of procedures, by specialty, was 66% (1463) orthopaedic, 21% (465) general surgery, 6% (139) head and neck, 5% (104) burns surgery and a further 4% (50) non-battle, non-emergency procedures. There was an almost twofold increase in surgical workload in the second year (1103 cases) compared to the first year of the deployment (565 cases, p<0.05).

DISCUSSION

Surgical workload over the study period has clearly increased markedly since the initial deployment of ISAF forces to Helmand Province. A 6-week deployment to Helmand Province currently provides an equivalent exposure to penetrating trauma as 3 years trauma experience in the UK NHS. The spectrum of injuries seen and the requisite skill set that the military surgeon must possess is outside that usually employed within the NHS. A number of different strategies; including the deployment of trainee specialist registrars to combat hospitals, more focused pre-deployment military surgery training courses, and wet-laboratory work are proposed to prepare for future generations of surgeons operating in conflict environments.

Assessing the surgeon's technical skills: analysis of the available tools

The concept of assessing competency in surgical practice is not new and has taken on an added urgency in view of the recent high-profile inquiries into "botched cases" involving surgeons of various levels in different parts of the world. Until very recently, surgeons in the United Kingdom and other parts of the world, although required to undergo formal and compulsory examinations to test their factual knowledge and decision making, were not required to demonstrate technical ability. Therefore, there existed (and still exist) no objective assessment criteria to test trainees' surgical skill, especially during the exit examination, which, if passed, provides unrestricted license to surgeons to practice their specialties. However, with the introduction of a new curriculum by various surgical societies and a demand from the lay community for better standards, new assessment tools are emerging that focus on technical competency and that could objectively and reliably measure surgical skills. Furthermore, training authorities and hospitals are keen to embrace these changes for satisfactory accreditation and reaccreditation processes and to assure the public of the safety of the public and private health care systems. In the United Kingdom, two new surgical tools (Surgical Direct Observation of Procedural Skill, and Procedure Based Assessments) have been simultaneously introduced to assess surgical trainees. The authors describe these two assessment methods, provide an overview of other assessment tools currently or previously used to assess surgical skills, critically analyze the two new assessment tools, and reflect on the merit of simultaneously introducing them.

Training in the practical application of damage control and early total care operative philosophy--perceptions of UK orthopaedic specialist trainees

INTRODUCTION

Despite improvements in the outcome of individuals sustaining significant injury, the optimum management of fractures in traumatised patients remains an area of debate and publication. There is, however, a paucity of studies regarding the specifics of acquired experience and training of junior orthopaedic surgeons in the practical application of these skills. Our null hypothesis is that, despite alteration in surgical training, the perceived confidence and adequacy of training of UK orthopaedic specialist trainees in the application of damage control orthopaedics (DCO) and early total care (ETC) philosophy is unaffected.

SUBJECTS AND METHODS

A web-based survey was sent to a sample of orthopaedic trainees. From 888 trainees, 222 responses were required to achieve a 5% error rate with 90% confidence.

RESULTS

A total of 232 responses were received. Trainees reported a high level of perceived confidence with both external fixation and intramedullary devices. Exposure to cases was sporadic although perceived training adequacy was high. A similar pattern was seen in perceived operative role with the majority of trainees expecting to be performing such operations, albeit under varying levels of supervision. In a more complicated case of spanning external fixation for a 'floating knee, trainees reported a decreased level of perceived confidence and limited exposure. One-third of trainees reported never having been involved in such a case. In contrast to nationally collated logbook data, exposure to and perceived confidence in managing cases involving ETC and DCO were similar.

CONCLUSIONS

Despite changes in the training of junior orthopaedic surgeons, trainee-reported confidence and adequacy of training in the practical application of DCO and ETC was high. Exposure to cases overall was, however, seen to be limited and there was a suggestion of disparity between current operative experiences of trainees and that recorded in the national trainee logbook.

Orthopaedic training worldwide: how they compare? A pilot study

INTRODUCTION

Advances in Orthopaedics and changing concepts should result in regular review and if necessary restructuring of training programmes periodically to accommodate these changes. However, most countries have orthopaedic training programmes which have remained unchanged over decades. There is very little information available in the literature concerning orthopaedic training programmes worldwide.

AIM

The aim of this study was to collect information concerning various training programmes worldwide for comparison. The top ten countries by population and all G8 countries were selected. A survey in the form of a questionnaire was disseminated to a selection of trainees and trainers from these countries and responses tabulated for analysis. Significant variation was evident in training structure, time spent on training and training environment.

CONCLUSION

There is a need for constant monitoring of training programmes worldwide to accommodate the ever-changing world of Orthopaedics which could potentially be of mutual benefit to trainees, trainers and the population at large of the countries involved.

Do Independent Sector Treatment Centres (ISTC) impact on specialist registrar training in primary hip and knee arthroplasty?

INTRODUCTION

The introduction of Modernising Medical Careers (MMC) is likely to reduce specialist registrar (SpR) operative experience during higher surgical training (HST). A further negative impact on training by local Independent Sector Treatment Centres (ISTCs) could reduce experience, and thus competence, in primary joint arthroplasty at completion of higher surgical training.

PATIENTS AND METHODS

Retrospective case note and radiograph analysis of patients receiving primary hip and knee arthroplasty in a teaching hospital, before and after the establishment of a local ISTC. Patients and operative details were recorded from the selected case notes. Corresponding radiographs were assessed and the severity of the disease process assessed.

RESULTS

Fewer primary hip and knee replacements were performed by SpRs in the time period after the establishment of an ISTC.

CONCLUSIONS

ISTCs may adversely affect SpR training in primary joint arthroplasty.

Trauma experience in the UK and Ireland: analysis of orthopaedic training using the FHI eLogbook

Surgical trainees routinely provide evidence of their training and operative experience for assessment. National comparative data on the number of procedures performed during training was previously unavailable in the UK. Since 2003 every trainee in Trauma and Orthopaedics (T&O) in the UK and Ireland has submitted data recording their operative experience electronically via the Faculty of Health Informatics (FHI) eLogbook. This provides detailed data on trainee, trainer, hospital and training programme performance. This data has been analysed for trauma surgery. By March 2007 there were 1053 T&O Specialist Registrars with operations logged. Trauma operations performed and uploaded during the previous 3 years have been included. Each trainee's work is analysed by 'year-in-training' (YIT, 1-6). Data on levels of supervision and potentially missed opportunities (where the trainee was present but assisted rather than performed the operation) was analysed. The mean number of trauma operations performed annually by trainees was 109, 120, 110, 122, 98 and 84 (total 643) for YIT 1-6, respectively. 22% of the operations at which a trainee was present were potentially missed opportunities. A high level of experience is gained in hip fracture surgery (121 operations performed), intramedullary nailing (38) and ankle (47) stabilisation over the 6 years of training. However, the mean number of tendon repairs (18), tension band wires (13), external fixators (12) and children's supracondylar fracture procedures (9) performed is low. We also report figures for complex fracture stabilisation. The eLogbook remains a powerful tool which can provide accurate information to support in-depth analysis of trainees, trainers, and training programmes. Based on this analysis, we suggest 'standard setting' to identify trainees performing fewer operations than required during their training. We have also established a baseline which can be used to identify the consequences of changes to length of training and working hours.