Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

A view from the top: A qualitative exploration of top-level health care leaders' perceptions on the implementation of simulation-based education in postgraduate medical education

INTRODUCTION

Although there is substantial evidence supporting the benefits of simulation-based education (SBE), its widespread and effective implementation remains challenging. The aim of this study was to explore the perceptions of top-level health care leaders regarding SBE and the barriers and facilitators that influence its wide implementation as part of the postgraduate surgical curricula in Denmark.

METHODS

We conducted semi-structured interviews with top-level health care leaders who were chosen based on their roles in ensuring high-quality patient care delivery and developing strategies to achieve the goals of the entire health care system. The interview transcripts were translated into English, and a thematic approach was used to code and inductively analyse the data. We used the Consolidated Framework for Implementation Research to identify and understand the determinants to a successful implementation of SBE.

RESULTS

We interviewed 13 participants from different political and administrative levels. We found that the participants had limited knowledge about SBE, which highlighted a disconnection between these leaders and the educational environment. This was further compounded by a lack of effective communication and inadequate information dissemination between simulation centres and higher-level organisations. While participants recognised the benefits of SBE for doctors in training, they expressed concerns about the implementation given the already strained health care system and limited resources. The need for evidence, particularly in the context of patient safety, was emphasised to facilitate SBE implementation. Although participants supported the implementation of SBE, it was unclear who should initiate action.

DISCUSSION

This study highlighted the perspectives of top-level health care leaders regarding SBE and identified the determinant factors for a successful implementation. Effective communication channels are crucial to enhance collaborations and reduce the disconnection between the different health care organisational levels. Strategic implementation processes, including the roles and responsibilities, should be defined and established. These will inform decisions regarding the implementation strategies to effectively integrate SBE into the residency training curricula.

Editor's Choice -- European Society for Vascular Surgery (ESVS) 2024 Clinical Practice Guidelines on the Management of Abdominal Aorto-Iliac Artery Aneurysms

OBJECTIVE

The European Society for Vascular Surgery (ESVS) has developed clinical practice guidelines for the care of patients with aneurysms of the abdominal aorta and iliac arteries in succession to the 2011 and 2019 versions, with the aim of assisting physicians and patients in selecting the best management strategy.

METHODS

The guideline is based on scientific evidence completed with expert opinion on the matter. By summarising and evaluating the best available evidence, recommendations for the evaluation and treatment of patients have been formulated. The recommendations are graded according to a modified European Society of Cardiology grading system, where the strength (class) of each recommendation is graded from I to III and the letters A to C mark the level of evidence.

RESULTS

A total of 160 recommendations have been issued on the following topics: Service standards, including surgical volume and training; Epidemiology, diagnosis, and screening; Management of patients with small abdominal aortic aneurysm (AAA), including surveillance, cardiovascular risk reduction, and indication for repair; Elective AAA repair, including operative risk assessment, open and endovascular repair, and early complications; Ruptured and symptomatic AAA, including peri-operative management, such as permissive hypotension and use of aortic occlusion balloon, open and endovascular repair, and early complications, such as abdominal compartment syndrome and colonic ischaemia; Long term outcome and follow up after AAA repair, including graft infection, endoleaks and follow up routines; Management of complex AAA, including open and endovascular repair; Management of iliac artery aneurysm, including indication for repair and open and endovascular repair; and Miscellaneous aortic problems, including mycotic, inflammatory, and saccular aortic aneurysm. In addition, Shared decision making is being addressed, with supporting information for patients, and Unresolved issues are discussed.

CONCLUSION

The ESVS Clinical Practice Guidelines provide the most comprehensive, up to date, and unbiased advice to clinicians and patients on the management of abdominal aorto-iliac artery aneurysms.

Operation time and clinical outcomes for open infrarenal abdominal aortic aneurysms to remain stable in the endovascular era

Objective

Endovascular aortic repair (EVAR) has become a routine procedure worldwide. Ultimately, the increasing number of EVAR cases entails changing conditions for open surgical repair (OSR) regarding patient selection, complexity, and surgical volume. This study aimed to assess the time trends of open abdominal aortic aneurysm (AAA) repair in a high-volume single center in Austria over a period of 20 years, focusing on the operation time and clinical outcomes.

Materials and methods

A retrospective analysis of all patients treated for infrarenal AAAs with OSR or EVAR between January 2000 and December 2019 was performed. Infrarenal AAA was defined as the presence of a >10-mm aortic neck. Cases with ruptured or juxtarenal AAAs were excluded from the analysis. Two cohorts of patients treated with OSR at different time periods, namely, 2000-2009 and 2010-2019, were assessed regarding demographical and procedure details and clinical outcomes. The time periods were defined based on the increasing single-center trend toward the EVAR approach from 2010 onward.

Results

A total of 743 OSR and 766 EVAR procedures were performed. Of OSR cases, 589 were infrarenal AAAs. Over time, the EVAR to OSR ratio was stable at around 50:50 (p = 0.488). After 2010, history of coronary arterial bypass (13.4% vs. 7.2%, p = 0.027), coronary artery disease (38.1% vs. 25.1%, p = 0.004), peripheral vascular disease (35.1% vs. 21.3%, p = 0.001), and smoking (61.6% vs. 34.3%, p < 0.001) decreased significantly. Age decreased from 68 to 66 years (p = 0.023). The operation time for OSR remained stable (215 vs. 225 min, first vs. second time period, respectively, p = 0.354). The intraoperative (5.8% vs. 7.2%, p = 0.502) and postoperative (18.3% vs. 20.8%, p = 0.479) complication rates also remained stable. The 30-day mortality rate did not change over both time periods (3.0% vs. 2.4%, p = 0.666).

Conclusion

Balanced EVAR to OSR ratio, similar complexity of cases, and volume over the two decades in OSR showed stable OSR time without compromise in clinical outcomes.

Facilitators and Barriers to Implementation of Simulation Based Education in Vascular Surgery in Europe

OBJECTIVE

This study explored the status and availability of simulation based education (SBE) for learning vascular surgical procedures identified in the 2019 General Needs Assessment in vascular surgery in Europe (GNA-2019) and identified facilitators and barriers to SBE implementation in vascular surgery.

METHODS

A three round iterative survey was distributed via the European Society for Vascular Surgery and the Union Européenne des Médecins Spécialistes. Members from leading committees and organisations within the European vascular surgical community were invited to participate as key opinion leaders (KOLs). Three online survey rounds explored demographics, SBE availability, and facilitators and barriers to SBE implementation.

RESULTS

Overall, 147 KOLs (target population 338) accepted invitation to round 1, representing 30 European countries. The dropout rates for rounds 2 and 3 were 29% and 40%, respectively. Most respondents (88%) were senior, consultant level or higher. No mandatory SBE training was required in their department before training on patients, according to 84% of the KOLs. There was high consensus on the need for structured SBE (87%) and mandatory SBE (81%). SBE is available for the top three prioritised procedures in GNA-2019 (basic open skills, basic endovascular skills, and vascular imaging interpretation) in 24, 23, and 20 of the 30 represented European countries, respectively. The highest ranking facilitators were structured SBE programmes, availability of simulation equipment locally and regionally, good quality simulators, and having a dedicated person running the SBE. The highest ranked barriers were lack of structured SBE curriculum, equipment costs, lack of SBE culture, no or limited dedicated time for faculty to teach in SBE, and clinical work overload.

CONCLUSION

Based largely on the opinions of KOLs in vascular surgery in Europe, this study revealed that SBE is needed in vascular surgery training and that systematic and structured programmes are required to ensure successful implementation.

A Systematic Review of Simulation-Based Training in Vascular Surgery

Introduction

Recent advancements in surgical technology, reduced working hours, and training opportunities exacerbated by the COVID-19 pandemic have led to an increase in simulation-based training. Furthermore, a rise in endovascular procedures has led to a requirement for high-fidelity simulators that offer comprehensive feedback. This review aims to identify vascular surgery simulation models and assess their validity and levels of effectiveness (LoE) for each model in order to successfully implement them into current training curricula.

Methods

PubMed and EMBASE were searched on January 1, 2021, for full-text English studies on vascular surgery simulators. Eligible articles were given validity ratings based on Messick’s modern concept of validity alongside an LoE score according to McGaghie’s translational outcomes.

Results

Overall 76 eligible articles validated 34 vascular surgery simulators and training courses for open and endovascular procedures. High validity ratings were achieved across studies for: content (35), response processes (12), the internal structure (5), relations to other variables (57), and consequences (2). Only seven studies achieved an LoE greater than 3/5. Overall, ANGIO Mentor was the most highly validated and effective simulator and was the only simulator to achieve an LoE of 5/5.

Conclusions

Simulation-based training in vascular surgery is a continuously developing field with exciting future prospects, demonstrated by the vast number of models and training courses. To effectively integrate simulation models into current vascular surgery curricula and assessments, there is a need for studies to look at trainee skill retention over a longer period of time. A more detailed discussion on cost-effectiveness is also needed.

Can We Still Teach Open Repair of Abdominal Aortic Aneurysm in The Endovascular Era? Single-Center Analysis on The Evolution of Procedural Characteristics Over 15 Years

OBJECTIVE

In many vascular centers an endovascular first policy for the treatment of abdominal aortic aneurysms (AAA) has resulted in endovascular aortic repair (EVAR) outnumbering open aortic repair (OAR). The declining routine in OAR raises the question whether this might influence procedural outcomes and diminish surgical expertise for current and future vascular surgeons. We aimed to analyze OAR outcomes, AAA morphology and procedural details over the past 15 years while an endovascular first approach was successively implemented.

PARTICICPANTS AND DESIGN

All patients operated for (i)ntact infra-/juxtarenal AAA between January 1, 2005 and December 31, 2019 were identified. Outcome parameters were length of stay (hospital/ICU), in-hospital mortality and medical/surgical complications. Operative details were clamping zone, access and graft configuration. AAA anatomy including neck and iliac parameters was analyzed with Endosize©. Logistic regression, uni- and multivariate analysis were applied.

RESULTS

293 patients received elective OAR for iAAA. Baseline characteristics (age, sex, hypertension, smoking, occlusive disease, coronary disease, hyperlipidemia, diabetes, renal insufficiency and obesity) did not change over time. The number of OAR dropped significantly (-0.5 cases/year p = 0.02). The procedure time (2005-2007: 192.2 ± 87.5min to 2017-2019: 235.6 ± 88.2min; p = 0.0001) and the length of stay (2005-2007: 12.0 ± 7.9 to 2017-2019: 17.0 ± 23.1; p = 0.03) increased significantly, whereas the in-hospital mortality, length of ICU stay and complication rates didn't, nor did AAA anatomy. Upon multivariate analysis, annual number of OAR and any additional anastomosis significantly influenced procedure time, trainee involvement, for example, did not. Hospital length-of-stay depended on patient age (p = 0.002), complication rates (p < 0.0001) and procedure time (p = 0.006).

CONCLUSION

Mortality and complication rates for OAR have remained low and constant. With the increase of EVAR, the absolute number of OARs has decreased significantly. However, the total procedure time has increased and depends significantly on the annual number of OARs in total and per surgeon. This might influence outcome parameters and should be implanted in future surgical education.

Validity Evidence for Procedure-Specific Competency Assessment Tools in Cardiovascular and Thoracic Surgery: A Scoping Review

OBJECTIVE

Surgical education has shifted from a time-based approach to the achievement and demonstration of procedural competency. High quality, objective assessment instruments are required to support this new approach. This study comprehensively reviewed the literature to identify and evaluate available procedure-specific assessment instruments in cardiothoracic and vascular surgery.

DESIGN

A systematic search of 8 databases identified studies containing procedure-specific operative assessment instruments in cardiothoracic and vascular surgery. Generic global rating scales were excluded, unless modified to be procedure-specific. Two reviewers independently evaluated the validity evidence, methodological rigour and educational utility of each instrument using objective scoring criteria. Validity evidence was evaluated with a scoring tool aligned with the contemporary framework of validity. Methodological rigour was evaluated using the Medical Education Research Study Quality Instrument. Educational utility was evaluated according to the Accreditation Council for Graduate Medical Education (ACGME) framework.

RESULTS

There were 2130 unique studies describing procedure-specific assessment in surgery. Of these, 9 studies evaluating 8 procedure-specific assessment instruments met inclusion criteria for cardiothoracic and vascular surgery. Four instruments were identified in thoracic surgery, 2 in cardiac surgery, and 2 in vascular surgery. Only 1 instrument was designed to evaluate surgeon performance, with the remainder designed to evaluate residents. No single instrument scored the maximum score of 15 for validity evidence. The highest score was 11, with 62.5% (n = 5) of instruments scoring greater than 10. All tools attained high scores in content validity, with minimal evidence generally presented regarding the consequences of assessment using a particular instrument. All but 1 instrument scored greater than 11 out of a maximum 16.5 points for methodological rigour. Very few studies reported on the ACGME domains of educational utility.

CONCLUSIONS

In an era where surgical education is shifting towards the demonstration of procedural competency, objective procedure-specific assessment is critical. This review identified that few procedure-specific assessment instruments in cardiothoracic and vascular surgery exist, emphasizing the need for such instruments to ensure the success of competency-based education models.

Ensuring competence in ultrasound-guided procedures-a validity study of a newly developed assessment tool

OBJECTIVES

To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score.

METHODS

Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method.

RESULTS

Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives.

CONCLUSIONS

Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making.

KEY POINTS

• A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification.

How often will midwives and obstetricians experience obstetric emergencies or high-risk deliveries: a national cross-sectional study

OBJECTIVE

To estimate how often midwives, specialty trainees and doctors specialised in obstetrics and gynaecology are attending to specific obstetric emergencies or high-risk deliveries (obstetric events).

DESIGN

A national cross-sectional study.

SETTING

All hospital labour wards in Denmark.

PARTICIPANTS

Midwives (n=1303), specialty trainees (n=179) and doctors specialised in obstetrics and gynaecology (n=343) working in hospital labour wards (n=21) in Denmark in 2018.

METHODS

Categories of obstetric events comprised of Apgar score <7/5 min, eclampsia, emergency caesarean sections, severe postpartum haemorrhage, shoulder dystocia, umbilical cord prolapse, vaginal breech deliveries, vaginal twin deliveries and vacuum extraction. Data on number of healthcare professionals were obtained through the Danish maternity wards, the Danish Health Authority and the Danish Society of Obstetricians and Gynaecologists. We calculated the time interval between attending each obstetric event by dividing the number of events occurred with the number of healthcare professionals.

OUTCOME MEASURES

The time interval between attending a specific obstetric event.

RESULTS

The average time between experiencing obstetric events ranged from days to years. Emergency caesarean sections, which occur relatively frequent, were attended on average every other month by midwives, every 9 days for specialty trainees and every 17 days by specialist doctors. On average, rare events like eclampsia were experienced by midwives only every 42 years, every 6 years by specialty trainees and every 11 years by specialist doctors.

CONCLUSIONS

Some obstetric events occur extremely rarely, hindering the ability to obtain and maintain the clinical skills to manage them through clinical practice alone. By assessing the frequency of a healthcare professionals attending an obstetric emergency, our study contributes to assessing the need for supplementary educational initiatives and interventions to learn and maintain clinical skills.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Identifying technical skills and clinical procedures in surgery for a simulation-based curriculum: a national general needs assessment

BACKGROUND

The efficacy of simulation-based training in surgical education is well known. However, the development of training programs should start with problem identification and a general needs assessment to ensure that the content is aligned with current surgical trainee needs. The objective of the present study is to identify the technical skills and clinical procedures that should be included in a simulation-based curriculum in general surgery.

METHODS

A national, three-round Delphi process was conducted to achieve consensus on which technical skills and clinical procedures should be included in a simulation-based curriculum in general surgery. In total, 87 key opinion leaders were identified and invited to the study.

RESULTS

Round 1 of the Delphi process had a response rate of 64% (56/87) and a total of 245 suggestions. Based on these suggestions, a consolidated list of 51 technical skills or clinical procedures was made. The response rate in Delphi round 2 was 62% (54/87) resulting in a pre-prioritized order of procedures for round 3. The response rate in Delphi round 3 was 65% (35/54). The final list included 13 technical skills and clinical procedures. Training was predominantly requested within general open surgical skills, laparoscopic skills, and endoscopic skills, and a few specific procedures such as appendectomy and cholecystectomy were included in the final prioritized list.

CONCLUSION

Based on the Delphi process 13 technical skills and clinical procedures were included in the final prioritized list, which can serve as a point of departure when developing simulation-based training in surgery.

Simulation Based Training and Assessment in Open Vascular Surgery: A Systematic Review

OBJECTIVE

The aim of this study was to systematically review the literature and give evidence based recommendations for future initiatives for simulation based training (SBT) and assessment in open vascular surgery.

DATA SOURCES

PubMed, Embase, and the Cochrane Library.

REVIEW METHODS

A systematic review of PubMed, Embase, and the Cochrane Library was performed, with the last search on 31 March 2020, to identify studies describing SBT and assessment in open vascular surgery. Kirkpatrick's levels for efficacy of training were evaluated. Validity evidence for assessment tools was evaluated according to the recommended contemporary framework by Messick.

RESULTS

Of 2 844 studies, 51 were included for data extraction. A high degree of heterogeneity in reporting standards and varying types of simulation was found. Vascular anastomosis was the most frequently simulated technical skill (43%). Assessment was mostly carried out using the Objective Structured Assessment of Technical Skills (55%). Validity evidence for assessment tools was found using outdated frameworks, and only one study used Messick's framework. Self directed training is valuable, the low trainer to trainee ratio is important to maximise efficiency, and experienced vascular surgeons are the most effective trainers.

CONCLUSION

Carefully designed and structured SBT is effective and can improve technical skills, especially in less experienced trainees. However, the supporting evidence lacks homogeneity in the reporting standards and types of simulations. Pass/fail standards that support proficiency based learning and studies investigating skills transfer should be the focus in future studies. Validity evidence of assessment tools needs to be addressed using contemporary frameworks.

Learning Curves and Competences of Vascular Trainees Performing Open Aortic Repair in a Simulation-Based Environment

BACKGROUND

The aim of this study was to analyze learning curves and competency gains of novice vascular trainees when performing open aortic repair in a simulation-based environment.

METHODS

This was a prospective study of 16 vascular trainees performing infrarenal open aortic repair on an inanimate abdominal aortic aneurysm simulator with pulsatile pressure and flow. Each participant performed 4 procedures as a primary surgeon while getting structured feedback by a supervising experienced vascular surgeon. All sessions were video recorded and were anonymously and independently assessed by 3 rater-trained experts on an online platform using the newly validated open abdominal aortic aneurysm repair of technical expertise assessment tool. All supervisor interferences and procedure time was noted.

RESULTS

Reliability between raters was excellent (intraclass correlation coefficient = 0.92). Participants' mean scores almost doubled during the course between the first (13.4, 95% confidence interval [CI], 6.8-20) and fourth session (29.8, 95% CI, 26.3-33.3) with a mean difference of 14.6 (P < 0.001). Supervisor interference also decreased significantly from mean 3.0 (95% CI, 1.5-3.6) in the first to 0.7 (95% CI, 0.4-1.0) in the fourth session (P = 0.004). Procedure time decreased with a mean of 24 minutes: from 81 min (95% CI, 71.8-90.3) to 57 min (95% CI, 51.1-63.2, P < 0.001). There was a significant negative correlation between procedure time and the Open Abdominal Aortic Aneurysm Repair of Technical Expertise score (Pearson's r = -0.72, P < 0.01). Only half of the participants passed the pass/fail score of 27.7 points during the course.

CONCLUSIONS

Novice vascular trainees achieve skills and competencies in open aortic repair in a simulated setting with dedicated supervision and feedback and can become ready for supervised surgery on real patients. Learning rates are individual, and it is important to construct training programs with emphasis on proficiency and not merely attending a course.