Validity of an Exam Assessment in Surgical Skill: EBSQ-VASC Pilot Study

The Impact of Simulator Training of Vascular Anastomosis and Video Assessment for Surgical Residents

BACKGROUND

The rapid increase of minimally invasive surgery and the shortened training period for surgical residents has resulted in limited opportunities to acquire proficiency in open surgical techniques, such as vascular anastomosis. However, vascular anastomosis remains an essential skill in every surgery for bleeding control. This study aimed to validate the effectiveness of surgical education model for vascular anastomosis and assess the impact on the comprehension, skill, and confidence of surgical residents in performing vascular anastomosis.

METHODS

A total of 21 surgical residents with first to third years of experience at Seoul National University Hospital participated in a 4-week vascular anastomosis training program. The program included an educational lecture and the performance of an end-to-side anastomosis on a procedural model, with evaluations being conducted using the Objective Structured Assessment of Technical Skills (OSATS) and the End-Product Rating Score (EPRS) in pretraining and posttraining surveys.

RESULTS

Significant improvement was observed in the OSATS score (from 9.22 ± 2.4 in week 1 to 12.87 ± 3.1 in week 4; P < 0.001) and the EPRS score (from 12.47 ± 4.1 in week 1 to 17.57 ± 2.2 in week 4; P < 0.001). Additionally, the surgical performance time significantly decreased from 20.99 ± 4.6 min to 16.33 ± 4.2 min (P = 0.019) CONCLUSIONS: Simulator training of in vitro vascular anastomosis, when accompanied by expert-led instruction, can effectively enhance the surgical proficiency, confidence, and overall surgical outcomes of residents, as inferred from the observed improvements in OSATS and EPRS scores. The results suggest that integration of this training model into surgical curricula could be a promising strategy for enhancing vascular surgical training.

Extrapolative Validity Evidence of the Anastomosis Objective Structured Assessment of Technical Skill (A-OSATS) for Robotic Ileocolic Anastomosis

OBJECTIVE

To collect validity evidence for the use of the Anastomosis Objective Structured Assessment of Technical Skills (A-OSATS) instrument, which has been developed to evaluate performance of a minimally invasive side-to-side bowel anastomosis with hand-sewn common enterotomy.

DESIGN

Residents performed a robotic ileocolic anastomosis simulation on an ex vivo porcine model. Faculty scored each resident with the A-OSATS and performed a provocative leak test on the completed anastomoses. Residents were reassessed on the sewing sub-score 1 month later. Data were compared with parametric and nonparametric analysis.

SETTING

Single academic general surgery residency PARTICIPANTS: PGY-4 and -5 general surgery residents (n = 17) RESULTS: PGY-5s performed better than PGY-4s in repeat A-OSATS sewing sub-score (mean 55/55 ± 0 vs 43 ± 4.9, p < 0.001) and time to complete (minutes, mean 14.5 ± 4.9 vs 21.2 ± 3.9, p = 0.01). There was a strong correlation between A-OSATS score and time (r = -0.67, p = 0.005). For the initial assessment, there was no significant difference in mean A-OSATS score between anastomoses that leaked and those that did not leak (137.3 ± 14.5 vs 150.1 ± 11.2, p = 0.098), but on repeat assessment, intact anastomoses had a higher mean A-OSATS sewing sub-score than those that leaked (52.2 ± 4.7 vs 39 ± 3.5, p = 0.007). There was no significant difference between initial A-OSATS score and repeat score (p = 0.14).

CONCLUSIONS

We provide extrapolative validity evidence for the A-OSATS instrument by comparing A-OSATS score to time to sew, provocative leak test, and discrimination between PGY-4s and PGY-5s. Generalizability validity evidence is provided by test-retest reliability. Further refinement is needed for the A-OSATS tool to be used for high-stakes entrustment decisions in resident-performed robotic ileocolic anastomoses.

Effectivity of Distance Learning in the Training of Basic Surgical Skills—A Randomized Controlled Trial

Background: Distance learning is an interactive way of education when teachers and students are physically separated. Our purpose was to examine its effectivity in training of basic surgical techniques and to provide an alternative sustainable methodology for the training of medical professionals. Methods: Sixty students were involved in our single blinded randomized controlled study. Six homogenized groups were created then randomized into three groups of distance learning and three groups of in-person teaching. The groups completed the same curriculum using our own “SkillBox”. All students took the same pre- and post-course test evaluated blindly. The students filled out an online feedback form after the course. A financial analysis was also made. Results: There was no significant difference in the post-course exam results (distance 28.200 vs. in-person 25.200). We managed to achieve significantly better improvements in the distance learning of suturing (distance 19.967 vs. in-person 15.900, p = 0.043). According to 93% of the study group students, the quality of teaching did not decrease compared to the traditional classes. Conclusion: The results of the students improved similarly in distance learning and in-person education. The online form of teaching was received positively among the students; they found it an effective and good alternative.

Evaluation of tools to assess operative competence in endovascular procedures: a systematic review

BACKGROUND

With an increase in the use of endovascular interventions as an alternative to open surgery and the unique technical skills required, current methods for assessing the competence of vascular surgery trainees may not be optimal, suggesting a need for a shift in assessment modalities. We conducted this systematic review to explore current assessment methods used in vascular surgery training to assess competence specific to endovascular procedures.

METHODS

A comprehensive literature search was performed with a structured search strategy using terms focusing on endovascular procedures and assessment. Inclusion and exclusion criteria were used in order to screen for suitable articles.

RESULTS

We identified 54 articles that satisfied the inclusion criteria. These included a single randomized controlled trial, a single systematic review, a single narrative review and a single literature review, with the vast majority having level 2 evidence. Global rating scales, proficiency assessments and written/oral examinations were described as standard current assessment tools. These modalities lack reproducibility and objectivity, neglecting the needs of assessment of endovascular procedures requiring specialized decision making and finger dexterity. Novel methods such as high fidelity simulation and virtual reality promote reproducible and objective assessment methods in the context of endovascular surgery, and have a promising future.

CONCLUSION

While current assessment methods in vascular surgery are widely supported the changing skills required of a vascular surgery trainee warrants a shift in assessment modalities to better align to these requirements. High fidelity simulations show promise, although they require more extensive research to understand their relative merits.

Assessment of Technical Skills in Axillary Lymph Node Dissection

OBJECTIVE

A simulator to enable safe practice and assessment of ALND has been designed, and face, content and construct validity has been investigated.

SUMMARY AND BACKGROUND DATA

The reduction in the number of ALNDs conducted has led to decreased resident exposure and confidence.

METHODS

A cross-sectional multicenter observational study was carried out between July 2017 and August 2018. Following model development, 30 surgeons of varying experience (n = "experts,' n = 11 "senior residents,' and n = 10 "junior residents") were asked to perform a simulated ALND. Face and content validity questionnaires were administered immediately after ALND. All ALND procedures were retrospectively assessed by 2 attending breast surgeons, blinded to operator identity, using a video-based assessment tool, and an end product assessment tool.

RESULTS

Statistically significant differences between groups were observed across all operative subphases on the axillary clearance assessment tool (P < 0.001). Significant differences between groups were observed for overall procedure quality (P < 0.05) and total number of lymph nodes harvested (P < 0.001). However, operator grade could not be distinguished across other end product variables such as axillary vein damage (P = 0.864) and long thoracic nerve injury (P = 0.094). Overall, participants indicated that the simulator has good anatomical (median score >7) and procedural realism (median score >7).

CONCLUSIONS

Video-based analysis demonstrates construct validity for ALND assessment. Given reduced ALND exposure, this simulation is a useful adjunct for both technical skills training and formative Deanery or Faculty administered assessments.

Ensuring Competency in Open Aortic Aneurysm Repair - Development and Validation of a New Assessment Tool

OBJECTIVE

The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard.

METHODS

Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score.

RESULTS

Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed.

CONCLUSION

Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt.

Development and evaluation of rhinoplasty spreader graft suture simulator for novice surgeons

OBJECTIVE

Surgical simulators aimed at mimicking elements of rhinoplasty surgery, specifically those aimed at improving cartilage suturing, are not available. Here, we present a surgical simulator for spreader graft placement that uses cartilage rather than synthetic materials and gauge improvement using objective measures for suture placement accuracy, speed, and efficiency of hand motion.

METHODS

Twenty-two otolaryngologists in two groups (residents [10] and experts [12]) were instructed to secure the two spreader graft specimen into position with three mattress sutures on a nose model that used porcine septal cartilage as a proxy for the human counterpart. Hand motion was tracked using an electromagnetic position sensing device. The time required to complete the suture task, total hand displacement, cumulative number of hand motion direction changes, and accuracy of suture insertion were measured. These measurements were compared between the two cohort groups for construct validity. The subjects completed a survey to evaluate realism and value of the model.

RESULTS

The expert group had a lower mean time required to complete the task (P < 0.05), total hand displacement (P < 0.01), and number of hand motion direction changes (P < 0.001). No significant difference was observed between the two groups in suture precision measurement. The subjects agreed on the face validity and usefulness of the trainer.

CONCLUSIONS

Our study suggests that the simulator may be a useful tool to objectively gauge suturing efficiency. Devices such as this may be useful for developing skill with suturing cartilage tissue and potentially be used to assess resident acquisition of surgical skill.

LEVEL OF EVIDENCE

NA Laryngoscope, 129:344-350, 2019.

Basic Endovascular Skills Trainer: A surgical simulator for the training of novice practitioners of endovascular procedures

In recent years the clinical interest for structured training in endovascular procedures has increased. Such procedures respect the physical integrity of the patient and at the same time ensure good therapeutic results. This study describes the development and testing of the B.E.S.T. (Basic Endovascular Skills Trainer) simulator. The B.E.S.T is an innovative physical endovascular simulator to learn basic skills of endovascular surgery. The simulator was tested by 25 clinicians with different levels of experience: novices, intermediates, and experts. All clinicians agree on affirming the importance of training in endovascular surgery; in particular they consider the B.E.S.T a valid simulator to learn specific basic skills of vascular surgery.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

Simulation training improves resident performance in hand-sewn vascular and bowel anastomoses

BACKGROUND

Surgical training has recently emphasized simulation-based training of core surgical skills and tasks such as bowel and vascular anastomoses. This may increase efficiency of training within the operating room. Objective data regarding the effectiveness of instruction or monitoring progress in simulating vascular and bowel anastomoses are lacking. The aim of this study is to provide subjective and objective assessments of simulation-based training among residents in hand-sewn vascular and bowel anastomoses.

METHODS

Residents received vascular and bowel anastomoses training. Each resident fashioned anastomoses on both cadaveric saphenous veins and small bowel. The residents repeated the anastomoses 1 week later. Performances were assessed subjectively and objectively by questionnaire, operative time, objective structured assessment of technical skills (OSATS) score, and leak pressures of the finished anastomosis.

RESULTS

Of 14 residents, 12 felt more confident with bowel anastomoses after the laboratory session, and 10 were more confident with vascular anastomoses. For vascular anastomoses, the operating time decreased (15.4 vs 14.2 minutes, p = 0.3), OSATS scores improved (14.9 vs 15.6, p = 0.15), and leak pressures improved (38.9 vs 71.8psi, p = 0.001) from the first to the second workshop. For bowel anastomoses, the operating time decreased (23 vs 18 minutes, p < 0.001), OSATS scores improved (12.9 vs 14.4, p < 0.001), and leak pressures improved (17.7 vs 26.9psi, p < 0.001).

DISCUSSION

After simulation-based training, residents performed vascular and bowel anastomoses more adeptly, quickly, and with a higher quality end product. Laboratory training can effectively improve residents' ability to perform anastomoses, which may result in increased efficiency of teaching in the operating room.

A systematic review of assessment of skill acquisition and operative competency in vascular surgical training

OBJECTIVE

The aim of this systematic review is to describe the literature and assessment tools evaluating vascular surgical operative performance that could potentially be used for the assessment of educational outcomes applicable to the Milestone Project and the Next Accreditation System.

METHODS

A systematic review of PubMed/MEDLINE, EMBASE, PsycINFO, and key journals from 1985 to 2013 was performed to identify English-language articles describing assessment of vascular surgical skills and competence. Qualifying studies were abstracted for data concerning study aims, study and assessment setting, skills measured, and metrics used to determine competency. Strengths, weaknesses, and psychometric robustness of the assessment tools were determined.

RESULTS

The literature search identified 617 citations. After title and abstract review, 65 articles were retrieved for full-text assessment and 48 articles were included in the final review. Twenty-nine articles assessed open vascular skills; 19, endovascular skills; six, nontechnical skills; and one, teamwork skills. The majority (84%) of studies were performed in a simulated environment, four (8%) were performed in the operating room, and the remaining three were performed in both a simulated environment and an operating room. Strengths and weaknesses of assessment tools were study and assessor dependent, with none applicable to all study scenarios or procedures.

CONCLUSIONS

The literature describing assessment tools pertinent to vascular surgery is diverse. Existing assessment tools may be relevant to individual technical skill acquisition assessment; however, an operative assessment tool relevant to vascular/endovascular surgery and generalizable to the wide spectrum of technical and nontechnical skills pertinent to vascular surgery needs to be developed, validated, and implemented to allow the practical assessment of resident readiness to operate in an unsupervised setting.

Assessment methods in surgical training in the United Kingdom

A career in surgery in the United Kingdom demands a commitment to a long journey of assessment. The assessment methods used must ensure that the appropriate candidates are selected into a programme of study or a job and must guarantee public safety by regulating the progression of surgical trainees and the certification of trained surgeons. This review attempts to analyse the psychometric properties of various assessment methods used in the selection of candidates to medical school, job selection, progression in training, and certification. Validity is an indicator of how well an assessment measures what it is designed to measure. Reliability informs us whether a test is consistent in its outcome by measuring the reproducibility and discriminating ability of the test. In the long journey of assessment in surgical training, the same assessment formats are frequently being used for selection into a programme of study, job selection, progression, and certification. Although similar assessment methods are being used for different purposes in surgical training, the psychometric properties of these assessment methods have not been examined separately for each purpose. Because of the significance of these assessments for trainees and patients, their reliability and validity should be examined thoroughly in every context where the assessment method is being used.

A randomized evaluation of simulation training on performance of vascular anastomosis on a high-fidelity in vivo model: the role of deliberate practice

OBJECTIVES

There is mounting evidence supporting the benefit of surgical skills practice in a simulated environment. However, the use of simulation in cardiac surgical training has been limited. The purpose of the current trial was to examine the effect of independent and deliberate simulator practice, during nonclinical time, on the performance of an end-to-side microvascular anastomosis in an in vivo model.

METHODS

This single-blinded, randomized controlled trial received institutional review board approval. Thirty-nine first- and second-year surgical trainees were randomized to an expert-guided tutorial on a procedural trainer or to the expert-guided tutorial combined with self-directed practice on the same procedural trainer. Self-directed practice consisted of 10 anastomoses performed on the procedural trainer: a low-fidelity, commercially available bench model using 4-mm polytetrafluoroethylene graft as simulated blood vessel. Two weeks after the tutorial, subjects performed an end-to-side anastomosis in a live porcine model, under realistic operating room conditions. Assessment of outcomes was performed by 2 blinded, expert observers, uings validated measurements of technical skill. The primary outcome was the score on the Objective Structured Assessment of Technical Skill (OSATS) scale. Secondary outcomes included an anastomosis-specific end-product evaluation and time to completion. Statistical analysis was conducted using nonparametric, univariate techniques.

RESULTS

Compared with residents who received expert-guided simulator training alone, those who in addition practiced on a simulator independently after hours scored significantly higher on the OSATS scale (23.7 ± 4.7 vs 18.5 ± 3.9, P = .003). Residents who practiced independently also scored significantly higher on the end-product evaluation (11.4 ± 3.2 vs 8.9 ± 2.1, P = .02) and performed the anastomosis significantly faster (777 seconds vs 977 seconds, P = .04). Interrater reliability was high between the expert observers (intraclass correlation coefficient = 0.8).

CONCLUSIONS

Residents who had the opportunity for self-directed simulator practice performed an end-to-side anastomosis more adeptly, more quickly, and with a higher quality end product. The results of this randomized trial suggest that independent training on a procedural trainer did transfer to improved performance in an operating room environment. Simulator training should be incorporated into cardiovascular surgical curricula and residents should have access to this modality for independent after-hours practice to improve operating room performance.

Education in vascular surgery: critical issues around the globe-training and qualification in vascular surgery in Europe

In 1958, the Union Européene des Médecins Spécialistes (UEMS), or European Union (EU) of Medical Specialists the European Union, was founded by the professional organizations of medical specialists in Europe. Among the objectives of the UEMS are to promote the highest level of patient care in the EU and to promote the harmonization of high-quality training programs within the various specialities throughout the EU. Within the 38 Specialist Sections of the UEMS are the European Boards, which are the working groups of the Specialist Sections. In 2005 Vascular Surgery was recognized as a separate and independent Section, a monospecialty, within the UEMS. The efforts of the UEMS are directed at facilitating the free exchange of training and work of trainees and medical specialists between EU countries. This situation, in combination with large differences in requirements and length of training in vascular surgery within the EU, stresses the importance of harmonization in training and certification in vascular surgery within the EU. For that reason, the European Board of Vascular Surgery has organized voluntary examinations yearly since 1996. The candidates who pass qualify as "Fellow of the European Board of Vascular Surgery" (FEBVS) since 2005. The first part of the examination evaluates the eligibility of the candidate (Certificate of Completion of Specialist Training, training center, logbook). The second part is a viva voce assessment that includes (1) case analyses, (2) a review of a scientific article, (3) an overall assessment, (4) a technical skills, and (5) an endovascular skills assessment. To pass the examination, the candidates must achieve a 67% success rate in each part of the examination. During the last 10 years, approximately 75% of the candidates have successfully taken the examination. In the near future the Section and Board, in close collaboration with the vascular societies in the EU, will develop a European vascular surgical syllabus and curriculum that will further harmonize and professionalize the training and certification of vascular surgery in Europe.

Self-assessment of technical skill in surgery: the need for expert feedback

INTRODUCTION

Technical skill has been formally assessed in the Fellow of the European Board of Vascular Surgery Examinations (FEBVS) since 2002. The aim of this study was to examine the relationship between expert assessment and trainee self-assessment.

MATERIALS AND METHODS

Forty-two examination candidates performed a saphenofemoral junction (SFJ) ligation and an anterior tibial anastomosis on a synthetic simulation. Each candidate was rated by two examiners using a validated rating scale for their generic surgical skill for both procedures. Candidates then anonymously rated their own performance using the same scale. Parametric tests were used in the statistical analysis; a P-value < 0.05 was considered significant.

RESULTS

The maximum mark in each assessment was 40; 24 was considered a competent score. The interobserver correlation for examiners marks were high (SFJ ligation, alpha = 0.68; distal anastomosis, alpha = 0.76). Examiners' marks were averaged. The mean examiner score for the SFJ ligation station was 27.8 (SD = 4.1) with 36 candidates (85.8%) attaining a competent score. The mean self-assessment score for this station was 30.7 (SD = 4.66). The mean examiners' marks for the distal anastomosis station was 29.2 (SD = 4.2); 39 candidates (92.8%) attained a competent score. The mean self-assessment score was 32.1 (SD = 4.0). There was no correlation between examiner and self-assessment scores in either station (Pearson's correlation coefficient: SFJ, r = 0.045, P = NS); distal anastomosis, r = 0.089, P = NS). Bland and Altman plots assessed the agreement between examiner and self-assessment. These showed candidates marked themselves higher than examiners with a mean difference of 2.9 marks in each station.

CONCLUSIONS

Candidates' self-assessment and expert independent assessment correlate poorly. Trainees overestimate their ability according to independent assessment; regular technical feedback during training is, therefore, essential.

Competence assessment of senior vascular trainees using a carotid endarterectomy bench model

Background

Competency-based assessment is being introduced to surgical training. The value of bench-top technical skills assessment using a synthetic carotid endarterectomy (CEA) model was evaluated in vascular trainees and consultants.

Methods

Forty-one surgeons (13 junior trainees, 15 senior trainees and 13 experienced consultants with experience of more than 50 CEAs) performed a three-throw knot-tying exercise on a jig and a CEA on the bench model. A composite score for knot-tying was calculated, incorporating electromagnetic motion analysis. CEA technical skill was assessed using validated rating scales by blinded video analysis.

Results

Senior trainees performed better than junior trainees in knot-tying (P = 0·025) as well as generic (P &lt; 0·001) and procedural (P &lt; 0·001) skills on CEA model assessment. There was no difference between senior trainees and consultants on any of these measures. The CEA model interobserver reliability was high for all rating scales (generic α = 0·974, procedural α = 0·952, end-product α = 0·976).

Conclusion

Senior trainees achieved the same score as consultants, suggesting a similar level of basic technical skill and knowledge required to perform CEA, and were significantly better than junior trainees. Performance on the bench model could provide an early assessment for suitability to proceed to operative training in a competency-based training and assessment programme.

Training for Carotid Intervention: Preparing the Next Generation

BACKGROUND

Carotid interventions are performed to reduce the cumulative risk of stroke. The success of the procedure is dependent upon maintaining low operative risk. This article reviews the current state of training for both carotid endarterectomy (CEA) and carotid angioplasty and stenting (CAS).

METHODS

Medline searches were performed to identify articles with the combination of the following key words: carotid, endarterectomy, stent, training, assessment and simulation. Manual searches of the reference lists and related papers was conducted.

RESULTS

Training and assessment for CEA and CAS follows the traditional apprenticeship model. There is no formal training protocol or objective means of assessment for either carotid endarterectomy or stenting. Models and simulators to allow for training and assessment away from the operative theatre have been developed, and exist for both CEA and CAS.

CONCLUSION

The technology exists to allow for both training and assessment of competency to take place in a controlled and objective environment for both CEA and CAS. The use of simulation needs to be robustly evaluated and assessed to both complement and augment existing training programs to ensure that the highest standards of care are maintained for treatment of carotid territory disease. Objective competency based training and assessment is no longer unattainable. Simulators augment this process and without them operative exposure is sporadic and crisis management infrequent.

Assessment of operative competence during carotid endarterectomy

Background

Objective Structured Assessment of Technical Skills (OSATS) has been used to assess operative competence. The aim was to evaluate its role in carotid endarterectomy (CEA).

Methods

Over 2 years, 17 trainees and 11 consultants were assessed by a single observer using a task-specific checklist and global ratings. Operating time and previous experience were also recorded.

Results

Trainees had previously performed fewer CEAs than consultants (median (interquartile range, i.q.r.) 15 (7–24) versus 94 (61–133) respectively; P &lt; 0·001) and they took longer to perform operations (median (i.q.r.) 128 (119–143) versus 90 (82·5–98·5) min; P &lt; 0·001). Trainees achieved lower task-specific (median (i.q.r.) 23 (17–26) versus 25 (24·5–27·5); P = 0·031) and global (33 (30–35) versus 40 (36·5–43); P = 0·001) scores than consultants. There was a strong positive correlation between task-specific scores and experience for trainees (r = 0·83, P &lt; 0·001), but not for consultants. Global ratings correlated with experience for trainees (r = 0·69, P = 0·002) and more so for consultants (r = 0·82, P = 0·002). There was a positive correlation between task-specific and global scores for trainees (r = 0·67, P = 0·003), but not for consultants.

Conclusion

Task-specific checklists were more discriminatory between trainees and may be useful for formative assessment (training). Global ratings were more discriminatory for consultants and may be useful for summative assessment (examination or revalidation).

The examination assessment of technical competence in vascular surgery

Background

The European Board of Surgery Qualification in Vascular Surgery is a pan-European examination for vascular surgeons who have attained a national certificate of completion of specialist training. A 2-year study was conducted before the introduction of a technical skills assessment in the examination.

Methods

The study included 30 surgeons: 22 candidates and eight examiners. They were tested on dissection (on a synthetic saphenofemoral junction model), anastomosis (on to anterior tibial artery of a synthetic leg model) and dexterity (a knot-tying simulator with electromagnetic motion analysis). Validated rating scales were used by two independent examiners. Composite knot-tying scores were calculated for the computerized station. The stations were weighted 35, 45 and 20 per cent, respectively.

Results

Examiners performed better than candidates in the dissection (P &lt; 0·001), anastomosis (P = 0·002) and dexterity (P = 0·005) stations. Participants performed consistently in the examination (dissection versus anastomosis: r = 0·79, P &lt; 0·001; dexterity versus total operative score: r = −0·73, P &lt; 0·001). Interobserver reliability was high (α = 0·91). No correlation was seen between a candidate's technical skill and oral examination performance or logbook-accredited scores.

Conclusion

Current surgical examinations do not address technical competence. This model appears to be a valid assessment of technical skills in an examination setting. The standards are set at a level appropriate for a specialist vascular surgeon.

Technical skills continue to improve beyond surgical training

BACKGROUND

There is growing focus on surgical technical competence and the means by which we are able to measure it. Ongoing studies have shown a plateau effect with increasing experience of the operator. The aim of this study was to assess the technical competence of five groups of surgeons with increasing experience and validate a new rating tool for use in surgical assessment.

METHODS

Fifty surgeons performed a saphenofemoral junction ligation on a synthetic groin model. The procedure was videotaped, blinded, and reviewed independently by three assessors. Performance was assessed using a previously validated global rating scale of generic surgical skill. In addition, each procedure was rated with the procedure-specific Imperial College Evaluation of Procedure-Specific Skill (ICEPS) rating scale to establish the construct validity (ability to differentiate on the basis of skill) and inter-observer reliability.

RESULTS

Both rating scales showed improved scores with ascending grades (P < .001) and demonstrated a high inter-observer reliability both for generic and procedure-specific skill (alpha = 0.97 and alpha = 0.96, respectively). Total operative scores demonstrated significant differences between surgeons in postgraduate years 1 and 2 and surgeons in years 3 and 4 and also between newly appointed and experienced consultants (P < .041). Procedure-specific performance showed a plateau effect at the registrar level. Generic skill continued to improve, and significant differences were seen between newly appointed and senior consultants (P < .026).

CONCLUSION

This study shows that surgical performance continues to improve significantly beyond consultancy, and the data suggest that generic and procedural performance continue to improve, with significant improvement in the former with increasing experience. The ICEPS rating scale demonstrates construct validity and a high inter-observer reliability supporting its use in formative and summative assessment.

Do workshops improved the technical skill of vascular surgical trainees?

AIMS

Adjuncts to conventional surgical training are needed in order to address the reduction in working hours. This purpose of this study was to objectively assess the efficacy of workshop training on simulators.

METHODS

Fifteen consecutive participants of the European Vascular Workshop in 2003 and 2004 were recruited to this study. Participants performed a proximal anastomosis on a commercially available abdominal aortic aneurysm simulator, were then given intensive training on sophisticated models for 3 days and re-assessed. Pre- and post-course procedures were videotaped and independently reviewed by three assessors (tapes were blinded and in random order). The operative end product was similarly assessed. Four measures of technical skill were used: generic skill, procedural skill; a five point technical rating of the anastomosis (assessed using validated rating scales) and procedure time. Non-parametric tests were used in the statistical analysis.

RESULTS

The video assessment scores for aneurysm repair increased significantly following completion of the course (p=0.006 and p=0.004 for generic and procedural skill, respectively). End product assessment scores increased significantly post-course (p=0.001) and participants performed aneurysm repair faster following the course (p<0.05). Inter-observer reliability ranged from alpha=0.84-0.98 for the three rating scales pre- and post-course.

CONCLUSION

Objective improvements in technical performance follow intensive workshop training. Participants' perform better, faster, and with an improved end product following the course. Such adjuncts to training play an important part in a focused integrated programme that addresses reduced work hours.

Setting standards for the assessment of operative competence

BACKGROUND

We have previously shown that the operative competence of trainees can be reliably assessed using structured checklists and video recording. These assessments are useful for training (formative assessment). However, a standard setting exercise is required before they can be used for examinations (summative assessment).

METHODS

Blinded videos of a saphenofemoral disconnection by an experienced (competent) trainee and an inexperienced trainee (not competent) were scored, using a structured checklist, by 14 consultant vascular surgeons and 14 vascular trainees. The observers were also asked to decide whether the surgeon was competent, borderline or not competent. Thirteen vascular operating room (OR) nurses performed the same exercise. The 'contrasting groups' method was used to compare the cut point between the scores.

RESULTS

There was complete separation between the surgeons' scores for the experienced trainee (median 16, range 13-18) and the inexperienced trainee (median 6.5, range 2-12), Chi-square p = 0.0001. This separation was confirmed by the judgements for competent (14-18), borderline (15-7), and not competent (8-2), p = 0.0001. Trainees awarded lower scores than the consultants to both videos, although this difference was not significant. The nurses performed almost as well as the surgeons.

CONCLUSIONS

Surgeons can discriminate between the video recordings of a competent and non-competent trainee. Such recordings could form part of a trainee's portfolio and contribute to subsequent examinations (summative assessment). It seems that nurses can also be used to assess the operative competence of surgical trainees.