Laparoscopic Nissen fundoplication assessment: task analysis as a model for the development of a procedural checklist

Validity and reliability evidence support task-specific metrics for laparoscopic fundoplication

BACKGROUND

Laparoscopic hiatal hernia repair (LHHR) is a complex operation requiring advanced surgical training. Surgical simulation offers a potential solution for learning complex operations without the need for high surgical volume. Our goal is to develop a virtual reality (VR) simulator for LHHR; however, data supporting task-specific metrics for this procedure are lacking. The purpose of this study was to develop and assess validity and reliability evidence of task-specific metrics for the fundoplication phase of LHHR.

METHODS

In phase I, structured interviews with expert foregut surgeons were conducted to develop task-specific metrics (TSM). In phase II, participants with varying levels of surgical expertise performed a laparoscopic Nissen fundoplication procedure on a porcine stomach explant. Video recordings were independently assessed by two blinded graders using global and TSM. An intraclass correlation coefficient (ICC) was used to assess interrater reliability (IRR). Performance scores were compared using a Kruskal-Wallis test. Spearman's rank correlation was used to evaluate the association between global and TSM.

RESULTS

Phase I of the study consisted of 12 interviews with expert foregut surgeons. Phase II engaged 31 surgery residents, a fellow, and 6 attendings in the simulation. Phase II results showed high IRR for both global (ICC = 0.84, p < 0.001) and TSM (ICC = 0.75, p < 0.001). Significant between-group differences were detected for both global (χ2 = 24.01, p < 0.001) and TSM (χ2 = 18.4, p < 0.001). Post hoc analysis showed significant differences in performance between the three groups for both metrics (p < 0.05). There was a strong positive correlation between the global and TSM (rs = 0.86, p < 0.001).

CONCLUSION

We developed task-specific metrics for LHHR and using a fundoplication model, we documented significant reliability and validity evidence. We anticipate that these LHHR task-specific metrics will be useful in our planned VR simulator.

A New Procedure-Based Assessment of Operative Skills in Gastric Bypass Surgery, Evaluated by Video Fragment Rating

PURPOSE

Feedback on technical and procedural skills is essential during the training of residents and fellows. The aim of this study was to assess the performance of a newly created instrument for the assessment of operative skills using laparoscopic Roux-en-Y gastric bypass (LRYGB) video fragments.

MATERIALS AND METHODS

A new procedure-based assessment (PBA) was created by combining LRYGB key steps with a 5-point independence scale. LRYGB performed by residents and surgeons with different levels of expertise were video recorded. Fragments of the pouch creation, gastro-jejunostomy and jejunojejunostomy, were review by 12 expert bariatric surgeons and the operative skills assessed with the PBA, Objective Structured Assessment of Technical Skill (OSATS), and the Bariatric OSATS (BOSATS). The PBA was compared to the OSATS and BOSATS. Mean scores for all items of the different assessments were summarized and compared using a T-test.

RESULTS

The scores of the procedural steps were combined and compared for all levels. The mean scores for beginner, intermediate, and expert level were 2.71, 3.70, and 3.90 for the PBA; for the OSATS 1.84, 2.86, and 3.44; and for the BOSATS 2.78, 3.56, and 4.19. Each of these assessments differentiated between the three skill levels (all p < 0.05).

CONCLUSION

The PBA discriminates well between different levels of operative skills. Similar patterns were found for the OSATS and BOSATS, showing that the randomly selected video fragments are representative samples for assessing skill level. Future research will demonstrate whether these results can be extrapolated to clinical training, and which scores allow for procedure certification.

Uncovering patient safety considerations in laparoscopic cholecystectomy using cognitive task analysis

INTRODUCTION

Educating residents on laparoscopic operations requires direct teaching and deliberate practice. Attending surgeons are often systematic when performing surgery, which creates a challenge when instructing surgery residents. The aim of this study was to use cognitive task analysis to expand laparoscopic cholecystectomy into microsteps reflecting expert surgeon cognition (perceptions, assessments, decisions, etc.) throughout the operation such that these could be better formalized and conveyed to residents in educational materials or assessments and to attending surgeons as teaching scripts.

MATERIALS AND METHODS

One surgeon, a surgical resident, and a human factors specialist conducted cognitive task analyses with three expert general surgeons and one hepatobiliary surgeon using semi-structured interviews. These interviews expanded an existing task model of laparoscopic cholecystectomy to specifically add patient safety aspects including injury prevention, risk management, and complication detection for each step. Interview analysis resulted in an expanded task diagram.

RESULTS

Cognitive task analysis expanded the current laparoscopic cholecystectomy task model from 19 to 97 microsteps. In addition to microsteps, an additional major step was identified, the planning step or step zero. Steps with the greatest number of microsteps included dissection with 15 microsteps and intraoperative cholangiogram with 10 microsteps.

DISCUSSION

Laparoscopic cholecystectomies are complex operations with multiple microsteps. Identification of these steps can lead to explicit strategies that can improve training of surgeons, with an end towards efficacy and safety. The identification of a planning step prior to beginning the operation is a significant finding that should arguably be included in all future cognitive task analyses regardless of operation or procedure, to emphasize to trainees what senior surgeons have learned through experience. These findings inform the development of interventions for surgical training and evaluation of competency.

Validity Evidence for Procedure-specific Competence Assessment Tools in General Surgery: A Scoping Review

OBJECTIVE

This study aims to provide a focused and detailed assessment of the validity evidence supporting procedure-specific operative assessment tools in general surgery.

SUMMARY OF BACKGROUND DATA

Competency-based assessment tools should be supported by robust validity evidence to be used reliably for evaluation of operative skills. The contemporary framework of validity relies on five sources of evidence: content, response process, internal structure, relation to other variables, and consequences.

METHODS

A systematic search of 8 databases was conducted for studies containing procedure-specific operative assessment tools in general surgery. The validity evidence supporting each tool was assessed and scored in alignment with the contemporary framework of validity. Methodological rigour of studies was assessed with the Medical Education Research Study Quality Instrument. The educational utility of each tool was assessed with the Accreditation Council for Graduate Medical Education framework.

RESULTS

There were 28 studies meeting inclusion criteria and 23 unique tools were assessed. Scores for validity evidence varied widely between tools, ranging from 3 - 14 (maximum 15). Medical Education Research Study Quality Instrument scores assessing the quality of study methodology were also variable (8.5-15.5, maximum 16.5). Direct reporting of educational utility criteria was limited.

CONCLUSIONS

This study has identified a small group of procedure-specific operative assessment tools in general surgery. Many of these tools have limited validity evidence and have not been studied sufficiently to be used reliably in high-stakes summative assessments. As general surgery transitions to competency-based training, a more robust library of operative assessment tools will be required to support resident education and evaluation.

Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score

INTRODUCTION

The aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS).

MATERIALS AND METHODS

The Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8).

RESULTS

The Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale.

CONCLUSION

With the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

The Complexity of the Retropubic Midurethral Sling: A Cognitive Task Analysis

OBJECTIVE

The primary aim of this study was to use cognitive task analysis to expand the retropubic midurethral sling into smaller steps, reflecting a surgeon's internal thought processes during the surgery.

METHODS

Two surgeons and a cognitive psychologist collaborated with expert urogynecologic surgeons in structured discussions and semistructured interviews, iteratively creating a list of clinical steps for the midurethral sling. They primarily considered 2 questions: (1) what action does the expert perform for this step, and (2) what information does the expert need to complete the step? We defined each additional piece of detail within a step as a microstep. The cognitive task analysis list was further reviewed by 4 external expert urogynecologic surgeons to obtain further detail. The process was repeated for every step until the maximum level of detail was reached. We used multiple methods to explore the relationship between microsteps and the cognitive load associated with various portions of the surgery.

RESULTS

Cognitive task analysis expanded the midurethral sling from 23 to 197 microsteps. Steps with the greatest number of microsteps included retropubic advancement with the trocar (19 microsteps) and ventral advancement of the trocar through the skin (17 microsteps).

CONCLUSIONS

The retropubic midurethral sling is a complex surgery with multiple microsteps embedded within in each step. Identification of these steps can lead to strategies to minimize cognitive load encouraging both efficacy and safety. Surgical training interventions and competency assessment can be developed based on this content.

A scoping review of assessment tools for laparoscopic suturing

BACKGROUND

A needs assessment identified a gap in teaching and assessment of laparoscopic suturing (LS) skills. The purpose of this review is to identify assessment tools that were used to assess LS skills, to evaluate validity evidence available, and to provide guidance for selecting the right assessment tool for specific assessment conditions.

METHODS

Bibliographic databases were searched till April 2017. Full-text articles were included if they reported on assessment tools used in the operating room/simulation to (1) assess procedures that require LS or (2) specifically assess LS skills.

RESULTS

Forty-two tools were identified, of which 26 were used for assessing LS skills specifically and 26 for procedures that require LS. Tools had the most evidence in internal structure and relationship to other variables, and least in consequences.

CONCLUSION

Through identification and evaluation of assessment tools, the results of this review could be used as a guideline when implementing assessment tools into training programs.

Development of an objective assessment tool for total laparoscopic hysterectomy: A Delphi method among experts and evaluation on a virtual reality simulator

INTRODUCTION

Total Laparoscopic hysterectomy (LH) requires an advanced level of operative skills and training. The aim of this study was to develop an objective scale specific for the assessment of technical skills for LH (H-OSATS) and to demonstrate feasibility of use and validity in a virtual reality setting.

MATERIAL AND METHODS

The scale was developed using a hierarchical task analysis and a panel of international experts. A Delphi method obtained consensus among experts on relevant steps that should be included into the H-OSATS scale for assessment of operative performances. Feasibility of use and validity of the scale were evaluated by reviewing video recordings of LH performed on a virtual reality laparoscopic simulator. Three groups of operators of different levels of experience were assessed in a Marseille teaching hospital (10 novices, 8 intermediates and 8 experienced surgeons). Correlations with scores obtained using a recognised generic global rating tool (OSATS) were calculated.

RESULTS

A total of 76 discrete steps were identified by the hierarchical task analysis. 14 experts completed the two rounds of the Delphi questionnaire. 64 steps reached consensus and were integrated in the scale. During the validation process, median time to rate each video recording was 25 minutes. There was a significant difference between the novice, intermediate and experienced group for total H-OSATS scores (133, 155.9 and 178.25 respectively; p = 0.002). H-OSATS scale demonstrated high inter-rater reliability (intraclass correlation coefficient [ICC] = 0.930; p<0.001) and test retest reliability (ICC = 0.877; p<0.001). High correlations were found between total H-OSATS scores and OSATS scores (rho = 0.928; p<0.001).

CONCLUSION

The H-OSATS scale displayed evidence of validity for assessment of technical performances for LH performed on a virtual reality simulator. The implementation of this scale is expected to facilitate deliberate practice. Next steps should focus on evaluating the validity of the scale in the operating room.

A Virtual Reality Training Curriculum for Laparoscopic Colorectal Surgery

OBJECTIVE

Training within a competency-based curriculum (CBC) outside the operating room enhances performance during real basic surgical procedures. This study aimed to design and validate a virtual reality CBC for an advanced laparoscopic procedure: sigmoid colectomy.

DESIGN

This was a multicenter randomized study. Novice (surgeons who had performed <5 laparoscopic colorectal resections as primary operator), intermediate (between 10 and 20), and experienced surgeons (>50) were enrolled. Validity evidence for the metrics given by the virtual reality simulator, the LAP Mentor, was based on the second attempt of each task in between groups. The tasks assessed were 3 modules of a laparoscopic sigmoid colectomy (medial dissection [MD], lateral dissection [LD], and anastomosis) and a full procedure (FP). Novice surgeons were randomized to 1 of 2 groups to perform 8 further attempts of all 3 modules or FP, for learning curve analysis.

SETTING

Two academic tertiary care centers-division of surgery of St. Mary's campus, Imperial College Healthcare NHS Trust, London and Nord Hospital, Assistance Publique-Hôpitaux de Marseille, Aix-Marseille Université, Marseille, were involved.

PARTICIPANTS

Novice surgeons were residents in digestive surgery at St. Mary's and Nord Hospitals. Intermediate and experienced surgeons were board-certified academic surgeons.

RESULTS

A total of 20 novice surgeons, 7 intermediate surgeons, and 6 experienced surgeons were enrolled. Evidence for validity based on experience was identified in MD, LD, and FP for time (p = 0.005, p = 0.003, and p = 0.001, respectively), number of movements (p = 0.013, p = 0.005, and p = 0.001, respectively), and path length (p = 0.03, p = 0.017, and p = 0.001, respectively), and only for time (p = 0.03) and path length (p = 0.013) in the anastomosis module. Novice surgeons' performance significantly improved through repetition for time, movements, and path length in MD, LD, and FP. Experienced surgeons' benchmark criteria were defined for all construct metrics showing validity evidence.

CONCLUSIONS

A CBC in laparoscopic colorectal surgery has been designed. Such training may reduce the learning curve during real colorectal resections in the operating room.

Development and content validation of performance assessments for endoscopic third ventriculostomy

PURPOSE

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

METHODS

The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.

RESULTS

Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80% agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50-80% agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.

CONCLUSIONS

We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

Advanced training in laparoscopic abdominal surgery: a systematic review

BACKGROUND

Simulation has spread widely this last decade, especially in laparoscopic surgery, and training out of the operating room has proven its positive impact on basic skills during real laparoscopic procedures. Few articles dealing with advanced training in laparoscopic abdominal surgery, however, have been published. Such training may decrease learning curves in the operating room for junior surgeons with limited access to complex laparoscopic procedures as a primary operator.

METHODS

Two reviewers, using MEDLINE, EMBASE, and The Cochrane Library conducted a systematic research with combinations of the following keywords: (teaching OR education OR computer simulation) AND laparoscopy AND (gastric OR stomach OR colorectal OR colon OR rectum OR small bowel OR liver OR spleen OR pancreas OR advanced surgery OR advanced procedure OR complex procedure). Additional studies were searched in the reference lists of all included articles.

RESULTS

Fifty-four original studies were retrieved. Their level of evidence was low: most of the studies were case series and one fifth were purely descriptive, but there were eight randomized trials. Pig models and video trainers as well as gastric and colorectal procedures were mainly assessed. The retrieved studies showed some encouraging trends in terms of trainee satisfaction with improvement after training, but the improvements were mainly on the training tool itself. Some tools have been proven to be construct-valid.

CONCLUSION

Higher-quality studies are required to appraise educational value in this field.

The Arthroscopic Surgical Skill Evaluation Tool (ASSET)

BACKGROUND

Surgeries employing arthroscopic techniques are among the most commonly performed in orthopaedic clinical practice; however, valid and reliable methods of assessing the arthroscopic skill of orthopaedic surgeons are lacking.

HYPOTHESIS

The Arthroscopic Surgery Skill Evaluation Tool (ASSET) will demonstrate content validity, concurrent criterion-oriented validity, and reliability when used to assess the technical ability of surgeons performing diagnostic knee arthroscopic surgery on cadaveric specimens.

STUDY DESIGN

Cross-sectional study; Level of evidence, 3.

METHODS

Content validity was determined by a group of 7 experts using the Delphi method. Intra-articular performance of a right and left diagnostic knee arthroscopic procedure was recorded for 28 residents and 2 sports medicine fellowship-trained attending surgeons. Surgeon performance was assessed by 2 blinded raters using the ASSET. Concurrent criterion-oriented validity, interrater reliability, and test-retest reliability were evaluated.

RESULTS

Content validity: The content development group identified 8 arthroscopic skill domains to evaluate using the ASSET. Concurrent criterion-oriented validity: Significant differences in the total ASSET score (P < .05) between novice, intermediate, and advanced experience groups were identified. Interrater reliability: The ASSET scores assigned by each rater were strongly correlated (r = 0.91, P < .01), and the intraclass correlation coefficient between raters for the total ASSET score was 0.90. Test-retest reliability: There was a significant correlation between ASSET scores for both procedures attempted by each surgeon (r = 0.79, P < .01).

CONCLUSION

The ASSET appears to be a useful, valid, and reliable method for assessing surgeon performance of diagnostic knee arthroscopic surgery in cadaveric specimens. Studies are ongoing to determine its generalizability to other procedures as well as to the live operating room and other simulated environments.

Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery

BACKGROUND

There is no objective scale for assessment of operative skill in laparoscopic gastric bypass (LGBP). The objective of this study was to develop and demonstrate feasibility of use, validity, and reliability of a Bariatric Objective Structured Assessment of Technical Skill (BOSATS) scale.

STUDY DESIGN

The BOSATS scale was developed using a hierarchical task analysis (HTA), a Delphi questionnaire, and a panel of international experts in bariatric surgery. The feasibility of use, reliability, and validity of the developed scale were demonstrated by reviewing 52 prospectively collected video recordings of LGBP performed by novice and experienced surgeons.

RESULTS

A total of 214 discrete steps were identified in HTA. A total of 12 and 17 panel members completed the first and second round of the Delphi questionnaire, respectively. Consensus among the panel was achieved after the second round (Cronbach's alpha = 0.85). The BOSATS scale demonstrated high inter-rater (intraclass correlation coefficient [ICC] = 0.954; p < 0.001) and test-retest reliability (ICC = 0.99; p < 0.001). Significant differences between BOSATS scores of experienced and novice surgeon groups were noted for the creation of jejunojejunostomy (JJ), gastric pouch, linear stapled gastrojejunostomy (GJ), circular stapled GJ, and hand-sewn GJ. Moderate to high correlations between BOSATS scale and Objective Structured Assessment of Technical Skills Global Rating Scale (OSATS GRS) were seen for JJ (rho = 0.59; p = 0.001), gastric pouch (rho = 0.48; p = 0.0004), linear stapled GJ (rho = 0.70; p = 0.0001), and hand-sewn GJ (rho = 0.96; p < 0.0001).

CONCLUSIONS

The BOSATS scale is a feasible to use, reliable, and valid instrument for objective assessment of operative performance in LGBP. Implementation of this scale is expected to facilitate deliberate practice and provide a means for future certification in bariatric surgery.

Neurosurgical checklists: a review

Morbidity due to avoidable medical errors is a crippling reality intrinsic to health care. In particular, iatrogenic surgical errors lead to significant morbidity, decreased quality of life, and attendant costs. In recent decades there has been an increased focus on health care quality improvement, with a concomitant focus on mitigating avoidable medical errors. The most notable tool developed to this end is the surgical checklist. Checklists have been implemented in various operating rooms internationally, with overwhelmingly positive results. Comparatively, the field of neurosurgery has only minimally addressed the utility of checklists as a health care improvement measure. Literature on the use of checklists in this field has been sparse. Considering the widespread efficacy of this tool in other fields, the authors seek to raise neurosurgical awareness regarding checklists by reviewing the current literature.

Impact of perceptual ability and mental imagery training on simulated laparoscopic knot-tying in surgical novices using a Nissen fundoplication model

BACKGROUND

Performing minimally invasive surgery requires training and visual-spatial intelligence. The aim of our study was to examine the impact of visual-spatial perception and additional mental training on the simulated laparoscopic knot-tying task performed by surgical novices.

METHODS

A total of 40 medical students randomly assigned to two groups underwent two sessions of laparoscopic basic training on a VR simulator (SimSurgery®, Oslo, Norway). The variables time and tip trajectory (total path length of the instrument tip trajectory) were used to assess the performance of the intracorporeal knot-tying task using a laparoscopic Nissen fundoplication model. The experimental group completed additional mental practice during the interval between the two training sessions. All performed a cube subtest of a standard intelligence test (I-S-T 2000 R) to evaluate visual-spatial ability.

RESULTS

All participants achieved an improvement in time (t = 9.861; p < 0.001) and tip trajectory (t = 6.833; p < 0.001) in the second training session. High scores on the visual-spatial test correlated with a faster performance (r = -0.557; p < 0.001) and more precise movements (r = -0.377; p = 0.016). Comparison of the two groups did not show any statistical significant differences in the parameters time and tip trajectory.

CONCLUSIONS

Visual-spatial intelligence tested by a cube test correlated with simulated laparoscopic knot-tying skills in surgical novices. Additional mental practice did not improve the overall knot-tying performance. Further studies are therefore required to determine whether mental practice might be beneficial for experienced laparoscopic surgeons or for more complex tasks.