SAGES Video-Based Assessment (VBA) program: a vision for life-long learning for surgeons

Evaluating the generalizability of video-based assessment of intraoperative surgical skill in capsulorhexis

PURPOSE

Assessment of intraoperative surgical skill is necessary to train surgeons and certify them for practice. The generalizability of deep learning models for video-based assessment (VBA) of surgical skill has not yet been evaluated. In this work, we evaluated one unsupervised domain adaptation (UDA) and three semi-supervised (SSDA) methods for generalizability of models for VBA of surgical skill in capsulorhexis by training on one dataset and testing on another.

METHODS

We used two datasets, D99 and Cataract-101 (publicly available), and two state-of-the-art models for capsulorhexis. The models include a convolutional neural network (CNN) to extract features from video images, followed by a long short-term memory (LSTM) network or a transformer. We augmented the CNN and the LSTM with attention modules. We estimated accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).

RESULTS

Maximum mean discrepancy (MMD) did not improve generalizability of CNN-LSTM but slightly improved CNN transformer. Among the SSDA methods, Group Distributionally Robust Supervised Learning improved generalizability in most cases.

CONCLUSION

Model performance improved with the domain adaptation methods we evaluated, but it fell short of within-dataset performance. Our results provide benchmarks on a public dataset for others to compare their methods.

Artificial intelligence-enhanced video-based assessment of surgical quality for training in laparoscopic right hemicolectomy: The "Marginal Gains" pilot study

INTRODUCTION

The study aims to propose a standardised workflow with critical views for surgical quality assessment (SQA) in laparoscopic right hemicolectomy (LRH), to disseminate it through a "Marginal Gains" course, and to evaluate its impact through artificial intelligence (AI) enhanced video-based assessment (VBA).

MATERIALS AND METHODS

Expert colorectal surgeons proposed a protocol for SQA in LRH based on evidence and consensus. A course ("Marginal Gains") comprising remote e-learning and on-site clinical immersion was organised to disseminate the proposed approach to LRH. Videos of procedures performed by participants before and after the course were analysed by experts (SQA items) and AI (workflows). Descriptive and inferential statistic was used to study the applicability of the proposed protocol and the impact of the course.

RESULTS

A protocol with 21 SQA items over 9 phases for LRH was proposed. Four surgeons successfully completed the pilot "Marginal Gains" course. Across the 8 videos uploaded, VBA showed that the proposed SQA items were appliable in 82.7 % (139/168 items) of the cases. Three out of 4 of the participants had higher SQA scores after the course, with an overall improvement of 30 % (20.75 ± 13.2 vs 32.75 ± 2.99 points; p = 0.126). All participants performed intracorporeal anastomosis after the course, with a significant quality improvement (1.5 ± 1.73 vs 3.75 ± 0.5 points; +56 %; p = 0.046). Overall, mean operative times increased by 00:23:38 after the course (01:36:03 ± 00:10:43 vs 01:59:41 ± 00:48:02; p = 0.465).

CONCLUSION

This study advocates for a paradigm shift in surgical education and practice by proposing, piloting, and measuring the impact of a structured, step-based approach to LRH.

Artificial intelligence assessment of tissue-dissection efficiency in laparoscopic colorectal surgery

PURPOSE

Several surgical-skill assessment tools emphasize the importance of efficient tissue-dissection, whose assessment relies on human judgment and is thus subject to bias. Automated assessment may help solve this problem. This study aimed to verify the feasibility of surgical-skill assessment using a deep learning-based recognition model.

METHODS

This retrospective study used multicenter intraoperative videos of laparoscopic colorectal surgery (sigmoidectomy or high anterior resection) for colorectal cancer obtained from 766 cases across Japan. Three groups with different skill levels were distinguished: high-, intermediate-, and low-skill. We developed a model to recognize tissue dissection by the monopolar device using deep learning-based computer-vision technology. Tissue-dissection time per monopolar device appearance time (efficient-dissection time ratio) was extracted as a quantitative parameter describing efficient dissection. We automatically measured the efficient-dissection time ratio using the recognition model of 8 surgical instruments and tissue-dissection on/off classification model. The efficient-dissection time ratio was compared among groups; the feasibility of distinguishing them was explored using the model. The model-calculated parameters were evaluated to determine whether they could differentiate high-, intermediate-, and low-skill groups.

RESULTS

The tissue-dissection recognition model had an overall accuracy of 0.91. There was a moderate correlation (0.542; 95% confidence interval, 0.288-0.724; P < 0.001) between manually and automatically measured efficient-dissection time ratios. Efficient-dissection time ratios by this model were significantly higher in the high-skill than in intermediate-skill (P = 0.0081) and low-skill (P = 0.0249) groups.

CONCLUSION

An automated efficient-dissection assessment model using a monopolar device was constructed with a feasible automated skill-assessment method.

Modified Delphi Procedure to Achieve Consensus for the Concept of a National Curriculum for Minimally Invasive and Robot-assisted Surgery in Germany (GeRMIQ)

The rapid development of minimally invasive surgery (MIS) and robot-assisted surgery (RAS) requires standardized training to ensure high-quality patient care. In Germany, there is currently a lack of a standardized curriculum that teaches these specialized skills. The aim of this study is to find a consensus for the development of a nationwide curriculum for MIS and RAS with the subsequent implementation of the consented content.A modified Delphi process was used to reach consensus among national experts in MIS and RAS. The process included a literature review, an online survey and an expert conference.All 12 invited experts participated in the survey. They primarily achieved consensus on 73% and secondarily within the expert conference on 95 out of 122 questions (77.9%). The preference for a basic curriculum as a foundation on which specialized modules can build on was particularly clear. The results support the development of an integrated curriculum for MIS and RAS that includes step-by-step training from theoretical knowledge via e-learning modules to practical skills in dry lab simulations and in the OR. Emphasis was placed on the need to promote clinical judgment and decision making through targeted assessment during the learning curve to ensure effective application of learned skills in clinical practice. There was also a consensus that training content must be aligned with learners' skill acquisition using objective performance assessments in line with the principle of proficiency-based progression (PBP). The continuous updating of the curriculum to keep it up to date with the latest technology was considered essential.The study underlines the urgent need for a standardized training curriculum for MIS and RAS in Germany in order to increase patient safety and improve the quality of surgical care. There is broad expert consensus for the implementation of such a curriculum. It aims to ensure a contemporary and internationally competitive uniform quality of training and to increase the attractiveness of surgical training.

Deep Learning for Video-Based Assessment in Surgery

This surgical innovation explains how applying deep neural networks could ensure the continued use of video-based assessment.

Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition

Automatic surgical phase recognition is a core technology for modern operating rooms and online surgical video assessment platforms. Current state-of-the-art methods use both spatial and temporal information to tackle the surgical phase recognition task. Building on this idea, we propose the Multi-Scale Action Segmentation Transformer (MS-AST) for offline surgical phase recognition and the Multi-Scale Action Segmentation Causal Transformer (MS-ASCT) for online surgical phase recognition. We use ResNet50 or EfficientNetV2-M for spatial feature extraction. Our MS-AST and MS-ASCT can model temporal information at different scales with multi-scale temporal self-attention and multi-scale temporal cross-attention, which enhances the capture of temporal relationships between frames and segments. We demonstrate that our method can achieve 95.26% and 96.15% accuracy on the Cholec80 dataset for online and offline surgical phase recognition, respectively, which achieves new state-of-the-art results. Our method can also achieve state-of-the-art results on non-medical datasets in the video action segmentation domain.

SF-TMN: SlowFast temporal modeling network for surgical phase recognition

PURPOSE

Automatic surgical phase recognition is crucial for video-based assessment systems in surgical education. Utilizing temporal information is crucial for surgical phase recognition; hence, various recent approaches extract frame-level features to conduct full video temporal modeling.

METHODS

For better temporal modeling, we propose SlowFast temporal modeling network (SF-TMN) for offline surgical phase recognition that can achieve not only frame-level full video temporal modeling but also segment-level full video temporal modeling. We employ a feature extraction network, pretrained on the target dataset, to extract features from video frames as the training data for SF-TMN. The Slow Path in SF-TMN utilizes all frame features for frame temporal modeling. The Fast Path in SF-TMN utilizes segment-level features summarized from frame features for segment temporal modeling. The proposed paradigm is flexible regarding the choice of temporal modeling networks.

RESULTS

We explore MS-TCN and ASFormer as temporal modeling networks and experiment with multiple combination strategies for Slow and Fast Paths. We evaluate SF-TMN on Cholec80 and Cataract-101 surgical phase recognition tasks and demonstrate that SF-TMN can achieve state-of-the-art results on all considered metrics. SF-TMN with ASFormer backbone outperforms the state-of-the-art Swin BiGRU by approximately 1% in accuracy and 1.5% in recall on Cholec80. We also evaluate SF-TMN on action segmentation datasets including 50salads, GTEA, and Breakfast, and achieve state-of-the-art results.

CONCLUSION

The improvement in the results shows that combining temporal information from both frame level and segment level by refining outputs with temporal refinement stages is beneficial for the temporal modeling of surgical phases.

One-shot skill assessment in high-stakes domains with limited data via meta learning

Deep Learning (DL) has achieved robust competency assessment in various high-stakes fields. However, the applicability of DL models is often hampered by their substantial data requirements and confinement to specific training domains. This prevents them from transitioning to new tasks where data is scarce. Therefore, domain adaptation emerges as a critical element for the practical implementation of DL in real-world scenarios. Herein, we introduce A-VBANet, a novel meta-learning model capable of delivering domain-agnostic skill assessment via one-shot learning. Our methodology has been tested by assessing surgical skills on five laparoscopic and robotic simulators and real-life laparoscopic cholecystectomy. Our model successfully adapted with accuracies up to 99.5 % in one-shot and 99.9 % in few-shot settings for simulated tasks and 89.7 % for laparoscopic cholecystectomy. This study marks the first instance of a domain-agnostic methodology for skill assessment in critical fields setting a precedent for the broad application of DL across diverse real-life domains with limited data.

Research on workflow recognition for liver rupture repair surgery

Liver rupture repair surgery serves as one tool to treat liver rupture, especially beneficial for cases of mild liver rupture hemorrhage. Liver rupture can catalyze critical conditions such as hemorrhage and shock. Surgical workflow recognition in liver rupture repair surgery videos presents a significant task aimed at reducing surgical mistakes and enhancing the quality of surgeries conducted by surgeons. A liver rupture repair simulation surgical dataset is proposed in this paper which consists of 45 videos collaboratively completed by nine surgeons. Furthermore, an end-to-end SA-RLNet, a self attention-based recurrent convolutional neural network, is introduced in this paper. The self-attention mechanism is used to automatically identify the importance of input features in various instances and associate the relationships between input features. The accuracy of the surgical phase classification of the SA-RLNet approach is 90.6%. The present study demonstrates that the SA-RLNet approach shows strong generalization capabilities on the dataset. SA-RLNet has proved to be advantageous in capturing subtle variations between surgical phases. The application of surgical workflow recognition has promising feasibility in liver rupture repair surgery.

Defining the Standard for Surgical Video Deidentification

This article reviews the implementation of standards for surgical video deidentification.

Surgical workflow recognition with temporal convolution and transformer for action segmentation

PURPOSE

Automatic surgical workflow recognition enabled by computer vision algorithms plays a key role in enhancing the learning experience of surgeons. It also supports building context-aware systems that allow better surgical planning and decision making which may in turn improve outcomes. Utilizing temporal information is crucial for recognizing context; hence, various recent approaches use recurrent neural networks or transformers to recognize actions.

METHODS

We design and implement a two-stage method for surgical workflow recognition. We utilize R(2+1)D for video clip modeling in the first stage. We propose Action Segmentation Temporal Convolutional Transformer (ASTCFormer) network for full video modeling in the second stage. ASTCFormer utilizes action segmentation transformers (ASFormers) and temporal convolutional networks (TCNs) to build a temporally aware surgical workflow recognition system.

RESULTS

We compare the proposed ASTCFormer with recurrent neural networks, multi-stage TCN, and ASFormer approaches. The comparison is done on a dataset comprised of 207 robotic and laparoscopic cholecystectomy surgical videos annotated for 7 surgical phases. The proposed method outperforms the compared methods achieving a [Formula: see text] relative improvement in the average segmental F1-score over the state-of-the-art ASFormer method. Moreover, our proposed method achieves state-of-the-art results on the publicly available Cholec80 dataset.

CONCLUSION

The improvement in the results when using the proposed method suggests that temporal context could be better captured when adding information from TCN to the ASFormer paradigm. This addition leads to better surgical workflow recognition.

An American Board of Surgery Pilot of Video Assessment of Surgeon Technical Performance in Surgery

OBJECTIVE

The American Board of Surgery (ABS) sought to investigate the suitability of video-based assessment (VBA) as an adjunct to certification for assessing technical skills.

BACKGROUND

Board certification is based on the successful completion of a residency program coupled with knowledge and reasoning assessments. VBA is a new modality for evaluating operative skills that have been shown to correlate with patient outcomes after surgery.

METHODS

Diplomates of the ABS were initially assessed for background knowledge and interest in VBA. Surgeons were then solicited to participate in the pilot. Three commercially available VBA platforms were identified and used for the pilot assessment. All participants served as reviewers and reviewees for videos. After the interaction, participants were surveyed regarding their experiences and recommendations to the ABS.

RESULTS

To the initial survey, 4853/25,715 diplomates responded. The majority were neither familiar with VBA, nor the tools used for operative assessments. Two hundred seventy-four surgeons actively engaged in the subsequent pilot. One hundred sixty-nine surgeons completed the postpilot survey. Most participants found the process straightforward. Of the participants, 74% felt that the feedback would help their surgical practice. The majority (81%) remain interested in VBA for continuing medical education credits. Using VBA in continuous certification could improve surgeon skills felt by 70%. Two-thirds of participants felt VBA could help identify and remediate underperforming surgeons. Identified barriers to VBA included limitations for open surgery, privacy issues, and technical concerns.

CONCLUSIONS

VBA is promising as an adjunct to the current board certification process and should be further considered by the ABS.

Universal meta-competencies of operative performances: a literature review and qualitative synthesis

BACKGROUND

Prioritizing patient health is essential, and given the risk of mortality, surgical techniques should be objectively evaluated. However, there is no comprehensive cross-disciplinary system that evaluates skills across all aspects among surgeons of varying levels. Therefore, this study aimed to uncover universal surgical competencies by decomposing and reconstructing specific descriptions in operative performance assessment tools, as the basis of building automated evaluation system using computer vision and machine learning-based analysis.

METHODS

The study participants were primarily expert surgeons in the gastrointestinal surgery field and the methodology comprised data collection, thematic analysis, and validation. For the data collection, participants identified global operative performance assessment tools according to detailed inclusion and exclusion criteria. Thereafter, thematic analysis was used to conduct detailed analyses of the descriptions in the tools where specific rules were coded, integrated, and discussed to obtain high-level concepts, namely, "Skill meta-competencies." "Skill meta-competencies" was recategorized for data validation and reliability assurance. Nine assessment tools were selected based on participant criteria.

RESULTS

In total, 189 types of skill performances were extracted from the nine tool descriptions and organized into the following five competencies: (1) Tissue handling, (2) Psychomotor skill, (3) Efficiency, (4) Dissection quality, and (5) Exposure quality. The evolutionary importance of these competences' different evaluation targets and purpose over time were assessed; the results showed relatively high reliability, indicating that the categorization was reproducible. The inclusion of basic (tissue handling, psychomotor skill, and efficiency) and advanced (dissection quality and exposure quality) skills in these competencies enhanced the tools' comprehensiveness.

CONCLUSIONS

The competencies identified to help surgeons formalize and implement tacit knowledge of operative performance are highly reproducible. These results can be used to form the basis of an automated skill evaluation system and help surgeons improve the provision of care and training, consequently, improving patient prognosis.

Video-based formative and summative assessment of surgical tasks using deep learning

To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated-none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA is manual, time-intensive, and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively provide a high-stakes summative assessment of surgical skill execution based on video feeds and low-stakes formative assessment to guide surgical skill acquisition. Formative assessment is generated using heatmaps of visual features that correlate with surgical performance. Hence, the DL model paves the way for the quantitative and reproducible evaluation of surgical tasks from videos with the potential for broad dissemination in surgical training, certification, and credentialing.

The association between video-based assessment of intraoperative technical performance and patient outcomes: a systematic review

BACKGROUND

Efforts to improve surgical safety and outcomes have traditionally placed little emphasis on intraoperative performance, partly due to difficulties in measurement. Video-based assessment (VBA) provides an opportunity for blinded and unbiased appraisal of surgeon performance. Therefore, we aimed to systematically review the existing literature on the association between intraoperative technical performance, measured using VBA, and patient outcomes.

METHODS

Major databases (Medline, Embase, Cochrane Database, and Web of Science) were systematically searched for studies assessing the association of intraoperative technical performance measured by tools supported by validity evidence with short-term (≤ 30 days) and/or long-term postoperative outcomes. Study quality was assessed using the Newcastle-Ottawa Scale. Results were appraised descriptively as study heterogeneity precluded meta-analysis.

RESULTS

A total of 11 observational studies were identified involving 8 different procedures in foregut/bariatric (n = 4), colorectal (n = 4), urologic (n = 2), and hepatobiliary surgery (n = 1). The number of surgeons assessed ranged from 1 to 34; patient sample size ranged from 47 to 10,242. High risk of bias was present in 5 of 8 studies assessing short-term outcomes and 2 of 6 studies assessing long-term outcomes. Short-term outcomes were reported in 8 studies (i.e., morbidity, mortality, and readmission), while 6 reported long-term outcomes (i.e., cancer outcomes, weight loss, and urinary continence). Better intraoperative performance was associated with fewer postoperative complications (6 of 7 studies), reoperations (3 of 4 studies), and readmissions (1 of 4 studies). Long-term outcomes were less commonly investigated, with mixed results.

CONCLUSION

Current evidence supports an association between superior intraoperative technical performance measured using surgical videos and improved short-term postoperative outcomes. Intraoperative performance analysis using video-based assessment represents a promising approach to surgical quality-improvement.

Computer vision in surgery: from potential to clinical value

Hundreds of millions of operations are performed worldwide each year, and the rising uptake in minimally invasive surgery has enabled fiber optic cameras and robots to become both important tools to conduct surgery and sensors from which to capture information about surgery. Computer vision (CV), the application of algorithms to analyze and interpret visual data, has become a critical technology through which to study the intraoperative phase of care with the goals of augmenting surgeons' decision-making processes, supporting safer surgery, and expanding access to surgical care. While much work has been performed on potential use cases, there are currently no CV tools widely used for diagnostic or therapeutic applications in surgery. Using laparoscopic cholecystectomy as an example, we reviewed current CV techniques that have been applied to minimally invasive surgery and their clinical applications. Finally, we discuss the challenges and obstacles that remain to be overcome for broader implementation and adoption of CV in surgery.

Competency Assessment for Laparoscopic Anti-Reflux Surgery: Design and Delphi Review, a Collaboration With the American Foregut Society

Background

Laparoscopic anti-reflux surgery, including hiatus hernia repair, is a common operation performed by both general and thoracic surgeons and an important learning objective for surgical trainees. This study aimed to design a competency assessment instrument for laparoscopic anti-reflux surgery.

Method

A comprehensive competency assessment instrument was designed by a process of logical analysis by four expert thoracic surgeons with an interest in foregut surgery, and then reviewed informally by a panel of experts. The instrument was then further assessed and refined using a modified Delphi process. The Delphi questionnaire was distributed to all members of the Fellowship Training Committee of the American Foregut Society (n=21).

Results

A first draft of the competency assessment instrument included 32 steps in four categories. The first round of the Delphi review was completed by 14 respondents (response rate 66.7%). A total of three rounds of Delphi review were performed. Ultimately, 25 items were retained from the original instrument and one modified and four new items were added. The final instrument has 30 steps in four categories.

Conclusions

An international and inter-specialty consensus was established on the key components of assessing competence to perform anti-reflux surgery. The resulting instrument could be used to guide competency based assessments of general and thoracic surgeons and trainees.

Surgical data science - from concepts toward clinical translation

Recent developments in data science in general and machine learning in particular have transformed the way experts envision the future of surgery. Surgical Data Science (SDS) is a new research field that aims to improve the quality of interventional healthcare through the capture, organization, analysis and modeling of data. While an increasing number of data-driven approaches and clinical applications have been studied in the fields of radiological and clinical data science, translational success stories are still lacking in surgery. In this publication, we shed light on the underlying reasons and provide a roadmap for future advances in the field. Based on an international workshop involving leading researchers in the field of SDS, we review current practice, key achievements and initiatives as well as available standards and tools for a number of topics relevant to the field, namely (1) infrastructure for data acquisition, storage and access in the presence of regulatory constraints, (2) data annotation and sharing and (3) data analytics. We further complement this technical perspective with (4) a review of currently available SDS products and the translational progress from academia and (5) a roadmap for faster clinical translation and exploitation of the full potential of SDS, based on an international multi-round Delphi process.

Intraoperative Time-Out to Promote the Implementation of the Critical View of Safety in Laparoscopic Cholecystectomy: A Video-Based Assessment of 343 Procedures

BACKGROUND

The critical view of safety (CVS) is poorly adopted in surgical practices, although it is recommended ubiquitously to prevent major bile duct injuries during laparoscopic cholecystectomy (LC). This study aimed to investigate whether performing a short intraoperative time-out can improve CVS implementation.

STUDY DESIGN

In this before vs after study, surgeons performing LCs at an academic center were invited to use a 5-second long time-out to verify CVS before dividing the cystic duct (5-second rule). The primary aim was to compare the rate of CVS achievement for LC performed in the year before vs the year after implementation of the 5-second rule. The CVS achievement rate was computed after exclusion of bailout procedures using a mediated video-based assessment made by 2 independent reviewers. Clinical outcomes, LC workflows, and postoperative reports were also compared.

RESULTS

Three hundred and forty-three of 381 LC performed between December 2017 and November 2019 (171 before and 172 after implementation of the 5-second rule) were analyzed. The 5-second rule was associated with a significantly increased rate of CVS achievement (15.9% vs 44.1% before vs after the 5-second rule, respectively; p < 0.001). Significant differences were also observed with respect to the rate of bailout procedures (8.2% vs 15.7%; p = 0.04), median time (hours:minutes:seconds) to clip the cystic duct or artery (00:17:26; interquartile range 00:11:48 to 00:28:35 vs 00:23:12; interquartile range 00:14:29 to 00:31:45 duration; p = 0.007), and the rate of postoperative CVS reporting (1.3% vs 28.8%; p < 0.001). Postoperative morbidity was comparable (1.8% vs 2.3%; p = 0.68).

CONCLUSIONS

Performing a short intraoperative time-out was associated with an improved CVS achievement rate. Systematic intraoperative cognitive aids should be studied to sustain the uptake of guidelines.

Intraoperative and postoperative complications in colorectal procedures: the role of continuous updating in medicine

Accepting surgical complications, especially those related to the learning curve, as unavoidable events in colorectal procedures, is like accepting to fly onboard an aircraft with a 10% to 20% chance of not arriving at final destination. Under this condition, it is very likely that the aviation industry and the concurrent reshaping of the world and of our lives would have not been possible in the absence of high reliability and reproducibility of safe flights. It is hard to imagine surgery without any intraoperative and/or postoperative complications. Nevertheless, there is a plenty of room for improvement by simply adopting what has been explicitly and scientifically demonstrated; training outside of the operating room (OR), usage of modern information technologies and application of evidence-based perioperative care protocols. Additionally, the possibility to objectively measure and monitor the technical and even non-technical skills and competencies of individual surgeons and even of OR teams through the application of structured and validated assessment tools can finally put an end to the self-referential, purely hierarchical, and indeed extremely unreliable process of being authorized or not to perform operations on patients. Last but not least, a wide range of new technologies spanning from augmented imaging modalities, virtual reality for intraoperative guidance, improved robotic manipulators, artificial intelligence to assist in preoperative patient specific risk assessment, and intraoperative decision-making has the potential to tackle several hidden roots of surgical complications.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Performance Assessment for Total Laparoscopic Hysterectomy in the Operating Room: Validity Evidence of a Procedure-specific Rating Scale

STUDY OBJECTIVE

The technical conduct of total laparoscopic hysterectomy (LH) is critical to surgical outcomes. This study explored the validity evidence of an objective scale specific to the assessment of technical skills (H-OSATS) for 7 tasks of an LH with salpingo-oophorectomy procedure performed in the operating room.

DESIGN

Observational cohort study.

SETTING

Two academic hospitals in Marseille and Montpellier, France.

PATIENTS

Three groups of operators (novice, intermediate, and experienced surgeons) were video recorded during their live performances of LH on a simple case. For each group, a dozen unedited videos were obtained for the following tasks: division of the round ligament, division of the infundibulopelvic ligament, creation of the bladder flap, opening of the posterior peritoneum, division of the uterine vessels, colpotomy, and closure of the vault.

INTERVENTIONS

Two qualified raters blindly assessed each video using the H-OSATS rating scale. Inter-rater reliability and test-retest reliability were calculated as measures of internal structure. In a separate round of evaluations, the raters provided a global competent/noncompetent decision for each performance. As a measure of consequential validity, a pass/fail score was set for each task using the contrasting group method.

MEASUREMENTS AND MAIN RESULTS

Three tasks (creation of the bladder flap, colpotomy, and closure of the vault) displayed sound validity evidence: a meaningful total score difference among the 3 groups of experience as well as between the intermediate and experienced surgeons, reliability outcomes of >0.7, and a pass/fail score with a theoretical false-positive rate of <10%.

CONCLUSION

The validity evidence of the H-OSATS rating scale differed for separate evaluations of the 7 tasks. Three tasks (i.e., creation of the bladder flap, colpotomy, and closure of the vault) revealed sound validity evidence, including at the level of the attending surgeon, whereas other tasks were more consistent with low-stakes formative evaluation standards.