Development, deployment and scaling of operating room-ready artificial intelligence for real-time surgical decision support

Deep learning for computer vision can be leveraged for interpreting surgical scenes and providing surgeons with real-time guidance to avoid complications. However, neither generalizability nor scalability of computer-vision-based surgical guidance systems have been demonstrated, especially to geographic locations that lack hardware and infrastructure necessary for real-time inference. We propose a new equipment-agnostic framework for real-time use in operating suites. Using laparoscopic cholecystectomy and semantic segmentation models for predicting safe/dangerous ("Go"/"No-Go") zones of dissection as an example use case, this study aimed to develop and test the performance of a novel data pipeline linked to a web-platform that enables real-time deployment from any edge device. To test this infrastructure and demonstrate its scalability and generalizability, lightweight U-Net and SegFormer models were trained on annotated frames from a large and diverse multicenter dataset from 136 institutions, and then tested on a separate prospectively collected dataset. A web-platform was created to enable real-time inference on any surgical video stream, and performance was tested on and optimized for a range of network speeds. The U-Net and SegFormer models respectively achieved mean Dice scores of 57% and 60%, precision 45% and 53%, and recall 82% and 75% for predicting the Go zone, and mean Dice scores of 76% and 76%, precision 68% and 68%, and recall 92% and 92% for predicting the No-Go zone. After optimization of the client-server interaction over the network, we deliver a prediction stream of at least 60 fps and with a maximum round-trip delay of 70 ms for speeds above 8 Mbps. Clinical deployment of machine learning models for surgical guidance is feasible and cost-effective using a generalizable, scalable and equipment-agnostic framework that lacks dependency on hardware with high computing performance or ultra-fast internet connection speed.

The Role of Artificial Intelligence on Tumor Boards: Perspectives from Surgeons, Medical Oncologists and Radiation Oncologists

The integration of multidisciplinary tumor boards (MTBs) is fundamental in delivering state-of-the-art cancer treatment, facilitating collaborative diagnosis and management by a diverse team of specialists. Despite the clear benefits in personalized patient care and improved outcomes, the increasing burden on MTBs due to rising cancer incidence and financial constraints necessitates innovative solutions. The advent of artificial intelligence (AI) in the medical field offers a promising avenue to support clinical decision-making. This review explores the perspectives of clinicians dedicated to the care of cancer patients-surgeons, medical oncologists, and radiation oncologists-on the application of AI within MTBs. Additionally, it examines the role of AI across various clinical specialties involved in cancer diagnosis and treatment. By analyzing both the potential and the challenges, this study underscores how AI can enhance multidisciplinary discussions and optimize treatment plans. The findings highlight the transformative role that AI may play in refining oncology care and sustaining the efficacy of MTBs amidst growing clinical demands.

Development of an Artificial Intelligence Tool for Intraoperative Guidance During Endovascular Abdominal Aortic Aneurysm Repair

BACKGROUND

Adverse events during surgery can occur in part due to errors in visual perception and judgment. Deep learning is a branch of artificial intelligence (AI) that has shown promise in providing real-time intraoperative guidance. This study aims to train and test the performance of a deep learning model that can identify inappropriate landing zones during endovascular aneurysm repair (EVAR).

METHODS

A deep learning model was trained to identify a "No-Go" landing zone during EVAR, defined by coverage of the lowest renal artery by the stent graft. Fluoroscopic images from elective EVAR procedures performed at a single institution and from open-access sources were selected. Annotations of the "No-Go" zone were performed by trained annotators. A 10-fold cross-validation technique was used to evaluate the performance of the model against human annotations. Primary outcomes were intersection-over-union (IoU) and F1 score and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

RESULTS

The AI model was trained using 369 images procured from 110 different patients/videos, including 18 patients/videos (44 images) from open-access sources. For the primary outcomes, IoU and F1 were 0.43 (standard deviation ± 0.29) and 0.53 (±0.32), respectively. For the secondary outcomes, accuracy, sensitivity, specificity, NPV, and PPV were 0.97 (±0.002), 0.51 (±0.34), 0.99 (±0.001). 0.99 (±0.002), and 0.62 (±0.34), respectively.

CONCLUSIONS

AI can effectively identify suboptimal areas of stent deployment during EVAR. Further directions include validating the model on datasets from other institutions and assessing its ability to predict optimal stent graft placement and clinical outcomes.

Training for excellence: using a multimodal videoconferencing platform to coach surgeons and improve intraoperative performance

INTRODUCTION

Continuing Professional Development opportunities for lifelong learning are fundamental to the acquisition of surgical expertise. However, few opportunities exist for longitudinal and structured learning to support the educational needs of surgeons in practice. While peer-to-peer coaching has been proposed as a potential solution, there remains significant logistical constraints and a lack of evidence to support its effectiveness. The purpose of this study is to determine whether the use of remote videoconferencing for video-based coaching improves operative performance.

METHODS

Early career surgeon mentees participated in a remote coaching intervention with a surgeon coach of their choice and using a virtual telestration platform (Zoom Video Communications, San Jose, CA). Feedback was articulated through annotating videos. The coach evaluated mentee performance using a modified Intraoperative Performance Assessment Tool (IPAT). Participants completed a 5-point Likert scale on the educational value of the coaching program.

RESULTS

Eight surgeons were enrolled in the study, six of whom completed a total of two coaching sessions (baseline, 6-month). Subspecialties included endocrine, hepatopancreatobiliary, and surgical oncology. Mean age of participants was 39 (SD 3.3), with mean 5 (SD 4.1) years in independent practice. Total IPAT scores increased significantly from the first session (mean 47.0, SD 1.9) to the second session (mean 51.8, SD 2.1), p = 0.03. Sub-category analysis showed a significant improvement in the Advanced Cognitive Skills domain with a mean of 33.2 (SD 2.5) versus a mean of 37.0 (SD 2.4), p < 0.01. There was no improvement in the psychomotor skills category. Participants agreed or strongly agreed that the coaching programs can improve surgical performance and decision-making (coaches 85%; mentees 100%).

CONCLUSION

Remote surgical coaching is feasible and has educational value using ubiquitous commercially available virtual platforms. Logistical issues with scheduling and finding cases aligned with learning objectives continue to challenge program adoption and widespread dissemination.

A cognitive task analysis of expert surgeons performing the robotic roux-en-y gastric bypass

BACKGROUND

The safe and effective performance of a robotic roux-en-y gastric bypass (RRNY) requires the application of a complex body of knowledge and skills. This qualitative study aims to: (1) define the tasks, subtasks, decision points, and pitfalls in a RRNY; (2) create a framework upon which training and objective evaluation of a RRNY can be based.

METHODS

Hierarchical and cognitive task analyses for a RRNY were performed using semi-structured interviews of expert bariatric surgeons to describe the thoughts and behaviors that exemplify optimal performance. Verbal data was recorded, transcribed verbatim, supplemented with literary and video resources, coded, and thematically analyzed.

RESULTS

A conceptual framework was synthesized based on three book chapters, three articles, eight online videos, nine field observations, and interviews of four subject matter experts (SME). At the time of the interview, SME had practiced a median of 12.5 years and had completed a median of 424 RRNY cases. They estimated the number of RRNY to achieve competence and expertise were 25 cases and 237.5 cases, respectively. After four rounds of inductive analysis, 83 subtasks, 75 potential errors, 60 technical tips, and 15 decision points were identified and categorized into eight major procedural steps (pre-procedure preparation, abdominal entry & port placement, gastric pouch creation, omega loop creation, gastrojejunal anastomosis, jejunojejunal anastomosis, closure of mesenteric defects, leak test & port closure). Nine cognitive behaviors were elucidated (respect for patient-specific factors, tactical modification, adherence to core surgical principles, task completion, judicious technique & instrument selection, visuospatial awareness, team-based communication, anticipation & forward planning, finessed tissue handling).

CONCLUSION

This study defines the key elements that formed the basis of a conceptual framework used by expert bariatric surgeons to perform the RRNY safely and effectively. This framework has the potential to serve as foundational tool for training novices.

Validation of an artificial intelligence platform for the guidance of safe laparoscopic cholecystectomy

BACKGROUND

Many surgical adverse events, such as bile duct injuries during laparoscopic cholecystectomy (LC), occur due to errors in visual perception and judgment. Artificial intelligence (AI) can potentially improve the quality and safety of surgery, such as through real-time intraoperative decision support. GoNoGoNet is a novel AI model capable of identifying safe ("Go") and dangerous ("No-Go") zones of dissection on surgical videos of LC. Yet, it is unknown how GoNoGoNet performs in comparison to expert surgeons. This study aims to evaluate the GoNoGoNet's ability to identify Go and No-Go zones compared to an external panel of expert surgeons.

METHODS

A panel of high-volume surgeons from the SAGES Safe Cholecystectomy Task Force was recruited to draw free-hand annotations on frames of prospectively collected videos of LC to identify the Go and No-Go zones. Expert consensus on the location of Go and No-Go zones was established using Visual Concordance Test pixel agreement. Identification of Go and No-Go zones by GoNoGoNet was compared to expert-derived consensus using mean F1 Dice Score, and pixel accuracy, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV).

RESULTS

A total of 47 frames from 25 LC videos, procured from 3 countries and 9 surgeons, were annotated simultaneously by an expert panel of 6 surgeons and GoNoGoNet. Mean (± standard deviation) F1 Dice score were 0.58 (0.22) and 0.80 (0.12) for Go and No-Go zones, respectively. Mean (± standard deviation) accuracy, sensitivity, specificity, PPV and NPV for the Go zones were 0.92 (0.05), 0.52 (0.24), 0.97 (0.03), 0.70 (0.21), and 0.94 (0.04) respectively. For No-Go zones, these metrics were 0.92 (0.05), 0.80 (0.17), 0.95 (0.04), 0.84 (0.13) and 0.95 (0.05), respectively.

CONCLUSIONS

AI can be used to identify safe and dangerous zones of dissection within the surgical field, with high specificity/PPV for Go zones and high sensitivity/NPV for No-Go zones. Overall, model prediction was better for No-Go zones compared to Go zones. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Artificial Intelligence for Intraoperative Guidance: Using Semantic Segmentation to Identify Surgical Anatomy During Laparoscopic Cholecystectomy

OBJECTIVE

The aim of this study was to develop and evaluate the performance of artificial intelligence (AI) models that can identify safe and dangerous zones of dissection, and anatomical landmarks during laparoscopic cholecystectomy (LC).

SUMMARY BACKGROUND DATA

Many adverse events during surgery occur due to errors in visual perception and judgment leading to misinterpretation of anatomy. Deep learning, a subfield of AI, can potentially be used to provide real-time guidance intraoperatively.

METHODS

Deep learning models were developed and trained to identify safe (Go) and dangerous (No-Go) zones of dissection, liver, gallbladder, and hepatocystic triangle during LC. Annotations were performed by 4 high-volume surgeons. AI predictions were evaluated using 10-fold cross-validation against annotations by expert surgeons. Primary outcomes were intersection- over-union (IOU) and F1 score (validated spatial correlation indices), and secondary outcomes were pixel-wise accuracy, sensitivity, specificity, ± standard deviation.

RESULTS

AI models were trained on 2627 random frames from 290 LC videos, procured from 37 countries, 136 institutions, and 153 surgeons. Mean IOU, F1 score, accuracy, sensitivity, and specificity for the AI to identify Go zones were 0.53 (±0.24), 0.70 (±0.28), 0.94 (±0.05), 0.69 (±0.20). and 0.94 (±0.03), respectively. For No-Go zones, these metrics were 0.71 (±0.29), 0.83 (±0.31), 0.95 (±0.06), 0.80 (±0.21), and 0.98 (±0.05), respectively. Mean IOU for identification of the liver, gallbladder, and hepatocystic triangle were: 0.86 (±0.12), 0.72 (±0.19), and 0.65 (±0.22), respectively.

CONCLUSIONS

AI can be used to identify anatomy within the surgical field. This technology may eventually be used to provide real-time guidance and minimize the risk of adverse events.

Development and evaluation of a virtual knowledge assessment tool for transanal total mesorectal excision

BACKGROUND

Transanal total mesorectal excision (TATME) is difficult to learn and can result in serious complications. Current paradigms for assessing performance and competency may be insufficient. This study aims to develop and provide preliminary validity evidence for a TATME virtual assessment tool (TATME-VAT) to assess the cognitive skills necessary to safely complete TATME dissection.

METHODS

Participants from North America, Europe, Japan and China completed the test via an interactive online platform between 11/2019 and 05/2020. They were grouped into expert, experienced and novice surgeons depending on the number of independently performed TATMEs. TATME-VAT is a 24-item web-based assessment evaluating advanced cognitive skills, designed according to a blueprint from consensus guidelines. Eight items were multiple choice questions. Sixteen items required making annotations on still frames of TATME videos (VCT) and were scored using a validated algorithm derived from experts' responses. Annotation (range 0-100), multiple choice (range 0-100), and overall scores (sum of annotation and multiple-choice scores, normalized to μ = 50 and σ = 10) were reported.

RESULTS

There were significant differences between the expert, experienced, and novice groups for the annotation (p < 0.001), multiple-choice (p < 0.001), and overall scores (p < 0.001). The annotation (p = 0.439) and overall (p = 0.152) scores were similar between the experienced and novice groups. Annotation scores were higher in participants with 51 or more vs. 30-50 vs. less than 30 cases. Scores were also lower in users with a self-reported recent complication vs. those without.

CONCLUSIONS

This study describes the development of an interactive video-based virtual assessment tool for TATME dissection and provides initial validity evidence for its use.

Artificial intelligence-based computer vision in surgery: Recent advances and future perspectives

Technology has advanced surgery, especially minimally invasive surgery (MIS), including laparoscopic surgery and robotic surgery. It has led to an increase in the number of technologies in the operating room. They can provide further information about a surgical procedure, e.g. instrument usage and trajectories. Among these surgery-related technologies, the amount of information extracted from a surgical video captured by an endoscope is especially great. Therefore, the automation of data analysis is essential in surgery to reduce the complexity of the data while maximizing its utility to enable new opportunities for research and development. Computer vision (CV) is the field of study that deals with how computers can understand digital images or videos and seeks to automate tasks that can be performed by the human visual system. Because this field deals with all the processes of real-world information acquisition by computers, the terminology "CV" is extensive, and ranges from hardware for image sensing to AI-based image recognition. AI-based image recognition for simple tasks, such as recognizing snapshots, has advanced and is comparable to humans in recent years. Although surgical video recognition is a more complex and challenging task, if we can effectively apply it to MIS, it leads to future surgical advancements, such as intraoperative decision-making support and image navigation surgery. Ultimately, automated surgery might be realized. In this article, we summarize the recent advances and future perspectives of AI-related research and development in the field of surgery.

Surgical data science and artificial intelligence for surgical education

Surgical data science (SDS) aims to improve the quality of interventional healthcare and its value through the capture, organization, analysis, and modeling of procedural data. As data capture has increased and artificial intelligence (AI) has advanced, SDS can help to unlock augmented and automated coaching, feedback, assessment, and decision support in surgery. We review major concepts in SDS and AI as applied to surgical education and surgical oncology.

Challenges in surgical video annotation

Annotation of surgical video is important for establishing ground truth in surgical data science endeavors that involve computer vision. With the growth of the field over the last decade, several challenges have been identified in annotating spatial, temporal, and clinical elements of surgical video as well as challenges in selecting annotators. In reviewing current challenges, we provide suggestions on opportunities for improvement and possible next steps to enable translation of surgical data science efforts in surgical video analysis to clinical research and practice.

Leveraging Videoconferencing Technology to Augment Surgical Training During a Pandemic

Our objective was to review the use of videoconferencing as a practical tool for remote surgical education and to propose a model to overcome the impact of a pandemic on resident training.

Summary Background Data

In response to the coronavirus disease 2019 pandemic, most institutions and residency programs have been restructured to minimize the number of residents in the hospital as well as their interactions with patients and to promote physical distancing measures. This has resulted in decreased resident operative exposure, responsibility, and autonomy, hindering their educational goals and ability to achieve surgical expertise necessary for independent practice.

Methods

We conducted a narrative review to explore the use of videoconferencing for remote broadcasting of surgical procedures, telecoaching using surgical videos, telesimulation for surgical skills training, and establishing a didactic lecture series.

Results and Conclusions

We present a multimodal approach for using practical videoconferencing tools that provide the means for audiovisual communication to help augment residents' operative experience and limit the impact of self-isolation, redeployment, and limited operative exposure on surgical training.