Deep learning-based automatic surgical step recognition in intraoperative videos for transanal total mesorectal excision

Artificial intelligence for intraoperative video analysis in robotic-assisted esophagectomy

BACKGROUND

Robotic-assisted minimally invasive esophagectomy (RAMIE) is a complex surgical procedure for treating esophageal cancer. Artificial intelligence (AI) is an uprising technology with increasing applications in the surgical field. This scoping review aimed to assess the current AI applications in RAMIE, with a focus on intraoperative video analysis.

METHODS

To identify all articles utilizing AI in RAMIE, a comprehensive literature search was performed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-analysis for scoping reviews of Medline and Embase databases and the Cochrane Library. Two independent reviewers assessed articles for quality and inclusion.

RESULTS

One hundred and seventeen articles were identified, of which four were included in the final analysis. Results demonstrated that the main AI applications in RAMIE were intraoperative video assessment and the evaluation of surgical technical skills to evaluate surgical performance. AI was also used for surgical phase recognition to support clinical decision-making through intraoperative guidance and identify key anatomical landmarks. Various deep-learning networks were used to generate AI models, and there was a strong emphasis on using high-quality standardized video frames.

CONCLUSIONS

The use of AI in RAMIE, especially in intraoperative video analysis and surgical phase recognition, is still a relatively new field that should be further explored. The advantages of using AI algorithms to evaluate intraoperative videos in an automated manner may be harnessed to improve technical performance and intraoperative decision-making, achieve a higher quality of surgery, and improve postoperative outcomes.

Artificial Intelligence-Powered Quality Assurance: Transforming Diagnostics, Surgery, and Patient Care-Innovations, Limitations, and Future Directions

Artificial intelligence is rapidly transforming quality assurance in healthcare, driving advancements in diagnostics, surgery, and patient care. This review presents a comprehensive analysis of artificial intelligence integration-particularly convolutional and recurrent neural networks-across key clinical domains, significantly enhancing diagnostic accuracy, surgical performance, and pathology evaluation. Artificial intelligence-based approaches have demonstrated clear superiority over conventional methods: convolutional neural networks achieved 91.56% accuracy in scanner fault detection, surpassing manual inspections; endoscopic lesion detection sensitivity rose from 2.3% to 6.1% with artificial intelligence assistance; and gastric cancer invasion depth classification reached 89.16% accuracy, outperforming human endoscopists by 17.25%. In pathology, artificial intelligence achieved 93.2% accuracy in identifying out-of-focus regions and an F1 score of 0.94 in lymphocyte quantification, promoting faster and more reliable diagnostics. Similarly, artificial intelligence improved surgical workflow recognition with over 81% accuracy and exceeded 95% accuracy in skill assessment classification. Beyond traditional diagnostics and surgical support, AI-powered wearable sensors, drug delivery systems, and biointegrated devices are advancing personalized treatment by optimizing physiological monitoring, automating care protocols, and enhancing therapeutic precision. Despite these achievements, challenges remain in areas such as data standardization, ethical governance, and model generalizability. Overall, the findings underscore artificial intelligence's potential to outperform traditional techniques across multiple parameters, emphasizing the need for continued development, rigorous clinical validation, and interdisciplinary collaboration to fully realize its role in precision medicine and patient safety.

Transanal vs Laparoscopic Total Mesorectal Excision and 3-Year Disease-Free Survival in Rectal Cancer: The TaLaR Randomized Clinical Trial

Importance

Previous studies have demonstrated the advantages of short-term histopathological outcomes and complications associated with transanal total mesorectal excision (TME) compared with laparoscopic TME. However, the long-term oncological outcomes of transanal TME remain ambiguous. This study aims to compare 3-year disease-free survival of transanal TME with laparoscopic TME.

Objective

To evaluate 3-year disease-free survival between transanal TME and laparoscopic TME in patients with rectal cancer.

Design, Setting, and Participants

This randomized, open-label, noninferiority, phase 3 clinical trial was performed in 16 different centers in China. Between April 2016 and June 2021, a total of 1115 patients with clinical stage I to III mid-low rectal cancer were enrolled. The last date of participant follow-up was in June 2024.

Interventions

Participants were randomly assigned in a 1:1 ratio before their surgical procedure to undergo either transanal TME (n = 558) or laparoscopic TME (n = 557).

Main Outcomes and Measures

The primary end point was 3-year disease-free survival, with a noninferiority margin of -10% for the comparison between transanal TME and laparoscopic TME. Secondary outcomes included 3-year overall survival and 3-year local recurrence.

Results

In the primary analysis set, the median patient age was 60 years. A total of 692 male and 397 female patients were included in the analysis. Three-year disease-free survival was 82.1% (97.5% CI, 78.4%-85.8%) for the transanal TME group and 79.4% (97.5% CI, 75.6%-83.4%) for the laparoscopic TME group, with a difference of 2.7% (97.5% CI, -3.0% to 8.1%). The lower tail of a 2-tailed 97.5% CI for the group difference in 3-year disease-free survival was above the noninferiority margin of -10 percentage points. Furthermore, the 3-year local recurrence was 3.6% (95% CI, 2.0%-5.1%) for transanal TME and 4.4% (95% CI, 2.6%-6.1%) for laparoscopic TME. Three-year overall survival was 92.6% (95% CI, 90.4%-94.8%) for transanal TME and 90.7% (95% CI, 88.3%-93.2%) for laparoscopic TME.

Conclusions and Relevance

In patients with mid-low rectal cancer, 3-year disease-free survival for transanal TME was noninferior to that of laparoscopic TME.

Trial Registration

ClinicalTrials.gov Identifier: NCT02966483.

Challenges in multi-centric generalization: phase and step recognition in Roux-en-Y gastric bypass surgery

PURPOSE

Most studies on surgical activity recognition utilizing artificial intelligence (AI) have focused mainly on recognizing one type of activity from small and mono-centric surgical video datasets. It remains speculative whether those models would generalize to other centers.

METHODS

In this work, we introduce a large multi-centric multi-activity dataset consisting of 140 surgical videos (MultiBypass140) of laparoscopic Roux-en-Y gastric bypass (LRYGB) surgeries performed at two medical centers, i.e., the University Hospital of Strasbourg, France (StrasBypass70) and Inselspital, Bern University Hospital, Switzerland (BernBypass70). The dataset has been fully annotated with phases and steps by two board-certified surgeons. Furthermore, we assess the generalizability and benchmark different deep learning models for the task of phase and step recognition in 7 experimental studies: (1) Training and evaluation on BernBypass70; (2) Training and evaluation on StrasBypass70; (3) Training and evaluation on the joint MultiBypass140 dataset; (4) Training on BernBypass70, evaluation on StrasBypass70; (5) Training on StrasBypass70, evaluation on BernBypass70; Training on MultiBypass140, (6) evaluation on BernBypass70 and (7) evaluation on StrasBypass70.

RESULTS

The model's performance is markedly influenced by the training data. The worst results were obtained in experiments (4) and (5) confirming the limited generalization capabilities of models trained on mono-centric data. The use of multi-centric training data, experiments (6) and (7), improves the generalization capabilities of the models, bringing them beyond the level of independent mono-centric training and validation (experiments (1) and (2)).

CONCLUSION

MultiBypass140 shows considerable variation in surgical technique and workflow of LRYGB procedures between centers. Therefore, generalization experiments demonstrate a remarkable difference in model performance. These results highlight the importance of multi-centric datasets for AI model generalization to account for variance in surgical technique and workflows. The dataset and code are publicly available at https://github.com/CAMMA-public/MultiBypass140.

Machine learning perioperative applications in visceral surgery: a narrative review

Artificial intelligence in surgery has seen an expansive rise in research and clinical implementation in recent years, with many of the models being driven by machine learning. In the preoperative setting, machine learning models have been utilized to guide indications for surgery, appropriate timing of operations, calculation of risks and prognostication, along with improving estimations of time and resources required for surgeries. Intraoperative applications that have been demonstrated are visual annotations of the surgical field, automated classification of surgical phases and prediction of intraoperative patient decompensation. Postoperative applications have been studied the most, with most efforts put towards prediction of postoperative complications, recurrence patterns of malignancy, enhanced surgical education and assessment of surgical skill. Challenges to implementation of these models in clinical practice include the need for more quantity and quality of standardized data to improve model performance, sufficient resources and infrastructure to train and use machine learning, along with addressing ethical and patient acceptance considerations.

Deep learning-based surgical step recognition for laparoscopic right-sided colectomy

PURPOSE

Understanding the complex anatomy and surgical steps involved in laparoscopic right-sided colectomy (LAP-RC) is essential for standardizing the surgical procedure. Deep-learning (DL)-based computer vision can achieve this. This study aimed to develop a step recognition model for LAP-RC using a dataset of surgical videos with annotated step information and evaluate its recognition performance.

METHODS

This single-center retrospective study utilized a video dataset of laparoscopic ileocecal resection (LAP-ICR) and laparoscopic right-sided hemicolectomy (LAP-RHC) for right-sided colon cancer performed between January 2018 and March 2022. The videos were split into still images, which were divided into training, validation, and test sets using 66%, 17%, and 17% of the data, respectively. Videos were manually classified into eight main steps: 1) medial mobilization, 2) central vascular ligation, 3) dissection of the superior mesenteric vein, 4) retroperitoneal mobilization, 5) lateral mobilization, 6) cranial mobilization, 7) mesocolon resection, and 8) intracorporeal anastomosis. In a simpler version, consecutive surgical steps were combined, resulting in five steps. Precision, recall, F1 scores, and overall accuracy were assessed to evaluate the model's performance in the surgical step classification task.

RESULTS

Seventy-eight patients were included; LAP-ICR and LAP-RHC were performed in 35 (44%) and 44 (56%) patients, respectively. The overall accuracy was 72.1% and 82.9% for the eight-step and combined five-step classification tasks, respectively.

CONCLUSIONS

The automatic surgical step-recognition model for LAP-RCs, developed using a DL algorithm, exhibited a fairly high classification performance. A model that understands the complex steps of LAP-RC will aid the standardization of the surgical procedure.

Real-time segmentation of biliary structure in pure laparoscopic donor hepatectomy

Pure laparoscopic donor hepatectomy (PLDH) has become a standard practice for living donor liver transplantation in expert centers. Accurate understanding of biliary structures is crucial during PLDH to minimize the risk of complications. This study aims to develop a deep learning-based segmentation model for real-time identification of biliary structures, assisting surgeons in determining the optimal transection site during PLDH. A single-institution retrospective feasibility analysis was conducted on 30 intraoperative videos of PLDH. All videos were selected for their use of the indocyanine green near-infrared fluorescence technique to identify biliary structure. From the analysis, 10 representative frames were extracted from each video specifically during the bile duct division phase, resulting in 300 frames. These frames underwent pixel-wise annotation to identify biliary structures and the transection site. A segmentation task was then performed using a DeepLabV3+ algorithm, equipped with a ResNet50 encoder, focusing on the bile duct (BD) and anterior wall (AW) for transection. The model's performance was evaluated using the dice similarity coefficient (DSC). The model predicted biliary structures with a mean DSC of 0.728 ± 0.01 for BD and 0.429 ± 0.06 for AW. Inference was performed at a speed of 15.3 frames per second, demonstrating the feasibility of real-time recognition of anatomical structures during surgery. The deep learning-based semantic segmentation model exhibited promising performance in identifying biliary structures during PLDH. Future studies should focus on validating the clinical utility and generalizability of the model and comparing its efficacy with current gold standard practices to better evaluate its potential clinical applications.

Impact of artificial intelligence in the management of esophageal, gastric and colorectal malignancies

The incidence of gastrointestinal malignancies has increased over the past decade at an alarming rate. Colorectal and gastric cancers are the third and fifth most commonly diagnosed cancers worldwide but are cited as the second and third leading causes of mortality. Early institution of appropriate therapy from timely diagnosis can optimize patient outcomes. Artificial intelligence (AI)-assisted diagnostic, prognostic, and therapeutic tools can assist in expeditious diagnosis, treatment planning/response prediction, and post-surgical prognostication. AI can intercept neoplastic lesions in their primordial stages, accurately flag suspicious and/or inconspicuous lesions with greater accuracy on radiologic, histopathological, and/or endoscopic analyses, and eliminate over-dependence on clinicians. AI-based models have shown to be on par, and sometimes even outperformed experienced gastroenterologists and radiologists. Convolutional neural networks (state-of-the-art deep learning models) are powerful computational models, invaluable to the field of precision oncology. These models not only reliably classify images, but also accurately predict response to chemotherapy, tumor recurrence, metastasis, and survival rates post-treatment. In this systematic review, we analyze the available evidence about the diagnostic, prognostic, and therapeutic utility of artificial intelligence in gastrointestinal oncology.

Automated artificial intelligence-based phase-recognition system for esophageal endoscopic submucosal dissection (with video)

BACKGROUND AND AIMS

Endoscopic submucosal dissection (ESD) for superficial esophageal cancer is a multistep treatment involving several endoscopic processes. Although analyzing each phase separately is worthwhile, it is not realistic in practice owing to the need for considerable manpower. To solve this problem, we aimed to establish a state-of-the-art artificial intelligence (AI)-based system, specifically, an automated phase-recognition system that can automatically identify each endoscopic phase based on video images.

METHODS

Ninety-four videos of ESD procedures for superficial esophageal cancer were evaluated in this single-center study. A deep neural network-based phase-recognition system was developed in an automated manner to recognize each of the endoscopic phases. The system was trained with the use of videos that were annotated and verified by 2 GI endoscopists.

RESULTS

The overall accuracy of the AI model for automated phase recognition was 90%, and the average precision, recall, and F value rates were 91%, 90%, and 90%, respectively. Two representative ESD videos predicted by the model indicated the usability of AI in clinical practice.

CONCLUSIONS

We demonstrated that an AI-based automated phase-recognition system for esophageal ESD can be established with high accuracy. To the best of our knowledge, this is the first report on automated recognition of ESD treatment phases. Because this system enabled a detailed analysis of phases, collecting large volumes of data in the future may help to identify quality indicators for treatment techniques and uncover unmet medical needs that necessitate the creation of new treatment methods and devices.

Decoding surgical skill: an objective and efficient algorithm for surgical skill classification based on surgical gesture features -experimental studies

BACKGROUND

Various surgical skills lead to differences in patient outcomes and identifying poorly skilled surgeons with constructive feedback contributes to surgical quality improvement. The aim of the study was to develop an algorithm for evaluating surgical skills in laparoscopic cholecystectomy based on the features of elementary functional surgical gestures (Surgestures).

MATERIALS AND METHODS

Seventy-five laparoscopic cholecystectomy videos were collected from 33 surgeons in five hospitals. The phase of mobilization hepatocystic triangle and gallbladder dissection from the liver bed of each video were annotated with 14 Surgestures. The videos were grouped into competent and incompetent based on the quantiles of modified global operative assessment of laparoscopic skills (mGOALS). Surgeon-related information, clinical data, and intraoperative events were analyzed. Sixty-three Surgesture features were extracted to develop the surgical skill classification algorithm. The area under the receiver operating characteristic curve of the classification and the top features were evaluated.

RESULTS

Correlation analysis revealed that most perioperative factors had no significant correlation with mGOALS scores. The incompetent group has a higher probability of cholecystic vascular injury compared to the competent group (30.8 vs 6.1%, P =0.004). The competent group demonstrated fewer inefficient Surgestures, lower shift frequency, and a larger dissection-exposure ratio of Surgestures during the procedure. The area under the receiver operating characteristic curve of the classification algorithm achieved 0.866. Different Surgesture features contributed variably to overall performance and specific skill items.

CONCLUSION

The computer algorithm accurately classified surgeons with different skill levels using objective Surgesture features, adding insight into designing automatic laparoscopic surgical skill assessment tools with technical feedback.

Automatic surgical phase recognition-based skill assessment in laparoscopic distal gastrectomy using multicenter videos

BACKGROUND

Gastric surgery involves numerous surgical phases; however, its steps can be clearly defined. Deep learning-based surgical phase recognition can promote stylization of gastric surgery with applications in automatic surgical skill assessment. This study aimed to develop a deep learning-based surgical phase-recognition model using multicenter videos of laparoscopic distal gastrectomy, and examine the feasibility of automatic surgical skill assessment using the developed model.

METHODS

Surgical videos from 20 hospitals were used. Laparoscopic distal gastrectomy was defined and annotated into nine phases and a deep learning-based image classification model was developed for phase recognition. We examined whether the developed model's output, including the number of frames in each phase and the adequacy of the surgical field development during the phase of supra-pancreatic lymphadenectomy, correlated with the manually assigned skill assessment score.

RESULTS

The overall accuracy of phase recognition was 88.8%. Regarding surgical skill assessment based on the number of frames during the phases of lymphadenectomy of the left greater curvature and reconstruction, the number of frames in the high-score group were significantly less than those in the low-score group (829 vs. 1,152, P < 0.01; 1,208 vs. 1,586, P = 0.01, respectively). The output score of the adequacy of the surgical field development, which is the developed model's output, was significantly higher in the high-score group than that in the low-score group (0.975 vs. 0.970, P = 0.04).

CONCLUSION

The developed model had high accuracy in phase-recognition tasks and has the potential for application in automatic surgical skill assessment systems.

Development of a deep learning model for safe direct optical trocar insertion in minimally invasive surgery: an innovative method to prevent trocar injuries

BACKGROUND

Direct optical trocar insertion is a common procedure in laparoscopic minimally invasive surgery. However, misinterpretations of the abdominal wall anatomy can lead to severe complications. Artificial intelligence has shown promise in surgical endoscopy, particularly in the employment of deep learning models for anatomical landmark identification. This study aimed to integrate a deep learning model with an alarm system algorithm for the precise detection of abdominal wall layers during trocar placement.

METHOD

Annotated bounding boxes and assigned classes were based on the six layers of the abdominal wall: subcutaneous, anterior rectus sheath, rectus muscle, posterior rectus sheath, peritoneum, and abdominal cavity. The cutting-edge YOLOv8 model was combined with a deep learning detector to train the dataset. The model was trained on still images and inferenced on laparoscopic videos to ensure real-time detection in the operating room. The alarm system was activated upon recognizing the peritoneum and abdominal cavity layers. We assessed the model's performance using mean average precision (mAP), precision, and recall metrics.

RESULTS

A total of 3600 images were captured from 89 laparoscopic video cases. The proposed model was trained on 3000 images, validated with a set of 200 images, and tested on a separate set of 400 images. The results from the test set were 95.8% mAP, 89.8% precision, and 91.7% recall. The alarm system was validated and accepted by experienced surgeons at our institute.

CONCLUSION

We demonstrated that deep learning has the potential to assist surgeons during direct optical trocar insertion. During trocar insertion, the proposed model promptly detects precise landmark references in real-time. The integration of this model with the alarm system enables timely reminders for surgeons to tilt the scope accordingly. Consequently, the implementation of the framework provides the potential to mitigate complications associated with direct optical trocar placement, thereby enhancing surgical safety and outcomes.

Artificial Intelligence in Surgery and Its Potential for Gastric Cancer

Artificial intelligence (AI) has made significant progress in recent years, and many medical fields are attempting to introduce AI technology into clinical practice. Currently, much research is being conducted to evaluate that AI can be incorporated into surgical procedures to make them safer and more efficient, subsequently to obtain better outcomes for patients. In this paper, we review basic AI research regarding surgery and discuss the potential for implementing AI technology in gastric cancer surgery. At present, research and development is focused on AI technologies that assist the surgeon's understandings and judgment during surgery, such as anatomical navigation. AI systems are also being developed to recognize in which the surgical phase is ongoing. Such a surgical phase recognition systems is considered for effective storage of surgical videos and education, in the future, for use in systems to objectively evaluate the skill of surgeons. At this time, it is not considered practical to let AI make intraoperative decisions or move forceps automatically from an ethical standpoint, too. At present, AI research on surgery has various limitations, and it is desirable to develop practical systems that will truly benefit clinical practice in the future.

Evolution of a surgical system using deep learning in minimally invasive surgery (Review)

Recently, artificial intelligence (AI) has been applied in various fields due to the development of new learning methods, such as deep learning, and the marked progress in computational processing speed. AI is also being applied in the medical field for medical image recognition and omics analysis of genomes and other data. Recently, AI applications for videos of minimally invasive surgeries have also advanced, and studies on such applications are increasing. In the present review, studies that focused on the following topics were selected: i) Organ and anatomy identification, ii) instrument identification, iii) procedure and surgical phase recognition, iv) surgery-time prediction, v) identification of an appropriate incision line, and vi) surgical education. The development of autonomous surgical robots is also progressing, with the Smart Tissue Autonomous Robot (STAR) and RAVEN systems being the most reported developments. STAR, in particular, is currently being used in laparoscopic imaging to recognize the surgical site from laparoscopic images and is in the process of establishing an automated suturing system, albeit in animal experiments. The present review examined the possibility of fully autonomous surgical robots in the future.

The relationship between the esophageal endoscopic submucosal dissection technical difficulty and its intraoperative process

BACKGROUND

Estimating the esophageal endoscopic submucosal dissection (ESD) technical difficulty is important to reduce complications. Endoscopic duration is one of the related factors to a technical difficulty. The relationship between the esophageal ESD technical difficulty and its intraoperative process was analyzed as a first step toward automatic technical difficulty recognition using artificial intelligence.

METHODS

This study enrolled 75 patients with superficial esophageal cancer who underwent esophageal ESD. The technical difficulty score was established, which consisted of three factors, including total procedure duration, en bloc resection, and complications. Additionally, technical difficulty-related factors, which were perioperative factors that included the intraoperative process, were investigated.

RESULTS

Eight (11%) patients were allocated to high difficulty, whereas 67 patients (89%) were allocated to low difficulty. The intraoperative process, which was shown as the extension of each endoscopic phase, was significantly related to a technical difficulty. The area under the curve (AUC) values were higher at all the phase duration than at the clinical characteristics. Submucosal dissection phase (AUC 0.902; 95% confidence intervals (CI) 0.752-1.000), marking phase (AUC 0.827; 95% CI 0.703-0.951), and early phase which was defined as the duration from the start of marking to the end of submucosal injection (AUC 0.847; 95% CI 0.701-0.992) were significantly related to technical difficulty.

CONCLUSIONS

The intraoperative process, particularly early phase, was strongly associated with esophageal ESD technical difficulty. This study demonstrated the potential for automatic evaluation of esophageal ESD technical difficulty when combined with an AI-based automatic phase evaluation system.

Role of artificial intelligence in risk prediction, prognostication, and therapy response assessment in colorectal cancer: current state and future directions

Artificial Intelligence (AI) is a branch of computer science that utilizes optimization, probabilistic and statistical approaches to analyze and make predictions based on a vast amount of data. In recent years, AI has revolutionized the field of oncology and spearheaded novel approaches in the management of various cancers, including colorectal cancer (CRC). Notably, the applications of AI to diagnose, prognosticate, and predict response to therapy in CRC, is gaining traction and proving to be promising. There have also been several advancements in AI technologies to help predict metastases in CRC and in Computer-Aided Detection (CAD) Systems to improve miss rates for colorectal neoplasia. This article provides a comprehensive review of the role of AI in predicting risk, prognosis, and response to therapies among patients with CRC.

Automatic surgical phase recognition in laparoscopic inguinal hernia repair with artificial intelligence

BACKGROUND

Because of the complexity of the intra-abdominal anatomy in the posterior approach, a longer learning curve has been observed in laparoscopic transabdominal preperitoneal (TAPP) inguinal hernia repair. Consequently, automatic tools using artificial intelligence (AI) to monitor TAPP procedures and assess learning curves are required. The primary objective of this study was to establish a deep learning-based automated surgical phase recognition system for TAPP. A secondary objective was to investigate the relationship between surgical skills and phase duration.

METHODS

This study enrolled 119 patients who underwent the TAPP procedure. The surgical videos were annotated (delineated in time) and split into seven surgical phases (preparation, peritoneal flap incision, peritoneal flap dissection, hernia dissection, mesh deployment, mesh fixation, peritoneal flap closure, and additional closure). An AI model was trained to automatically recognize surgical phases from videos. The relationship between phase duration and surgical skills were also evaluated.

RESULTS

A fourfold cross-validation was used to assess the performance of the AI model. The accuracy was 88.81 and 85.82%, in unilateral and bilateral cases, respectively. In unilateral hernia cases, the duration of peritoneal incision (p = 0.003) and hernia dissection (p = 0.014) detected via AI were significantly shorter for experts than for trainees.

CONCLUSION

An automated surgical phase recognition system was established for TAPP using deep learning with a high accuracy. Our AI-based system can be useful for the automatic monitoring of surgery progress, improving OR efficiency, evaluating surgical skills and video-based surgical education. Specific phase durations detected via the AI model were significantly associated with the surgeons' learning curve.

Artificial Intelligence in Colorectal Cancer Surgery: Present and Future Perspectives

Artificial intelligence (AI) and computer vision (CV) are beginning to impact medicine. While evidence on the clinical value of AI-based solutions for the screening and staging of colorectal cancer (CRC) is mounting, CV and AI applications to enhance the surgical treatment of CRC are still in their early stage. This manuscript introduces key AI concepts to a surgical audience, illustrates fundamental steps to develop CV for surgical applications, and provides a comprehensive overview on the state-of-the-art of AI applications for the treatment of CRC. Notably, studies show that AI can be trained to automatically recognize surgical phases and actions with high accuracy even in complex colorectal procedures such as transanal total mesorectal excision (TaTME). In addition, AI models were trained to interpret fluorescent signals and recognize correct dissection planes during total mesorectal excision (TME), suggesting CV as a potentially valuable tool for intraoperative decision-making and guidance. Finally, AI could have a role in surgical training, providing automatic surgical skills assessment in the operating room. While promising, these proofs of concept require further development, validation in multi-institutional data, and clinical studies to confirm AI as a valuable tool to enhance CRC treatment.

Automated Surgical-Phase Recognition for Robot-Assisted Minimally Invasive Esophagectomy Using Artificial Intelligence

BACKGROUND

Although a number of robot-assisted minimally invasive esophagectomy (RAMIE) procedures have been performed due to three-dimensional field of view, image stabilization, and flexible joint function, both the surgeons and surgical teams require proficiency. This study aimed to establish an artificial intelligence (AI)-based automated surgical-phase recognition system for RAMIE by analyzing robotic surgical videos.

METHODS

This study enrolled 31 patients who underwent RAMIE. The videos were annotated into the following nine surgical phases: preparation, lower mediastinal dissection, upper mediastinal dissection, azygos vein division, subcarinal lymph node dissection (LND), right recurrent laryngeal nerve (RLN) LND, left RLN LND, esophageal transection, and post-dissection to completion of surgery to train the AI for automated phase recognition. An additional phase ("no step") was used to indicate video sequences upon removal of the camera from the thoracic cavity. All the patients were divided into two groups, namely, early period (20 patients) and late period (11 patients), after which the relationship between the surgical-phase duration and the surgical periods was assessed.

RESULTS

Fourfold cross validation was applied to evaluate the performance of the current model. The AI had an accuracy of 84%. The preparation (p = 0.012), post-dissection to completion of surgery (p = 0.003), and "no step" (p < 0.001) phases predicted by the AI were significantly shorter in the late period than in the early period.

CONCLUSIONS

A highly accurate automated surgical-phase recognition system for RAMIE was established using deep learning. Specific phase durations were significantly associated with the surgical period at the authors' institution.

Artificial intelligence software available for medical devices: surgical phase recognition in laparoscopic cholecystectomy

Background

Surgical process modeling automatically identifies surgical phases, and further improvement in recognition accuracy is expected with deep learning. Surgical tool or time series information has been used to improve the recognition accuracy of a model. However, it is difficult to collect this information continuously intraoperatively. The present study aimed to develop a deep convolution neural network (CNN) model that correctly identifies the surgical phase during laparoscopic cholecystectomy (LC).

Methods

We divided LC into six surgical phases (P1–P6) and one redundant phase (P0). We prepared 115 LC videos and converted them to image frames at 3 fps. Three experienced doctors labeled the surgical phases in all image frames. Our deep CNN model was trained with 106 of the 115 annotation datasets and was evaluated with the remaining datasets. By depending on both the prediction probability and frequency for a certain period, we aimed for highly accurate surgical phase recognition in the operation room.

Results

Nine full LC videos were converted into image frames and were fed to our deep CNN model. The average accuracy, precision, and recall were 0.970, 0.855, and 0.863, respectively.

Conclusion

The deep learning CNN model in this study successfully identified both the six surgical phases and the redundant phase, P0, which may increase the versatility of the surgical process recognition model for clinical use. We believe that this model can be used in artificial intelligence for medical devices. The degree of recognition accuracy is expected to improve with developments in advanced deep learning algorithms.