Surgical data science and artificial intelligence for surgical education

Surgeons vs ChatGPT: Assessment and Feedback Performance Based on Real Surgical Scenarios

INTRODUCTION

Artificial intelligence tools are being progressively integrated into medicine and surgical education. Large language models, such as ChatGPT, could provide relevant feedback aimed at improving surgical skills. The purpose of this study is to assess ChatGPT´s ability to provide feedback based on surgical scenarios.

METHODS

Surgical situations were transformed into texts using a neutral narrative. Texts were evaluated by ChatGPT 4.0 and 3 surgeons (A, B, C) after a brief instruction was delivered: identify errors and provide feedback accordingly. Surgical residents were provided with each of the situations and feedback obtained during the first stage, as written by each surgeon and ChatGPT, and were asked to assess the utility of feedback (FCUR) and its quality (FQ). As control measurement, an Education-Expert (EE) and a Clinical-Expert (CE) were asked to assess FCUR and FQ.

RESULTS

Regarding residents' evaluations, 96.43% of times, outputs provided by ChatGPT were considered useful, comparable to what surgeons' B and C obtained. Assessing FQ, ChatGPT and all surgeons received similar scores. Regarding EE's assessment, ChatGPT obtained a significantly higher FQ score when compared to surgeons A and B (p = 0.019; p = 0.033) with a median score of 8 vs. 7 and 7.5, respectively; and no difference respect surgeon C (score of 8; p = 0.2). Regarding CE´s assessment, surgeon B obtained the highest FQ score while ChatGPT received scores comparable to that of surgeons A and C. When participants were asked to identify the source of the feedback, residents, CE, and EE perceived ChatGPT's outputs as human-provided in 33.9%, 28.5%, and 14.3% of cases, respectively.

CONCLUSION

When given brief written surgical situations, ChatGPT was able to identify errors with a detection rate comparable to that of experienced surgeons and to generate feedback that was considered useful for skill improvement in a surgical context performing as well as surgical instructors across assessments made by general surgery residents, an experienced surgeon, and a nonsurgeon feedback expert.

The transition of surgical simulation training and its learning curve: a bibliometric analysis from 2000 to 2023

BACKGROUND

Proficient surgical skills are essential for surgeons, making surgical training an important part of surgical education. The development of technology promotes the diversification of surgical training types. This study analyzes the changes in surgical training patterns from the perspective of bibliometrics, and applies the learning curves as a measure to demonstrate their teaching ability.

METHOD

Related papers were searched in the Web of Science database using the following formula: TS=((training OR simulation) AND (learning curve) AND (surgical)). Two researchers browsed the papers to ensure that the topics of articles were focused on the impact of surgical simulation training on the learning curve. CiteSpace, VOSviewer and R packages were applied to analyze the publication trends, countries, authors, keywords and references of selected articles.

RESULT

Ultimately, 2461 documents were screened and analyzed. The USA is the most productive and influential country in this field. Surgical endoscopy and other interventional techniques publish the most articles, while surgical endoscopy and other interventional techniques is the most cited journal. Aggarwal Rajesh is the most productive and influential author. Keyword and reference analyses reveal that laparoscopic surgery, robotic surgery, virtue reality (VR) and artificial intelligence (AI) were the hotspots in the field.

CONCLUSION

This study provided a global overview of the current state and future trend in the surgical education field. The study surmised the applicability of different surgical simulation types by comparing and analyzing the learning curves, which is helpful for the development of this field.

Current and future applications of artificial intelligence in surgery: implications for clinical practice and research

Surgeons are skilled at making complex decisions over invasive procedures that can save lives and alleviate pain and avoid complications in patients. The knowledge to make these decisions is accumulated over years of schooling and practice. Their experience is in turn shared with others, also via peer-reviewed articles, which get published in larger and larger amounts every year. In this work, we review the literature related to the use of Artificial Intelligence (AI) in surgery. We focus on what is currently available and what is likely to come in the near future in both clinical care and research. We show that AI has the potential to be a key tool to elevate the effectiveness of training and decision-making in surgery and the discovery of relevant and valid scientific knowledge in the surgical domain. We also address concerns about AI technology, including the inability for users to interpret algorithms as well as incorrect predictions. A better understanding of AI will allow surgeons to use new tools wisely for the benefit of their patients.

A scoping review of artificial intelligence in medical education: BEME Guide No. 84

BACKGROUND

Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research.

METHODS

This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape.

RESULTS

The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education.

CONCLUSION

The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

Remote Monitoring and Artificial Intelligence: Outlook for 2050

Remote monitoring and artificial intelligence will become common and intertwined in anesthesiology by 2050. In the intraoperative period, technology will lead to the development of integrated monitoring systems that will integrate multiple data streams and allow anesthesiologists to track patients more effectively. This will free up anesthesiologists to focus on more complex tasks, such as managing risk and making value-based decisions. This will also enable the continued integration of remote monitoring and control towers having profound effects on coverage and practice models. In the PACU and ICU, the technology will lead to the development of early warning systems that can identify patients who are at risk of complications, enabling early interventions and more proactive care. The integration of augmented reality will allow for better integration of diverse types of data and better decision-making. Postoperatively, the proliferation of wearable devices that can monitor patient vital signs and track their progress will allow patients to be discharged from the hospital sooner and receive care at home. This will require increased use of telemedicine, which will allow patients to consult with doctors remotely. All of these advances will require changes to legal and regulatory frameworks that will enable new workflows that are different from those familiar to today's providers.

Using Artificial Intelligence to Gauge Competency on a Novel Laparoscopic Training System

OBJECTIVE

Laparoscopic surgical skill assessment and machine learning are often inaccessible to low-and-middle-income countries (LMIC). Our team developed a low-cost laparoscopic training system to teach and assess psychomotor skills required in laparoscopic salpingostomy in LMICs. We performed video review using AI to assess global surgical techniques. The objective of this study was to assess the validity of artificial intelligence (AI) generated scoring measures of laparoscopic simulation videos by comparing the accuracy of AI results to human-generated scores.

DESIGN

Seventy-four surgical simulation videos were collected and graded by human participants using a modified OSATS (Objective Structured Assessment of Technical Skills). The videos were then analyzed via AI using 3 different time and distance-based calculations of the laparoscopic instruments including path length, dimensionless jerk, and standard deviation of tool position. Predicted scores were generated using 5-fold cross validation and K-Nearest-Neighbors to train classifiers.

SETTING

Surgical novices and experts from a variety of hospitals in Ethiopia, Cameroon, Kenya, and the United States contributed 74 laparoscopic salpingostomy simulation videos.

RESULTS

Complete accuracy of AI compared to human assessment ranged from 65-77%. There were no statistical differences in rank mean scores for 3 domains, Flow of Operation, Respect for Tissue, and Economy of Motion, while there were significant differences in ratings for Instrument Handling, Overall Performance, and the total summed score of all 5 domains (Summed). Estimated effect sizes were all less than 0.11, indicating very small practical effect. Estimated intraclass correlation coefficient (ICC) of Summed was 0.72 indicating moderate correlation between AI and Human scores.

CONCLUSIONS

Video review using AI technology of global characteristics was similar to that of human review in our laparoscopic training system. Machine learning may help fill an educational gap in LMICs where direct apprenticeship may not be feasible.

MT-FiST: A Multi-Task Fine-Grained Spatial-Temporal Framework for Surgical Action Triplet Recognition

Surgical action triplet recognition plays a significant role in helping surgeons facilitate scene analysis and decision-making in computer-assisted surgeries. Compared to traditional context-aware tasks such as phase recognition, surgical action triplets, comprising the instrument, verb, and target, can offer more comprehensive and detailed information. However, current triplet recognition methods fall short in distinguishing the fine-grained subclasses and disregard temporal correlation in action triplets. In this article, we propose a multi-task fine-grained spatial-temporal framework for surgical action triplet recognition named MT-FiST. The proposed method utilizes a multi-label mutual channel loss, which consists of diversity and discriminative components. This loss function decouples global task features into class-aligned features, enabling the learning of more local details from the surgical scene. The proposed framework utilizes partial shared-parameters LSTM units to capture temporal correlations between adjacent frames. We conducted experiments on the CholecT50 dataset proposed in the MICCAI 2021 Surgical Action Triplet Recognition Challenge. Our framework is evaluated on the private test set of the challenge to ensure fair comparisons. Our model apparently outperformed state-of-the-art models in instrument, verb, target, and action triplet recognition tasks, with mAPs of 82.1% (+4.6%), 51.5% (+4.0%), 45.50% (+7.8%), and 35.8% (+3.1%), respectively. The proposed MT-FiST boosts the recognition of surgical action triplets in a context-aware surgical assistant system, further solving multi-task recognition by effective temporal aggregation and fine-grained features.

Artificial Intelligence in Surgical Training for Kidney Cancer: A Systematic Review of the Literature

The prevalence of renal cell carcinoma (RCC) is increasing due to advanced imaging techniques. Surgical resection is the standard treatment, involving complex radical and partial nephrectomy procedures that demand extensive training and planning. Furthermore, artificial intelligence (AI) can potentially aid the training process in the field of kidney cancer. This review explores how artificial intelligence (AI) can create a framework for kidney cancer surgery to address training difficulties. Following PRISMA 2020 criteria, an exhaustive search of PubMed and SCOPUS databases was conducted without any filters or restrictions. Inclusion criteria encompassed original English articles focusing on AI's role in kidney cancer surgical training. On the other hand, all non-original articles and articles published in any language other than English were excluded. Two independent reviewers assessed the articles, with a third party settling any disagreement. Study specifics, AI tools, methodologies, endpoints, and outcomes were extracted by the same authors. The Oxford Center for Evidence-Based Medicine's evidence levels were employed to assess the studies. Out of 468 identified records, 14 eligible studies were selected. Potential AI applications in kidney cancer surgical training include analyzing surgical workflow, annotating instruments, identifying tissues, and 3D reconstruction. AI is capable of appraising surgical skills, including the identification of procedural steps and instrument tracking. While AI and augmented reality (AR) enhance training, challenges persist in real-time tracking and registration. The utilization of AI-driven 3D reconstruction proves beneficial for intraoperative guidance and preoperative preparation. Artificial intelligence (AI) shows potential for advancing surgical training by providing unbiased evaluations, personalized feedback, and enhanced learning processes. Yet challenges such as consistent metric measurement, ethical concerns, and data privacy must be addressed. The integration of AI into kidney cancer surgical training offers solutions to training difficulties and a boost to surgical education. However, to fully harness its potential, additional studies are imperative.

A methodology for the annotation of surgical videos for supervised machine learning applications

PURPOSE

Surgical data science is an emerging field focused on quantitative analysis of pre-, intra-, and postoperative patient data (Maier-Hein et al. in Med Image Anal 76: 102306, 2022). Data science approaches can decompose complex procedures, train surgical novices, assess outcomes of actions, and create predictive models of surgical outcomes (Marcus et al. in Pituitary 24: 839-853, 2021; Røadsch et al. in Nat Mach Intell, 2022). Surgical videos contain powerful signals of events that may impact patient outcomes. A necessary step before the deployment of supervised machine learning methods is the development of labels for objects and anatomy. We describe a complete method for annotating videos of transsphenoidal surgery.

METHODS

Endoscopic video recordings of transsphenoidal pituitary tumor removal surgeries were collected from a multicenter research collaborative. These videos were anonymized and stored in a cloud-based platform. Videos were uploaded to an online annotation platform. Annotation framework was developed based on a literature review and surgical observations to ensure proper understanding of the tools, anatomy, and steps present. A user guide was developed to trained annotators to ensure standardization.

RESULTS

A fully annotated video of a transsphenoidal pituitary tumor removal surgery was produced. This annotated video included over 129,826 frames. To prevent any missing annotations, all frames were later reviewed by highly experienced annotators and a surgeon reviewer. Iterations to annotated videos allowed for the creation of an annotated video complete with labeled surgical tools, anatomy, and phases. In addition, a user guide was developed for the training of novice annotators, which provides information about the annotation software to ensure the production of standardized annotations.

CONCLUSIONS

A standardized and reproducible workflow for managing surgical video data is a necessary prerequisite to surgical data science applications. We developed a standard methodology for annotating surgical videos that may facilitate the quantitative analysis of videos using machine learning applications. Future work will demonstrate the clinical relevance and impact of this workflow by developing process modeling and outcome predictors.

Innovations in surgical training: exploring the role of artificial intelligence and large language models (LLM)

The landscape of surgical training is rapidly evolving with the advent of artificial intelligence (AI) and its integration into education and simulation. This manuscript aims to explore the potential applications and benefits of AI-assisted surgical training, particularly the use of large language models (LLMs), in enhancing communication, personalizing feedback, and promoting skill development. We discuss the advancements in simulation-based training, AI-driven assessment tools, video-based assessment systems, virtual reality (VR) and augmented reality (AR) platforms, and the potential role of LLMs in the transcription, translation, and summarization of feedback. Despite the promising opportunities presented by AI integration, several challenges must be addressed, including accuracy and reliability, ethical and privacy concerns, bias in AI models, integration with existing training systems, and training and adoption of AI-assisted tools. By proactively addressing these challenges and harnessing the potential of AI, the future of surgical training may be reshaped to provide a more comprehensive, safe, and effective learning experience for trainees, ultimately leading to better patient outcomes. .

Psychometric properties of the persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS)

INTRODUCTION

There are numerous cases where artificial intelligence (AI) can be applied to improve the outcomes of medical education. The extent to which medical practitioners and students are ready to work and leverage this paradigm is unclear in Iran. This study investigated the psychometric properties of a Persian version of the Medical Artificial Intelligence Readiness Scale for Medical Students (MAIRS-MS) developed by Karaca, et al. in 2021. In future studies, the medical AI readiness for Iranian medical students could be investigated using this scale, and effective interventions might be planned and implemented according to the results.

METHODS

In this study, 502 medical students (mean age 22.66(± 2.767); 55% female) responded to the Persian questionnaire in an online survey. The original questionnaire was translated into Persian using a back translation procedure, and all participants completed the demographic component and the entire MAIRS-MS. Internal and external consistencies, factor analysis, construct validity, and confirmatory factor analysis were examined to analyze the collected data. A P ≤ 0.05 was considered as the level of statistical significance.

RESULTS

Four subscales emerged from the exploratory factor analysis (Cognition, Ability, Vision, and Ethics), and confirmatory factor analysis confirmed the four subscales. The Cronbach alpha value for internal consistency was 0.944 for the total scale and 0.886, 0.905, 0.865, and 0.856 for cognition, ability, vision, and ethics, respectively.

CONCLUSIONS

The Persian version of MAIRS-MS was fairly equivalent to the original one regarding the conceptual and linguistic aspects. This study also confirmed the validity and reliability of the Persian version of MAIRS-MS. Therefore, the Persian version can be a suitable and brief instrument to assess Iranian Medical Students' readiness for medical artificial intelligence.

What Is Machine Learning, Artificial Neural Networks and Deep Learning?-Examples of Practical Applications in Medicine

Machine learning (ML), artificial neural networks (ANNs), and deep learning (DL) are all topics that fall under the heading of artificial intelligence (AI) and have gained popularity in recent years. ML involves the application of algorithms to automate decision-making processes using models that have not been manually programmed but have been trained on data. ANNs that are a part of ML aim to simulate the structure and function of the human brain. DL, on the other hand, uses multiple layers of interconnected neurons. This enables the processing and analysis of large and complex databases. In medicine, these techniques are being introduced to improve the speed and efficiency of disease diagnosis and treatment. Each of the AI techniques presented in the paper is supported with an example of a possible medical application. Given the rapid development of technology, the use of AI in medicine shows promising results in the context of patient care. It is particularly important to keep a close eye on this issue and conduct further research in order to fully explore the potential of ML, ANNs, and DL, and bring further applications into clinical use in the future.

Human visual explanations mitigate bias in AI-based assessment of surgeon skills

Artificial intelligence (AI) systems can now reliably assess surgeon skills through videos of intraoperative surgical activity. With such systems informing future high-stakes decisions such as whether to credential surgeons and grant them the privilege to operate on patients, it is critical that they treat all surgeons fairly. However, it remains an open question whether surgical AI systems exhibit bias against surgeon sub-cohorts, and, if so, whether such bias can be mitigated. Here, we examine and mitigate the bias exhibited by a family of surgical AI systems-SAIS-deployed on videos of robotic surgeries from three geographically-diverse hospitals (USA and EU). We show that SAIS exhibits an underskilling bias, erroneously downgrading surgical performance, and an overskilling bias, erroneously upgrading surgical performance, at different rates across surgeon sub-cohorts. To mitigate such bias, we leverage a strategy -TWIX-which teaches an AI system to provide a visual explanation for its skill assessment that otherwise would have been provided by human experts. We show that whereas baseline strategies inconsistently mitigate algorithmic bias, TWIX can effectively mitigate the underskilling and overskilling bias while simultaneously improving the performance of these AI systems across hospitals. We discovered that these findings carry over to the training environment where we assess medical students' skills today. Our study is a critical prerequisite to the eventual implementation of AI-augmented global surgeon credentialing programs, ensuring that all surgeons are treated fairly.

SAGES video acquisition framework-analysis of available OR recording technologies by the SAGES AI task force

BACKGROUND

Surgical video recording provides the opportunity to acquire intraoperative data that can subsequently be used for a variety of quality improvement, research, and educational applications. Various recording devices are available for standard operating room camera systems. Some allow for collateral data acquisition including activities of the OR staff, kinematic measurements (motion of surgical instruments), and recording of the endoscopic video streams. Additional analysis through computer vision (CV), which allows software to understand and perform predictive tasks on images, can allow for automatic phase segmentation, instrument tracking, and derivative performance-geared metrics. With this survey, we summarize available surgical video acquisition technologies and associated performance analysis platforms.

METHODS

In an effort promoted by the SAGES Artificial Intelligence Task Force, we surveyed the available video recording technology companies. Of thirteen companies approached, nine were interviewed, each over an hour-long video conference. A standard set of 17 questions was administered. Questions spanned from data acquisition capacity, quality, and synchronization of video with other data, availability of analytic tools, privacy, and access.

RESULTS

Most platforms (89%) store video in full-HD (1080p) resolution at a frame rate of 30 fps. Most (67%) of available platforms store data in a Cloud-based databank as opposed to institutional hard drives. CV powered analysis is featured in some platforms: phase segmentation in 44% platforms, out of body blurring or tool tracking in 33%, and suture time in 11%. Kinematic data are provided by 22% and perfusion imaging in one device.

CONCLUSION

Video acquisition platforms on the market allow for in depth performance analysis through manual and automated review. Most of these devices will be integrated in upcoming robotic surgical platforms. Platform analytic supplementation, including CV, may allow for more refined performance analysis to surgeons and trainees. Most current AI features are related to phase segmentation, instrument tracking, and video blurring.

Artificial intelligence in improving the outcome of surgical treatment in colorectal cancer

Background

A considerable number of recent research have used artificial intelligence (AI) in the area of colorectal cancer (CRC). Surgical treatment of CRC still remains the most important curative component. Artificial intelligence in CRC surgery is not nearly as advanced as it is in screening (colonoscopy), diagnosis and prognosis, especially due to the increased complexity and variability of structures and elements in all fields of view, as well as a general shortage of annotated video banks for utilization.

Methods

A literature search was made and relevant studies were included in the minireview.

Results

The intraoperative steps which, at this moment, can benefit from AI in CRC are: phase and action recognition, excision plane navigation, endoscopy control, real-time circulation analysis, knot tying, automatic optical biopsy and hyperspectral imaging. This minireview also analyses the current advances in robotic treatment of CRC as well as the present possibility of automated CRC robotic surgery.

Conclusions

The use of AI in CRC surgery is still at its beginnings. The development of AI models capable of reproducing a colorectal expert surgeon's skill, the creation of large and complex datasets and the standardization of surgical colorectal procedures will contribute to the widespread use of AI in CRC surgical treatment.

A Synopsis of Artificial Intelligence and its Applications in Surgery

Artificial intelligence (AI) has made steady in-roads into the healthcare scenario over the last decade. While widespread adoption into clinical practice remains elusive, the outreach of this discipline has progressed beyond the physician scientist, and different facets of this technology have been incorporated into the care of surgical patients. New AI applications are developing at rapid pace, and it is imperative that the general surgeon be aware of the broad utility of AI as applicable in his or her day-to-day practice, so that healthcare continues to remain up-to-date and evidence based. This review provides a broad account of the tip of the AI iceberg and highlights it potential for positively impacting surgical care.

Developing artificial intelligence models for medical student suturing and knot-tying video-based assessment and coaching

BACKGROUND

Early introduction and distributed learning have been shown to improve student comfort with basic requisite suturing skills. The need for more frequent and directed feedback, however, remains an enduring concern for both remote and in-person training. A previous in-person curriculum for our second-year medical students transitioning to clerkships was adapted to an at-home video-based assessment model due to the social distancing implications of COVID-19. We aimed to develop an Artificial Intelligence (AI) model to perform video-based assessment.

METHODS

Second-year medical students were asked to submit a video of a simple interrupted knot on a penrose drain with instrument tying technique after self-training to proficiency. Proficiency was defined as performing the task under two minutes with no critical errors. All the videos were first manually rated with a pass-fail rating and then subsequently underwent task segmentation. We developed and trained two AI models based on convolutional neural networks to identify errors (instrument holding and knot-tying) and provide automated ratings.

RESULTS

A total of 229 medical student videos were reviewed (150 pass, 79 fail). Of those who failed, the critical error distribution was 15 knot-tying, 47 instrument-holding, and 17 multiple. A total of 216 videos were used to train the models after excluding the low-quality videos. A k-fold cross-validation (k = 10) was used. The accuracy of the instrument holding model was 89% with an F-1 score of 74%. For the knot-tying model, the accuracy was 91% with an F-1 score of 54%.

CONCLUSIONS

Medical students require assessment and directed feedback to better acquire surgical skill, but this is often time-consuming and inadequately done. AI techniques can instead be employed to perform automated surgical video analysis. Future work will optimize the current model to identify discrete errors in order to supplement video-based rating with specific feedback.

Robotically assisted augmented reality system for identification of targeted lymph nodes in laparoscopic gynecological surgery: a first step toward the identification of sentinel node : Augmented reality in gynecological surgery

BACKGROUND

To prove feasibility of multimodal and temporal fusion of laparoscopic images with preoperative computed tomography scans for a real-time in vivo-targeted lymph node (TLN) detection during minimally invasive pelvic lymphadenectomy and to validate and enable such guidance for safe and accurate sentinel lymph node dissection, including anatomical landmarks in an experimental model.

METHODS

A measurement campaign determined the most accurate tracking system (UR5-Cobot versus NDI Polaris). The subsequent interventions on two pigs consisted of an identification of artificial TLN and anatomical landmarks without and with augmented reality (AR) assistance. The AR overlay on target structures was quantitatively evaluated. The clinical relevance of our system was assessed via a questionnaire completed by experienced and trainee surgeons.

RESULTS

An AR-based robotic assistance system that performed real-time multimodal and temporal fusion of laparoscopic images with preoperative medical images was developed and tested. It enabled the detection of TLN and their surrounding anatomical structures during pelvic lymphadenectomy. Accuracy of the CT overlay was > 90%, with overflow rates < 6%. When comparing AR to direct vision, we found that scores were significatively higher in AR for all target structures. AR aided both experienced surgeons and trainees, whether it was for TLN, ureter, or vessel identification.

CONCLUSION

This computer-assisted system was reliable, safe, and accurate, and the present achievements represent a first step toward a clinical study.

Multicentric validation of EndoDigest: a computer vision platform for video documentation of the critical view of safety in laparoscopic cholecystectomy

BACKGROUND

A computer vision (CV) platform named EndoDigest was recently developed to facilitate the use of surgical videos. Specifically, EndoDigest automatically provides short video clips to effectively document the critical view of safety (CVS) in laparoscopic cholecystectomy (LC). The aim of the present study is to validate EndoDigest on a multicentric dataset of LC videos.

METHODS

LC videos from 4 centers were manually annotated with the time of the cystic duct division and an assessment of CVS criteria. Incomplete recordings, bailout procedures and procedures with an intraoperative cholangiogram were excluded. EndoDigest leveraged predictions of deep learning models for workflow analysis in a rule-based inference system designed to estimate the time of the cystic duct division. Performance was assessed by computing the error in estimating the manually annotated time of the cystic duct division. To provide concise video documentation of CVS, EndoDigest extracted video clips showing the 2 min preceding and the 30 s following the predicted cystic duct division. The relevance of the documentation was evaluated by assessing CVS in automatically extracted 2.5-min-long video clips.

RESULTS

144 of the 174 LC videos from 4 centers were analyzed. EndoDigest located the time of the cystic duct division with a mean error of 124.0 ± 270.6 s despite the use of fluorescent cholangiography in 27 procedures and great variations in surgical workflows across centers. The surgical evaluation found that 108 (75.0%) of the automatically extracted short video clips documented CVS effectively.

CONCLUSIONS

EndoDigest was robust enough to reliably locate the time of the cystic duct division and efficiently video document CVS despite the highly variable workflows. Training specifically on data from each center could improve results; however, this multicentric validation shows the potential for clinical translation of this surgical data science tool to efficiently document surgical safety.

Computer vision in surgery: from potential to clinical value

Hundreds of millions of operations are performed worldwide each year, and the rising uptake in minimally invasive surgery has enabled fiber optic cameras and robots to become both important tools to conduct surgery and sensors from which to capture information about surgery. Computer vision (CV), the application of algorithms to analyze and interpret visual data, has become a critical technology through which to study the intraoperative phase of care with the goals of augmenting surgeons' decision-making processes, supporting safer surgery, and expanding access to surgical care. While much work has been performed on potential use cases, there are currently no CV tools widely used for diagnostic or therapeutic applications in surgery. Using laparoscopic cholecystectomy as an example, we reviewed current CV techniques that have been applied to minimally invasive surgery and their clinical applications. Finally, we discuss the challenges and obstacles that remain to be overcome for broader implementation and adoption of CV in surgery.

Artificial intelligence prediction of cholecystectomy operative course from automated identification of gallbladder inflammation

BACKGROUND

Operative courses of laparoscopic cholecystectomies vary widely due to differing pathologies. Efforts to assess intra-operative difficulty include the Parkland grading scale (PGS), which scores inflammation from the initial view of the gallbladder on a 1-5 scale. We investigated the impact of PGS on intra-operative outcomes, including laparoscopic duration, attainment of the critical view of safety (CVS), and gallbladder injury. We additionally trained an artificial intelligence (AI) model to identify PGS.

METHODS

One surgeon labeled surgical phases, PGS, CVS attainment, and gallbladder injury in 200 cholecystectomy videos. We used multilevel Bayesian regression models to analyze the PGS's effect on intra-operative outcomes. We trained AI models to identify PGS from an initial view of the gallbladder and compared model performance to annotations by a second surgeon.

RESULTS

Slightly inflamed gallbladders (PGS-2) minimally increased duration, adding 2.7 [95% compatibility interval (CI) 0.3-7.0] minutes to an operation. This contrasted with maximally inflamed gallbladders (PGS-5), where on average 16.9 (95% CI 4.4-33.9) minutes were added, with 31.3 (95% CI 8.0-67.5) minutes added for the most affected surgeon. Inadvertent gallbladder injury occurred in 25% of cases, with a minimal increase in gallbladder injury observed with added inflammation. However, up to a 28% (95% CI - 2, 63) increase in probability of a gallbladder hole during PGS-5 cases was observed for some surgeons. Inflammation had no substantial effect on whether or not a surgeon attained the CVS. An AI model could reliably (Krippendorff's α = 0.71, 95% CI 0.65-0.77) quantify inflammation when compared to a second surgeon (α = 0.82, 95% CI 0.75-0.87).

CONCLUSIONS

An AI model can identify the degree of gallbladder inflammation, which is predictive of cholecystectomy intra-operative course. This automated assessment could be useful for operating room workflow optimization and for targeted per-surgeon and per-resident feedback to accelerate acquisition of operative skills.

Artificial Intelligence in Colorectal Cancer Surgery: Present and Future Perspectives

Artificial intelligence (AI) and computer vision (CV) are beginning to impact medicine. While evidence on the clinical value of AI-based solutions for the screening and staging of colorectal cancer (CRC) is mounting, CV and AI applications to enhance the surgical treatment of CRC are still in their early stage. This manuscript introduces key AI concepts to a surgical audience, illustrates fundamental steps to develop CV for surgical applications, and provides a comprehensive overview on the state-of-the-art of AI applications for the treatment of CRC. Notably, studies show that AI can be trained to automatically recognize surgical phases and actions with high accuracy even in complex colorectal procedures such as transanal total mesorectal excision (TaTME). In addition, AI models were trained to interpret fluorescent signals and recognize correct dissection planes during total mesorectal excision (TME), suggesting CV as a potentially valuable tool for intraoperative decision-making and guidance. Finally, AI could have a role in surgical training, providing automatic surgical skills assessment in the operating room. While promising, these proofs of concept require further development, validation in multi-institutional data, and clinical studies to confirm AI as a valuable tool to enhance CRC treatment.

Artificial Intelligence and Surgery: Ethical Dilemmas and Open Issues

BACKGROUND

Artificial intelligence (AI) applications aiming to support surgical decision-making processes are generating novel threats to ethical surgical care. To understand and address these threats, we summarize the main ethical issues that may arise from applying AI to surgery, starting from the Ethics Guidelines for Trustworthy Artificial Intelligence framework recently promoted by the European Commission.

STUDY DESIGN

A modified Delphi process has been employed to achieve expert consensus.

RESULTS

The main ethical issues that arise from applying AI to surgery, described in detail here, relate to human agency, accountability for errors, technical robustness, privacy and data governance, transparency, diversity, non-discrimination, and fairness. It may be possible to address many of these ethical issues by expanding the breadth of surgical AI research to focus on implementation science. The potential for AI to disrupt surgical practice suggests that formal digital health education is becoming increasingly important for surgeons and surgical trainees.

CONCLUSIONS

A multidisciplinary focus on implementation science and digital health education is desirable to balance opportunities offered by emerging AI technologies and respect for the ethical principles of a patient-centric philosophy.

Artificial Intelligence Methods and Artificial Intelligence-Enabled Metrics for Surgical Education: A Multidisciplinary Consensus

BACKGROUND

Artificial intelligence (AI) methods and AI-enabled metrics hold tremendous potential to advance surgical education. Our objective was to generate consensus guidance on specific needs for AI methods and AI-enabled metrics for surgical education.

STUDY DESIGN

The study included a systematic literature search, a virtual conference, and a 3-round Delphi survey of 40 representative multidisciplinary stakeholders with domain expertise selected through purposeful sampling. The accelerated Delphi process was completed within 10 days. The survey covered overall utility, anticipated future (10-year time horizon), and applications for surgical training, assessment, and feedback. Consensus was agreement among 80% or more respondents. We coded survey questions into 11 themes and descriptively analyzed the responses.

RESULTS

The respondents included surgeons (40%), engineers (15%), affiliates of industry (27.5%), professional societies (7.5%), regulatory agencies (7.5%), and a lawyer (2.5%). The survey included 155 questions; consensus was achieved on 136 (87.7%). The panel listed 6 deliverables each for AI-enhanced learning curve analytics and surgical skill assessment. For feedback, the panel identified 10 priority deliverables spanning 2-year (n = 2), 5-year (n = 4), and 10-year (n = 4) timeframes. Within 2 years, the panel expects development of methods to recognize anatomy in images of the surgical field and to provide surgeons with performance feedback immediately after an operation. The panel also identified 5 essential that should be included in operative performance reports for surgeons.

CONCLUSIONS

The Delphi panel consensus provides a specific, bold, and forward-looking roadmap for AI methods and AI-enabled metrics for surgical education.

Code-free machine learning for object detection in surgical video: a benchmarking, feasibility, and cost study

OBJECTIVE

While the utilization of machine learning (ML) for data analysis typically requires significant technical expertise, novel platforms can deploy ML methods without requiring the user to have any coding experience (termed AutoML). The potential for these methods to be applied to neurosurgical video and surgical data science is unknown.

METHODS

AutoML, a code-free ML (CFML) system, was used to identify surgical instruments contained within each frame of endoscopic, endonasal intraoperative video obtained from a previously validated internal carotid injury training exercise performed on a high-fidelity cadaver model. Instrument-detection performances using CFML were compared with two state-of-the-art ML models built using the Python coding language on the same intraoperative video data set.

RESULTS

The CFML system successfully ingested surgical video without the use of any code. A total of 31,443 images were used to develop this model; 27,223 images were uploaded for training, 2292 images for validation, and 1928 images for testing. The mean average precision on the test set across all instruments was 0.708. The CFML model outperformed two standard object detection networks, RetinaNet and YOLOv3, which had mean average precisions of 0.669 and 0.527, respectively, in analyzing the same data set. Significant advantages to the CFML system included ease of use, relatively low cost, displays of true/false positives and negatives in a user-friendly interface, and the ability to deploy models for further analysis with ease. Significant drawbacks of the CFML model included an inability to view the structure of the trained model, an inability to update the ML model once trained with new examples, and the inability for robust downstream analysis of model performance and error modes.

CONCLUSIONS

This first report describes the baseline performance of CFML in an object detection task using a publicly available surgical video data set as a test bed. Compared with standard, code-based object detection networks, CFML exceeded performance standards. This finding is encouraging for surgeon-scientists seeking to perform object detection tasks to answer clinical questions, perform quality improvement, and develop novel research ideas. The limited interpretability and customization of CFML models remain ongoing challenges. With the further development of code-free platforms, CFML will become increasingly important across biomedical research. Using CFML, surgeons without significant coding experience can perform exploratory ML analyses rapidly and efficiently.

Artificial intelligence assisted display in thoracic surgery: development and possibilities

In this golden age of rapid development of artificial intelligence (AI), researchers and surgeons realized that AI could contribute to healthcare in all aspects, especially in surgery. The popularity of low-dose computed tomography (LDCT) and the improvement of the video-assisted thoracoscopic surgery (VATS) not only bring opportunities for thoracic surgery but also bring challenges on the way forward. Preoperatively localizing lung nodules precisely, intraoperatively identifying anatomical structures accurately, and avoiding complications requires a visual display of individuals’ specific anatomy for surgical simulation and assistance. With the advance of AI-assisted display technologies, including 3D reconstruction/3D printing, virtual reality (VR), augmented reality (AR), and mixed reality (MR), computer tomography (CT) imaging in thoracic surgery has been fully utilized for transforming 2D images to 3D model, which facilitates surgical teaching, planning, and simulation. AI-assisted display based on surgical videos is a new surgical application, which is still in its infancy. Notably, it has potential applications in thoracic surgery education, surgical quality evaluation, intraoperative assistance, and postoperative analysis. In this review, we illustrated the current AI-assisted display applications based on CT in thoracic surgery; focused on the emerging AI applications in thoracic surgery based on surgical videos by reviewing its relevant researches in other surgical fields and anticipate its potential development in thoracic surgery.

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

Most surgeons are skeptical as to the feasibility of autonomous actions in surgery. Interestingly, many examples of autonomous actions already exist and have been around for years. Since the beginning of this millennium, the field of artificial intelligence (AI) has grown exponentially with the development of machine learning (ML), deep learning (DL), computer vision (CV) and natural language processing (NLP). All of these facets of AI will be fundamental to the development of more autonomous actions in surgery, unfortunately, only a limited number of surgeons have or seek expertise in this rapidly evolving field. As opposed to AI in medicine, AI surgery (AIS) involves autonomous movements. Fortuitously, as the field of robotics in surgery has improved, more surgeons are becoming interested in technology and the potential of autonomous actions in procedures such as interventional radiology, endoscopy and surgery. The lack of haptics, or the sensation of touch, has hindered the wider adoption of robotics by many surgeons; however, now that the true potential of robotics can be comprehended, the embracing of AI by the surgical community is more important than ever before. Although current complete surgical systems are mainly only examples of tele-manipulation, for surgeons to get to more autonomously functioning robots, haptics is perhaps not the most important aspect. If the goal is for robots to ultimately become more and more independent, perhaps research should not focus on the concept of haptics as it is perceived by humans, and the focus should be on haptics as it is perceived by robots/computers. This article will discuss aspects of ML, DL, CV and NLP as they pertain to the modern practice of surgery, with a focus on current AI issues and advances that will enable us to get to more autonomous actions in surgery. Ultimately, there may be a paradigm shift that needs to occur in the surgical community as more surgeons with expertise in AI may be needed to fully unlock the potential of AIS in a safe, efficacious and timely manner.