Automatic tracking of laparoscopic instruments for autonomous control of a cameraman robot

Biophysical modeling and artificial intelligence for quantitative assessment of anastomotic blood supply in laparoscopic low anterior rectal resection

PURPOSE

Fluorescence imaging is critical for intraoperative intestinal perfusion assessment in colorectal surgery, yet its clinical adoption remains limited by subjective interpretation and lack of quantitative standards. This study introduces an integrated approach combining fluorescence curve analysis, biophysical modeling, and machine learning to improve intraoperative perfusion assessment.

METHODS

Laparoscopic fluorescence videos from 68 low rectal cancer patients were analyzed, with 1,263 measurement points (15-20 per case) selected along colonic bands. Fluorescence intensity dynamics were extracted via color space transformation, video stabilization and image registration, then modeled using the Random Sample Consensus (RANSAC) algorithm and nonlinear least squares fitting to derive biophysical parameters. Three clinicians independently scored perfusion quality (0-100 scale) using morphological features and biophysical metrics. An XGBoost model was trained on these parameters to automate scoring.

RESULTS

The model achieved superior test performance, with a root mean square error (RMSE) of 2.47, a mean absolute error (MAE) of 1.99, and an R2 of 97.2%, outperforming conventional time-factor analyses. It demonstrated robust generalizability, showing no statistically significant prediction differences across age, diabetes, or smoking subgroups (P > 0.05). Clinically, low perfusion scores in distal anastomotic regions were significantly associated with postoperative complications (P < 0.001), validating the scoring system's clinical relevance and discriminative capacity. The automated software we developed completed analyses within 2 min, enabling rapid intraoperative assessment.

CONCLUSION

This framework synergistically enhances surgical evaluation through three innovations: (1) Biophysical modeling quantifies perfusion dynamics beyond time-based parameters; (2) Machine learning integrates multimodal data for surgeon-level accuracy; (3) Automated workflow enables practical clinical translation. By addressing limitations of visual assessment through quantitative, rapid, and generalizable analysis, this method advances intraoperative perfusion monitoring and decision-making in colorectal surgery.

Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art

In the field of computer- and robot-assisted minimally invasive surgery, enormous progress has been made in recent years based on the recognition of surgical instruments in endoscopic images and videos. In particular, the determination of the position and type of instruments is of great interest. Current work involves both spatial and temporal information, with the idea that predicting the movement of surgical tools over time may improve the quality of the final segmentations. The provision of publicly available datasets has recently encouraged the development of new methods, mainly based on deep learning. In this review, we identify and characterize datasets used for method development and evaluation and quantify their frequency of use in the literature. We further present an overview of the current state of research regarding the segmentation and tracking of minimally invasive surgical instruments in endoscopic images and videos. The paper focuses on methods that work purely visually, without markers of any kind attached to the instruments, considering both single-frame semantic and instance segmentation approaches, as well as those that incorporate temporal information. The publications analyzed were identified through the platforms Google Scholar, Web of Science, and PubMed. The search terms used were "instrument segmentation", "instrument tracking", "surgical tool segmentation", and "surgical tool tracking", resulting in a total of 741 articles published between 01/2015 and 07/2023, of which 123 were included using systematic selection criteria. A discussion of the reviewed literature is provided, highlighting existing shortcomings and emphasizing the available potential for future developments.

Robotics in laparoscopic surgery - A review

Because of the increasing use of laparoscopic surgeries, robotic technologies have been developed to overcome the challenges these surgeries impose on surgeons. This paper presents an overview of the current state of surgical robots used in laparoscopic surgeries. Four main categories were discussed: handheld laparoscopic devices, laparoscope positioning robots, master–slave teleoperated systems with dedicated consoles, and robotic training systems. A generalized control block diagram is developed to demonstrate the general control scheme for each category of surgical robots. In order to review these robotic technologies, related published works were investigated and discussed. Detailed discussions and comparison tables are presented to compare their effectiveness in laparoscopic surgeries. Each of these technologies has proved to be beneficial in laparoscopic surgeries.

Splitting the imaging system from the laparoendoscopic single site surgery device in endoscopic uniportal surgery: ZEUS project

The use of a rigid tube endoscope (RTE) through a laparoendoscopic single site surgery device for intracavitary visualization during endoscopic uniportal surgery (EUS) is directly related to the fulcrum effect and physical competition (externally between surgeons and on instrumentation through the single port). These facts may have precluded a wider spread of EUS. In an attempt to separate the imaging system from the laparoendoscopic single site surgery device, a tiered project (3 models) that can help overcome these drawbacks is detailed. ZEUS1, a new non-RTE imaging system device for EUS was successfully tested in EUS surgery in a pig model. Still under investigation, ZEUS2 becomes a wired structure that serves as the basis for an endoscopic microcamera with intracavitary pan-tilt-zoom functions. ZEUS3 uses the inner ring of the surgical wound protector as a microchamber crown. Further studies are needed to materialize the ZEUS2 and ZEUS3 models and confirm their efficacy and safety.

Splitting the imaging system from the laparoendoscopic single site surgery device in endoscopic uniportal surgery: ZEUS project

The use of a rigid tube endoscope (RTE) through a laparoendoscopic single site surgery device for intracavitary visualization during endoscopic uniportal surgery (EUS) is directly related to the fulcrum effect and physical competition (externally between surgeons and on instrumentation through the single port). These facts may have precluded a wider spread of EUS. In an attempt to separate the imaging system from the laparoendoscopic single site surgery device, a tiered project (3 models) that can help overcome these drawbacks is detailed. ZEUS1, a new non-RTE imaging system device for EUS was successfully tested in EUS surgery in a pig model. Still under investigation, ZEUS2 becomes a wired structure that serves as the basis for an endoscopic microcamera with intracavitary pan-tilt-zoom functions. ZEUS3 uses the inner ring of the surgical wound protector as a microchamber crown. Further studies are needed to materialize the ZEUS2 and ZEUS3 models and confirm their efficacy and safety.

"Hey Siri! Perform a type 3 hysterectomy. Please watch out for the ureter!" What is autonomous surgery and what are the latest developments?

As a result of major advances in deep learning algorithms and computer processing power, there have been important developments in the fields of medicine and robotics. Although fully autonomous surgery systems where human impact will be minimized are still a long way off, systems with partial autonomy have gradually entered clinical use. In this review, articles on autonomous surgery classified and summarized, with the aim of informing the reader about questions such as "What is autonomic surgery?" and in which areas studies are progressing.

Comparative validation of multi-instance instrument segmentation in endoscopy: Results of the ROBUST-MIS 2019 challenge

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).