Tracking and visualization of the sensing area for a tethered laparoscopic gamma probe

SurgiTrack: Fine-grained multi-class multi-tool tracking in surgical videos

Accurate tool tracking is essential for the success of computer-assisted intervention. Previous efforts often modeled tool trajectories rigidly, overlooking the dynamic nature of surgical procedures, especially tracking scenarios like out-of-body and out-of-camera views. Addressing this limitation, the new CholecTrack20 dataset provides detailed labels that account for multiple tool trajectories in three perspectives: (1) intraoperative, (2) intracorporeal, and (3) visibility, representing the different types of temporal duration of tool tracks. These fine-grained labels enhance tracking flexibility but also increase the task complexity. Re-identifying tools after occlusion or re-insertion into the body remains challenging due to high visual similarity, especially among tools of the same category. This work recognizes the critical role of the tool operators in distinguishing tool track instances, especially those belonging to the same tool category. The operators' information are however not explicitly captured in surgical videos. We therefore propose SurgiTrack, a novel deep learning method that leverages YOLOv7 for precise tool detection and employs an attention mechanism to model the originating direction of the tools, as a proxy to their operators, for tool re-identification. To handle diverse tool trajectory perspectives, SurgiTrack employs a harmonizing bipartite matching graph, minimizing conflicts and ensuring accurate tool identity association. Experimental results on CholecTrack20 demonstrate SurgiTrack's effectiveness, outperforming baselines and state-of-the-art methods with real-time inference capability. This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries.

Occlusion-robust markerless surgical instrument pose estimation

The estimation of the pose of surgical instruments is important in Robot-assisted Minimally Invasive Surgery (RMIS) to assist surgical navigation and enable autonomous robotic task execution. The performance of current instrument pose estimation methods deteriorates significantly in the presence of partial tool visibility, occlusions, and changes in the surgical scene. In this work, a vision-based framework is proposed for markerless estimation of the 6DoF pose of surgical instruments. To deal with partial instrument visibility, a keypoint object representation is used and stable and accurate instrument poses are computed using a PnP solver. To boost the learning process of the model under occlusion, a new mask-based data augmentation approach has been proposed. To validate the model, a dataset for instrument pose estimation with highly accurate ground truth data has been generated using different surgical robotic instruments. The proposed network can achieve submillimeter accuracy and the experimental results verify its generalisability to different shapes of occlusion.

Surgical radioguidance with beta-emitting radionuclides; challenges and possibilities: A position paper by the EANM

Radioguidance that makes use of β-emitting radionuclides is gaining in popularity and could have potential to strengthen the range of existing radioguidance techniques. While there is a strong tendency to develop new PET radiotracers, due to favorable imaging characteristics and the success of theranostics research, there are practical challenges that need to be overcome when considering use of β-emitters for surgical radioguidance. In this position paper, the EANM identifies the possibilities and challenges that relate to the successful implementation of β-emitters in surgical guidance, covering aspects related to instrumentation, radiation protection, and modes of implementation.

Simultaneous Depth Estimation and Surgical Tool Segmentation in Laparoscopic Images

Surgical instrument segmentation and depth estimation are crucial steps to improve autonomy in robotic surgery. Most recent works treat these problems separately, making the deployment challenging. In this paper, we propose a unified framework for depth estimation and surgical tool segmentation in laparoscopic images. The network has an encoder-decoder architecture and comprises two branches for simultaneously performing depth estimation and segmentation. To train the network end to end, we propose a new multi-task loss function that effectively learns to estimate depth in an unsupervised manner, while requiring only semi-ground truth for surgical tool segmentation. We conducted extensive experiments on different datasets to validate these findings. The results showed that the end-to-end network successfully improved the state-of-the-art for both tasks while reducing the complexity during their deployment.

Real-time tracking of a diffuse reflectance spectroscopy probe used to aid histological validation of margin assessment in upper gastrointestinal cancer resection surgery

SIGNIFICANCE

Diffuse reflectance spectroscopy (DRS) allows discrimination of tissue type. Its application is limited by the inability to mark the scanned tissue and the lack of real-time measurements.

AIM

This study aimed to develop a real-time tracking system to enable localization of a DRS probe to aid the classification of tumor and non-tumor tissue.

APPROACH

A green-colored marker attached to the DRS probe was detected using hue-saturation-value (HSV) segmentation. A live, augmented view of tracked optical biopsy sites was recorded in real time. Supervised classifiers were evaluated in terms of sensitivity, specificity, and overall accuracy. A developed software was used for data collection, processing, and statistical analysis.

RESULTS

The measured root mean square error (RMSE) of DRS probe tip tracking was 1.18  ±  0.58  mm and 1.05  ±  0.28  mm for the x and y dimensions, respectively. The diagnostic accuracy of the system to classify tumor and non-tumor tissue in real time was 94% for stomach and 96% for the esophagus.

CONCLUSIONS

We have successfully developed a real-time tracking and classification system for a DRS probe. When used on stomach and esophageal tissue for tumor detection, the accuracy derived demonstrates the strength and clinical value of the technique to aid margin assessment in cancer resection surgery.

How molecular imaging will enable robotic precision surgery : The role of artificial intelligence, augmented reality, and navigation

Molecular imaging is one of the pillars of precision surgery. Its applications range from early diagnostics to therapy planning, execution, and the accurate assessment of outcomes. In particular, molecular imaging solutions are in high demand in minimally invasive surgical strategies, such as the substantially increasing field of robotic surgery. This review aims at connecting the molecular imaging and nuclear medicine community to the rapidly expanding armory of surgical medical devices. Such devices entail technologies ranging from artificial intelligence and computer-aided visualization technologies (software) to innovative molecular imaging modalities and surgical navigation (hardware). We discuss technologies based on their role at different steps of the surgical workflow, i.e., from surgical decision and planning, over to target localization and excision guidance, all the way to (back table) surgical verification. This provides a glimpse of how innovations from the technology fields can realize an exciting future for the molecular imaging and surgery communities.

Optical Navigation of the Drop-In γ-Probe as a Means to Strengthen the Connection Between Robot-Assisted and Radioguided Surgery

With translation of the Drop-In γ-probe, radioguidance has advanced into laparoscopic robot-assisted surgery. Global-positioning-system-like navigation can further enhance the symbiosis between nuclear medicine and surgery. Therefore, we developed a fluorescence-video-based tracking method that integrates the Drop-In with navigated robotic surgery. Methods: Fluorescent markers, integrated into the Drop-In, were automatically detected using a daVinci Firefly laparoscope. Subsequently, a declipseSPECT-navigation platform calculated the Drop-In location within the surgical field. Using a phantom (n = 3), we pursued robotic navigation on SPECT/CT, whereas intraoperative feasibility was validated during porcine surgery (n = 4). Results: Video-based tracking allowed for navigation of the Drop-In toward all lesions detected on SPECT/CT (external iliac and common iliac artery regions). Augmented-reality visualization in the surgical console indicated the distance to these lesions in real time, confirmed by the Drop-In readout. Porcine surgery underlined the feasibility of the concept. Conclusion: Optical navigation of the Drop-In probe provides a next step toward connecting nuclear medicine with robotic surgery.