1
|
Liu M, Han Y, Wang J, Wang C, Wang Y, Meijering E. LSKANet: Long Strip Kernel Attention Network for Robotic Surgical Scene Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1308-1322. [PMID: 38015689 DOI: 10.1109/tmi.2023.3335406] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/30/2023]
Abstract
Surgical scene segmentation is a critical task in Robotic-assisted surgery. However, the complexity of the surgical scene, which mainly includes local feature similarity (e.g., between different anatomical tissues), intraoperative complex artifacts, and indistinguishable boundaries, poses significant challenges to accurate segmentation. To tackle these problems, we propose the Long Strip Kernel Attention network (LSKANet), including two well-designed modules named Dual-block Large Kernel Attention module (DLKA) and Multiscale Affinity Feature Fusion module (MAFF), which can implement precise segmentation of surgical images. Specifically, by introducing strip convolutions with different topologies (cascaded and parallel) in two blocks and a large kernel design, DLKA can make full use of region- and strip-like surgical features and extract both visual and structural information to reduce the false segmentation caused by local feature similarity. In MAFF, affinity matrices calculated from multiscale feature maps are applied as feature fusion weights, which helps to address the interference of artifacts by suppressing the activations of irrelevant regions. Besides, the hybrid loss with Boundary Guided Head (BGH) is proposed to help the network segment indistinguishable boundaries effectively. We evaluate the proposed LSKANet on three datasets with different surgical scenes. The experimental results show that our method achieves new state-of-the-art results on all three datasets with improvements of 2.6%, 1.4%, and 3.4% mIoU, respectively. Furthermore, our method is compatible with different backbones and can significantly increase their segmentation accuracy. Code is available at https://github.com/YubinHan73/LSKANet.
Collapse
|
2
|
Yan X, Jia L, Cao H, Yu Y, Wang T, Zhang F, Guan Q. Multitargets Joint Training Lightweight Model for Object Detection of Substation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:2413-2424. [PMID: 35877791 DOI: 10.1109/tnnls.2022.3190139] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The object detection of the substation is the key to ensuring the safety and reliable operation of the substation. The traditional image detection algorithms use the corresponding texture features of single-class objects and would not handle other different class objects easily. The object detection algorithm based on deep networks has generalization, and its sizeable complex backbone limits the application in the substation monitoring terminals with weak computing power. This article proposes a multitargets joint training lightweight model. The proposed model uses the feature maps of the complex model and the labels of objects in images as training multitargets. The feature maps have deeper feature information, and the feature maps of complex networks have higher information entropy than lightweight networks have. This article proposes the heat pixels method to improve the adequate object information because of the imbalance of the proportion between the foreground and the background. The heat pixels method is designed as a kind of reverse network calculation and reflects the object's position to the pixels of the feature maps. The temperature of the pixels indicates the probability of the existence of the objects in the locations. Three different lightweight networks use the complex model feature maps and the traditional tags as the training multitargets. The public dataset VOC and the substation equipment dataset are adopted in the experiments. The experimental results demonstrate that the proposed model can effectively improve object detection accuracy and reduce the time-consuming and calculation amount.
Collapse
|
3
|
Liu Z, Zheng L, Yang S, Zhong Z, Zhang G. MFF-Net: Multiscale feature fusion semantic segmentation network for intracranial surgical instruments. Int J Med Robot 2023:e2595. [PMID: 37932905 DOI: 10.1002/rcs.2595] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 09/25/2023] [Accepted: 10/26/2023] [Indexed: 11/08/2023]
Abstract
BACKGROUND In robot-assisted surgery, automatic segmentation of surgical instrument images is crucial for surgical safety. The proposed method addresses challenges in the craniotomy environment, such as occlusion and illumination, through an efficient surgical instrument segmentation network. METHODS The network uses YOLOv8 as the target detection framework and integrates a semantic segmentation head to achieve detection and segmentation capabilities. A concatenation of multi-channel feature maps is designed to enhance model generalisation by fusing deep and shallow features. The innovative GBC2f module ensures the lightweight of the network and the ability to capture global information. RESULTS Experimental validation of the intracranial glioma surgical instrument dataset shows excellent performance: 94.9% MPA score, 89.9% MIoU value, and 126.6 FPS. CONCLUSIONS According to the experimental results, the segmentation model proposed in this study has significant advantages over other state-of-the-art models. This provides a valuable reference for the further development of intelligent surgical robots.
Collapse
Affiliation(s)
- Zhenzhong Liu
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education (Tianjin University of Technology), Tianjin, China
| | - Laiwang Zheng
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education (Tianjin University of Technology), Tianjin, China
| | - Shubin Yang
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education (Tianjin University of Technology), Tianjin, China
| | - Zichen Zhong
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education (Tianjin University of Technology), Tianjin, China
| | - Guobin Zhang
- Tianjin Key Laboratory for Advanced Mechatronic System Design and Intelligent Control, School of Mechanical Engineering, Tianjin University of Technology, Tianjin, China
- National Demonstration Center for Experimental Mechanical and Electrical Engineering Education (Tianjin University of Technology), Tianjin, China
| |
Collapse
|
4
|
Shen W, Wang Y, Liu M, Wang J, Ding R, Zhang Z, Meijering E. Branch Aggregation Attention Network for Robotic Surgical Instrument Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3408-3419. [PMID: 37342952 DOI: 10.1109/tmi.2023.3288127] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/23/2023]
Abstract
Surgical instrument segmentation is of great significance to robot-assisted surgery, but the noise caused by reflection, water mist, and motion blur during the surgery as well as the different forms of surgical instruments would greatly increase the difficulty of precise segmentation. A novel method called Branch Aggregation Attention network (BAANet) is proposed to address these challenges, which adopts a lightweight encoder and two designed modules, named Branch Balance Aggregation module (BBA) and Block Attention Fusion module (BAF), for efficient feature localization and denoising. By introducing the unique BBA module, features from multiple branches are balanced and optimized through a combination of addition and multiplication to complement strengths and effectively suppress noise. Furthermore, to fully integrate the contextual information and capture the region of interest, the BAF module is proposed in the decoder, which receives adjacent feature maps from the BBA module and localizes the surgical instruments from both global and local perspectives by utilizing a dual branch attention mechanism. According to the experimental results, the proposed method has the advantage of being lightweight while outperforming the second-best method by 4.03%, 1.53%, and 1.34% in mIoU scores on three challenging surgical instrument datasets, respectively, compared to the existing state-of-the-art methods. Code is available at https://github.com/SWT-1014/BAANet.
Collapse
|
5
|
Ran QY, Miao J, Zhou SP, Hua SH, He SY, Zhou P, Wang HX, Zheng YP, Zhou GQ. Automatic 3-D spine curve measurement in freehand ultrasound via structure-aware reinforcement learning spinous process localization. ULTRASONICS 2023; 132:107012. [PMID: 37071944 DOI: 10.1016/j.ultras.2023.107012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 03/18/2023] [Accepted: 04/10/2023] [Indexed: 05/03/2023]
Abstract
Freehand 3-D ultrasound systems have been advanced in scoliosis assessment to avoid radiation hazards, especially for teenagers. This novel 3-D imaging method also makes it possible to evaluate the spine curvature automatically from the corresponding 3-D projection images. However, most approaches neglect the three-dimensional spine deformity by only using the rendering images, thus limiting their usage in clinical applications. In this study, we proposed a structure-aware localization model to directly identify the spinous processes for automatic 3-D spine curve measurement using the images acquired with freehand 3-D ultrasound imaging. The pivot is to leverage a novel reinforcement learning (RL) framework to localize the landmarks, which adopts a multi-scale agent to boost structure representation with positional information. We also introduced a structure similarity prediction mechanism to perceive the targets with apparent spinous process structures. Finally, a two-fold filtering strategy was proposed to screen the detected spinous processes landmarks iteratively, followed by a three-dimensional spine curve fitting for the spine curvature assessments. We evaluated the proposed model on 3-D ultrasound images among subjects with different scoliotic angles. The results showed that the mean localization accuracy of the proposed landmark localization algorithm was 5.95 pixels. Also, the curvature angles on the coronal plane obtained by the new method had a high linear correlation with those by manual measurement (R = 0.86, p < 0.001). These results demonstrated the potential of our proposed method for facilitating the 3-D assessment of scoliosis, especially for 3-D spine deformity assessment.
Collapse
Affiliation(s)
- Qi-Yong Ran
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; State Key Laboratory of Bioelectronics, School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing, China
| | - Juzheng Miao
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | - Si-Ping Zhou
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; State Key Laboratory of Bioelectronics, School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing, China
| | - Shi-Hao Hua
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; State Key Laboratory of Bioelectronics, School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing, China
| | - Si-Yuan He
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; State Key Laboratory of Bioelectronics, School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing, China
| | - Ping Zhou
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China
| | - Hong-Xing Wang
- The Department of Rehabilitation Medicine, Zhongda Hospital, Southeast University, Nanjing, China
| | - Yong-Ping Zheng
- The Department of Biomedical Engineering, The Hong Kong Polytechnic University, Hong Kong, China
| | - Guang-Quan Zhou
- The School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; State Key Laboratory of Bioelectronics, School of Biological Science and Medical Engineering, Southeast University, Nanjing, China; Jiangsu Key Laboratory of Biomaterials and Devices, Southeast University, Nanjing, China.
| |
Collapse
|
6
|
Bykanov A, Danilov G, Kostumov V, Pilipenko O, Nutfullin B, Rastvorova O, Pitskhelauri D. Artificial Intelligence Technologies in the Microsurgical Operating Room (Review). Sovrem Tekhnologii Med 2023; 15:86-94. [PMID: 37389018 PMCID: PMC10306972 DOI: 10.17691/stm2023.15.2.08] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Indexed: 07/01/2023] Open
Abstract
Surgery performed by a novice neurosurgeon under constant supervision of a senior surgeon with the experience of thousands of operations, able to handle any intraoperative complications and predict them in advance, and never getting tired, is currently an elusive dream, but can become a reality with the development of artificial intelligence methods. This paper has presented a review of the literature on the use of artificial intelligence technologies in the microsurgical operating room. Searching for sources was carried out in the PubMed text database of medical and biological publications. The key words used were "surgical procedures", "dexterity", "microsurgery" AND "artificial intelligence" OR "machine learning" OR "neural networks". Articles in English and Russian were considered with no limitation to publication date. The main directions of research on the use of artificial intelligence technologies in the microsurgical operating room have been highlighted. Despite the fact that in recent years machine learning has been increasingly introduced into the medical field, a small number of studies related to the problem of interest have been published, and their results have not proved to be of practical use yet. However, the social significance of this direction is an important argument for its development.
Collapse
Affiliation(s)
- A.E. Bykanov
- Neurosurgeon, 7 Department of Neurosurgery, Researcher; National Medical Research Center for Neurosurgery named after Academician N.N. Burdenko, Ministry of Healthcare of the Russian Federation, 16, 4 Tverskaya-Yamskaya St., Moscow, 125047, Russia
| | - G.V. Danilov
- Academic Secretary; National Medical Research Center for Neurosurgery named after Academician N.N. Burdenko, Ministry of Healthcare of the Russian Federation, 16, 4 Tverskaya-Yamskaya St., Moscow, 125047, Russia
| | - V.V. Kostumov
- PhD Student, Programmer, the CMC Faculty; Lomonosov Moscow State University, 1 Leninskiye Gory, Moscow, 119991, Russia
| | - O.G. Pilipenko
- PhD Student, Programmer, the CMC Faculty; Lomonosov Moscow State University, 1 Leninskiye Gory, Moscow, 119991, Russia
| | - B.M. Nutfullin
- PhD Student, Programmer, the CMC Faculty; Lomonosov Moscow State University, 1 Leninskiye Gory, Moscow, 119991, Russia
| | - O.A. Rastvorova
- Resident, 7 Department of Neurosurgery; National Medical Research Center for Neurosurgery named after Academician N.N. Burdenko, Ministry of Healthcare of the Russian Federation, 16, 4 Tverskaya-Yamskaya St., Moscow, 125047, Russia
| | - D.I. Pitskhelauri
- Professor, Head of the 7 Department of Neurosurgery; National Medical Research Center for Neurosurgery named after Academician N.N. Burdenko, Ministry of Healthcare of the Russian Federation, 16, 4 Tverskaya-Yamskaya St., Moscow, 125047, Russia
| |
Collapse
|
7
|
Chadebecq F, Lovat LB, Stoyanov D. Artificial intelligence and automation in endoscopy and surgery. Nat Rev Gastroenterol Hepatol 2023; 20:171-182. [PMID: 36352158 DOI: 10.1038/s41575-022-00701-y] [Citation(s) in RCA: 13] [Impact Index Per Article: 13.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 10/03/2022] [Indexed: 11/10/2022]
Abstract
Modern endoscopy relies on digital technology, from high-resolution imaging sensors and displays to electronics connecting configurable illumination and actuation systems for robotic articulation. In addition to enabling more effective diagnostic and therapeutic interventions, the digitization of the procedural toolset enables video data capture of the internal human anatomy at unprecedented levels. Interventional video data encapsulate functional and structural information about a patient's anatomy as well as events, activity and action logs about the surgical process. This detailed but difficult-to-interpret record from endoscopic procedures can be linked to preoperative and postoperative records or patient imaging information. Rapid advances in artificial intelligence, especially in supervised deep learning, can utilize data from endoscopic procedures to develop systems for assisting procedures leading to computer-assisted interventions that can enable better navigation during procedures, automation of image interpretation and robotically assisted tool manipulation. In this Perspective, we summarize state-of-the-art artificial intelligence for computer-assisted interventions in gastroenterology and surgery.
Collapse
Affiliation(s)
- François Chadebecq
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Laurence B Lovat
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK
| | - Danail Stoyanov
- Wellcome/EPSRC Centre for Interventional and Surgical Sciences, University College London, London, UK.
| |
Collapse
|
8
|
Kitaguchi D, Fujino T, Takeshita N, Hasegawa H, Mori K, Ito M. Limited generalizability of single deep neural network for surgical instrument segmentation in different surgical environments. Sci Rep 2022; 12:12575. [PMID: 35869249 PMCID: PMC9307578 DOI: 10.1038/s41598-022-16923-8] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2021] [Accepted: 07/18/2022] [Indexed: 12/05/2022] Open
Abstract
Clarifying the generalizability of deep-learning-based surgical-instrument segmentation networks in diverse surgical environments is important in recognizing the challenges of overfitting in surgical-device development. This study comprehensively evaluated deep neural network generalizability for surgical instrument segmentation using 5238 images randomly extracted from 128 intraoperative videos. The video dataset contained 112 laparoscopic colorectal resection, 5 laparoscopic distal gastrectomy, 5 laparoscopic cholecystectomy, and 6 laparoscopic partial hepatectomy cases. Deep-learning-based surgical-instrument segmentation was performed for test sets with (1) the same conditions as the training set; (2) the same recognition target surgical instrument and surgery type but different laparoscopic recording systems; (3) the same laparoscopic recording system and surgery type but slightly different recognition target laparoscopic surgical forceps; (4) the same laparoscopic recording system and recognition target surgical instrument but different surgery types. The mean average precision and mean intersection over union for test sets 1, 2, 3, and 4 were 0.941 and 0.887, 0.866 and 0.671, 0.772 and 0.676, and 0.588 and 0.395, respectively. Therefore, the recognition accuracy decreased even under slightly different conditions. The results of this study reveal the limited generalizability of deep neural networks in the field of surgical artificial intelligence and caution against deep-learning-based biased datasets and models. Trial Registration Number: 2020-315, date of registration: October 5, 2020.
Collapse
|
9
|
Markarian N, Kugener G, Pangal DJ, Unadkat V, Sinha A, Zhu Y, Roshannai A, Chan J, Hung AJ, Wrobel BB, Anandkumar A, Zada G, Donoho DA. Validation of Machine Learning-Based Automated Surgical Instrument Annotation Using Publicly Available Intraoperative Video. Oper Neurosurg (Hagerstown) 2022; 23:235-240. [PMID: 35972087 DOI: 10.1227/ons.0000000000000274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 03/05/2022] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND Intraoperative tool movement data have been demonstrated to be clinically useful in quantifying surgical performance. However, collecting this information from intraoperative video requires laborious hand annotation. The ability to automatically annotate tools in surgical video would advance surgical data science by eliminating a time-intensive step in research. OBJECTIVE To identify whether machine learning (ML) can automatically identify surgical instruments contained within neurosurgical video. METHODS A ML model which automatically identifies surgical instruments in frame was developed and trained on multiple publicly available surgical video data sets with instrument location annotations. A total of 39 693 frames from 4 data sets were used (endoscopic endonasal surgery [EEA] [30 015 frames], cataract surgery [4670], laparoscopic cholecystectomy [2532], and microscope-assisted brain/spine tumor removal [2476]). A second model trained only on EEA video was also developed. Intraoperative EEA videos from YouTube were used for test data (3 videos, 1239 frames). RESULTS The YouTube data set contained 2169 total instruments. Mean average precision (mAP) for instrument detection on the YouTube data set was 0.74. The mAP for each individual video was 0.65, 0.74, and 0.89. The second model trained only on EEA video also had an overall mAP of 0.74 (0.62, 0.84, and 0.88 for individual videos). Development costs were $130 for manual video annotation and under $100 for computation. CONCLUSION Surgical instruments contained within endoscopic endonasal intraoperative video can be detected using a fully automated ML model. The addition of disparate surgical data sets did not improve model performance, although these data sets may improve generalizability of the model in other use cases.
Collapse
Affiliation(s)
- Nicholas Markarian
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Guillaume Kugener
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Dhiraj J Pangal
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Vyom Unadkat
- Department of Computer Science, Viterbi School of Engineering, University of Southern California, Los Angeles, California, USA
| | | | - Yichao Zhu
- Department of Computer Science, Viterbi School of Engineering, University of Southern California, Los Angeles, California, USA
| | - Arman Roshannai
- Department of Computer Science, Viterbi School of Engineering, University of Southern California, Los Angeles, California, USA
| | - Justin Chan
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Andrew J Hung
- Center for Robotic Simulation and Education, USC Institute of Urology, Keck School of Medicine of the University of Southern California, Los Angeles, California, USA
| | - Bozena B Wrobel
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA.,USC Caruso Department of Otolaryngology-Head and Neck Surgery, University of Southern California, Los Angeles, California, USA
| | - Animashree Anandkumar
- Department of Computing + Mathematical Sciences, California Institute of Technology, Pasadena, California, USA
| | - Gabriel Zada
- Department of Neurosurgery, Keck School of Medicine, University of Southern California, Los Angeles, California, USA
| | - Daniel A Donoho
- Division of Neurosurgery, Department of Surgery, Texas Children's Hospital, Baylor College of Medicine, Houston, Texas, USA
| |
Collapse
|
10
|
Surgical Tool Datasets for Machine Learning Research: A Survey. Int J Comput Vis 2022. [DOI: 10.1007/s11263-022-01640-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
Abstract
AbstractThis paper is a comprehensive survey of datasets for surgical tool detection and related surgical data science and machine learning techniques and algorithms. The survey offers a high level perspective of current research in this area, analyses the taxonomy of approaches adopted by researchers using surgical tool datasets, and addresses key areas of research, such as the datasets used, evaluation metrics applied and deep learning techniques utilised. Our presentation and taxonomy provides a framework that facilitates greater understanding of current work, and highlights the challenges and opportunities for further innovative and useful research.
Collapse
|
11
|
Expert surgeons and deep learning models can predict the outcome of surgical hemorrhage from 1 min of video. Sci Rep 2022; 12:8137. [PMID: 35581213 PMCID: PMC9114003 DOI: 10.1038/s41598-022-11549-2] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Accepted: 04/18/2022] [Indexed: 01/28/2023] Open
Abstract
Major vascular injury resulting in uncontrolled bleeding is a catastrophic and often fatal complication of minimally invasive surgery. At the outset of these events, surgeons do not know how much blood will be lost or whether they will successfully control the hemorrhage (achieve hemostasis). We evaluate the ability of a deep learning neural network (DNN) to predict hemostasis control ability using the first minute of surgical video and compare model performance with human experts viewing the same video. The publicly available SOCAL dataset contains 147 videos of attending and resident surgeons managing hemorrhage in a validated, high-fidelity cadaveric simulator. Videos are labeled with outcome and blood loss (mL). The first minute of 20 videos was shown to four, blinded, fellowship trained skull-base neurosurgery instructors, and to SOCALNet (a DNN trained on SOCAL videos). SOCALNet architecture included a convolutional network (ResNet) identifying spatial features and a recurrent network identifying temporal features (LSTM). Experts independently assessed surgeon skill, predicted outcome and blood loss (mL). Outcome and blood loss predictions were compared with SOCALNet. Expert inter-rater reliability was 0.95. Experts correctly predicted 14/20 trials (Sensitivity: 82%, Specificity: 55%, Positive Predictive Value (PPV): 69%, Negative Predictive Value (NPV): 71%). SOCALNet correctly predicted 17/20 trials (Sensitivity 100%, Specificity 66%, PPV 79%, NPV 100%) and correctly identified all successful attempts. Expert predictions of the highest and lowest skill surgeons and expert predictions reported with maximum confidence were more accurate. Experts systematically underestimated blood loss (mean error - 131 mL, RMSE 350 mL, R2 0.70) and fewer than half of expert predictions identified blood loss > 500 mL (47.5%, 19/40). SOCALNet had superior performance (mean error - 57 mL, RMSE 295 mL, R2 0.74) and detected most episodes of blood loss > 500 mL (80%, 8/10). In validation experiments, SOCALNet evaluation of a critical on-screen surgical maneuver and high/low-skill composite videos were concordant with expert evaluation. Using only the first minute of video, experts and SOCALNet can predict outcome and blood loss during surgical hemorrhage. Experts systematically underestimated blood loss, and SOCALNet had no false negatives. DNNs can provide accurate, meaningful assessments of surgical video. We call for the creation of datasets of surgical adverse events for quality improvement research.
Collapse
|
12
|
Zenteno O, Trinh DH, Treuillet S, Lucas Y, Bazin T, Lamarque D, Daul C. Optical biopsy mapping on endoscopic image mosaics with a marker-free probe. Comput Biol Med 2022; 143:105234. [PMID: 35093845 DOI: 10.1016/j.compbiomed.2022.105234] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2021] [Revised: 12/25/2021] [Accepted: 01/10/2022] [Indexed: 12/24/2022]
Abstract
Gastric cancer is the second leading cause of cancer-related deaths worldwide. Early diagnosis significantly increases the chances of survival; therefore, improved assisted exploration and screening techniques are necessary. Previously, we made use of an augmented multi-spectral endoscope by inserting an optical probe into the instrumentation channel. However, the limited field of view and the lack of markings left by optical biopsies on the tissue complicate the navigation and revisit of the suspect areas probed in-vivo. In this contribution two innovative tools are introduced to significantly increase the traceability and monitoring of patients in clinical practice: (i) video mosaicing to build a more comprehensive and panoramic view of large gastric areas; (ii) optical biopsy targeting and registration with the endoscopic images. The proposed optical flow-based mosaicing technique selects images that minimize texture discontinuities and is robust despite the lack of texture and illumination variations. The optical biopsy targeting is based on automatic tracking of a free-marker probe in the endoscopic view using deep learning to dynamically estimate its pose during exploration. The accuracy of pose estimation is sufficient to ensure a precise overlapping of the standard white-light color image and the hyperspectral probe image, assuming that the small target area of the organ is almost flat. This allows the mapping of all spatio-temporally tracked biopsy sites onto the panoramic mosaic. Experimental validations are carried out from videos acquired on patients in hospital. The proposed technique is purely software-based and therefore easily integrable into clinical practice. It is also generic and compatible to any imaging modality that connects to a fiberscope.
Collapse
Affiliation(s)
- Omar Zenteno
- Laboratoire PRISME, Université d'Orléans, Orléans, France
| | - Dinh-Hoan Trinh
- CRAN, UMR 7039 CNRS and Université de Lorraine, Vandœuvre-lès-Nancy, France
| | | | - Yves Lucas
- Laboratoire PRISME, Université d'Orléans, Orléans, France
| | - Thomas Bazin
- Service d'Hépato-gastroentérologie et oncologie digestive, Hôpital Ambroise Paré, Boulogne-Billancourt, France
| | - Dominique Lamarque
- Service d'Hépato-gastroentérologie et oncologie digestive, Hôpital Ambroise Paré, Boulogne-Billancourt, France
| | - Christian Daul
- CRAN, UMR 7039 CNRS and Université de Lorraine, Vandœuvre-lès-Nancy, France.
| |
Collapse
|
13
|
Sun Y, Pan B, Fu Y. Lightweight Deep Neural Network for Articulated Joint Detection of Surgical Instrument in Minimally Invasive Surgical Robot. J Digit Imaging 2022; 35:923-937. [PMID: 35266089 PMCID: PMC9485358 DOI: 10.1007/s10278-022-00616-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/24/2021] [Revised: 01/15/2022] [Accepted: 02/27/2022] [Indexed: 11/29/2022] Open
Abstract
Vision-based detection and tracking of surgical instrument are attractive because it relies purely on surgical instrument already in the operating scenario. The vision knowledge of the surgical instruments is a crucial piece of topic for surgical task understanding, autonomous robot control and human-robot collaborative surgeries to enhance surgical outcomes. In this work, a novel method has been demonstrated by developing a multitask lightweight deep neural network framework to explore surgical instrument articulated joint detection. The model has an end-to-end architecture with two branches, which share the same high-level visual features provided by a lightweight backbone while holding respective layers targeting for specific tasks. We have designed a novel subnetwork with joint detection branch and an instrument classification branch to sufficiently take advantage of the relatedness of surgical instrument presence detection and surgical instrument articulated joint detection tasks. The lightweight joint detection branch has been employed to efficiently locate the articulated joint position with simultaneously holding low computational cost. Moreover, the surgical instrument classification branch is introduced to boost the performance of joint detection. The two branches are merged to output the articulated joint location with respective instrument type. Extensive validation has been conducted to evaluate the proposed method. The results demonstrate promising performance of our proposed method. The work represents the feasibility to perform real-time surgical instrument articulated joint detection by taking advantage of the components of surgical robot system, contributing to the reference for further surgical intelligence.
Collapse
Affiliation(s)
- Yanwen Sun
- State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin, China
| | - Bo Pan
- State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin, China.
| | - Yili Fu
- State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin, China
| |
Collapse
|
14
|
Gkouzionis I, Nazarian S, Kawka M, Darzi A, Patel N, Peters CJ, Elson DS. Real-time tracking of a diffuse reflectance spectroscopy probe used to aid histological validation of margin assessment in upper gastrointestinal cancer resection surgery. JOURNAL OF BIOMEDICAL OPTICS 2022; 27:JBO-210293R. [PMID: 35106980 PMCID: PMC8804336 DOI: 10.1117/1.jbo.27.2.025001] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/16/2021] [Accepted: 01/10/2022] [Indexed: 05/27/2023]
Abstract
SIGNIFICANCE Diffuse reflectance spectroscopy (DRS) allows discrimination of tissue type. Its application is limited by the inability to mark the scanned tissue and the lack of real-time measurements. AIM This study aimed to develop a real-time tracking system to enable localization of a DRS probe to aid the classification of tumor and non-tumor tissue. APPROACH A green-colored marker attached to the DRS probe was detected using hue-saturation-value (HSV) segmentation. A live, augmented view of tracked optical biopsy sites was recorded in real time. Supervised classifiers were evaluated in terms of sensitivity, specificity, and overall accuracy. A developed software was used for data collection, processing, and statistical analysis. RESULTS The measured root mean square error (RMSE) of DRS probe tip tracking was 1.18 ± 0.58 mm and 1.05 ± 0.28 mm for the x and y dimensions, respectively. The diagnostic accuracy of the system to classify tumor and non-tumor tissue in real time was 94% for stomach and 96% for the esophagus. CONCLUSIONS We have successfully developed a real-time tracking and classification system for a DRS probe. When used on stomach and esophageal tissue for tumor detection, the accuracy derived demonstrates the strength and clinical value of the technique to aid margin assessment in cancer resection surgery.
Collapse
Affiliation(s)
- Ioannis Gkouzionis
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
- Imperial College London, Hamlyn Centre, London, United Kingdom
| | - Scarlet Nazarian
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
| | - Michal Kawka
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
| | - Ara Darzi
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
- Imperial College London, Hamlyn Centre, London, United Kingdom
| | - Nisha Patel
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
| | | | - Daniel S. Elson
- Imperial College London, Department of Surgery and Cancer, London, United Kingdom
- Imperial College London, Hamlyn Centre, London, United Kingdom
| |
Collapse
|
15
|
Video-based fully automatic assessment of open surgery suturing skills. Int J Comput Assist Radiol Surg 2022; 17:437-448. [PMID: 35103921 PMCID: PMC8805431 DOI: 10.1007/s11548-022-02559-6] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2021] [Accepted: 01/03/2022] [Indexed: 01/09/2023]
Abstract
Purpose The goal of this study was to develop a new reliable open surgery suturing simulation system for training medical students in situations where resources are limited or in the domestic setup. Namely, we developed an algorithm for tools and hands localization as well as identifying the interactions between them based on simple webcam video data, calculating motion metrics for assessment of surgical skill. Methods Twenty-five participants performed multiple suturing tasks using our simulator. The YOLO network was modified to a multi-task network for the purpose of tool localization and tool–hand interaction detection. This was accomplished by splitting the YOLO detection heads so that they supported both tasks with minimal addition to computer run-time. Furthermore, based on the outcome of the system, motion metrics were calculated. These metrics included traditional metrics such as time and path length as well as new metrics assessing the technique participants use for holding the tools. Results The dual-task network performance was similar to that of two networks, while computational load was only slightly bigger than one network. In addition, the motion metrics showed significant differences between experts and novices. Conclusion While video capture is an essential part of minimal invasive surgery, it is not an integral component of open surgery. Thus, new algorithms, focusing on the unique challenges open surgery videos present, are required. In this study, a dual-task network was developed to solve both a localization task and a hand–tool interaction task. The dual network may be easily expanded to a multi-task network, which may be useful for images with multiple layers and for evaluating the interaction between these different layers. Supplementary Information The online version contains supplementary material available at 10.1007/s11548-022-02559-6.
Collapse
|
16
|
Marker-free Surgical Navigation of Rod Bending using a Stereo Neural Network and Augmented Reality in Spinal Fusion. Med Image Anal 2022; 77:102365. [DOI: 10.1016/j.media.2022.102365] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2021] [Revised: 11/16/2021] [Accepted: 01/10/2022] [Indexed: 11/20/2022]
|
17
|
Luca AR, Ursuleanu TF, Gheorghe L, Grigorovici R, Iancu S, Hlusneac M, Grigorovici A. Impact of quality, type and volume of data used by deep learning models in the analysis of medical images. INFORMATICS IN MEDICINE UNLOCKED 2022. [DOI: 10.1016/j.imu.2022.100911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022] Open
|
18
|
Li RQ, Xie XL, Zhou XH, Liu SQ, Ni ZL, Zhou YJ, Bian GB, Hou ZG. A Unified Framework for Multi-Guidewire Endpoint Localization in Fluoroscopy Images. IEEE Trans Biomed Eng 2021; 69:1406-1416. [PMID: 34613905 DOI: 10.1109/tbme.2021.3118001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECTIVE In this paper, Keypoint Localization Region-based CNN (KL R-CNN) is proposed, which can simultaneously accomplish the guidewire detection and endpoint localization in a unified model. METHODS KL R-CNN modifies Mask R-CNN by replacing the mask branch with a novel keypoint localization branch. Besides, some settings of Mask R-CNN are also modified to generate the keypoint localization results at a higher detail level. At the same time, based on the existing metrics of Average Precision (AP) and Percentage of Correct Keypoints (PCK), a new metric named AP PCK is proposed to evaluate the overall performance on the multi-guidewire endpoint localization task. Compared with existing metrics, AP PCK is easy to use and its results are more intuitive. RESULTS Compared with existing methods, KL R-CNN has better performance when the threshold is loose, reaching a mean AP PCK of 90.65% when the threshold is 9 pixels. CONCLUSION KL R-CNN achieves the state-of-the-art performance on the multi-guidewire endpoint localization task and has application potentials. SIGNIFICANCE KL R-CNN can achieve the localization of guidewire endpoints in fluoroscopy images, which is a prerequisite for computer-assisted percutaneous coronary intervention. KL R-CNN can also be extended to other multi-instrument localization tasks.
Collapse
|
19
|
Li RQ, Xie XL, Zhou XH, Liu SQ, Ni ZL, Zhou YJ, Bian GB, Hou ZG. Real-Time Multi-Guidewire Endpoint Localization in Fluoroscopy Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2002-2014. [PMID: 33788685 DOI: 10.1109/tmi.2021.3069998] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).
Collapse
|
20
|
Ursuleanu TF, Luca AR, Gheorghe L, Grigorovici R, Iancu S, Hlusneac M, Preda C, Grigorovici A. Deep Learning Application for Analyzing of Constituents and Their Correlations in the Interpretations of Medical Images. Diagnostics (Basel) 2021; 11:1373. [PMID: 34441307 PMCID: PMC8393354 DOI: 10.3390/diagnostics11081373] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/07/2021] [Revised: 07/25/2021] [Accepted: 07/27/2021] [Indexed: 12/13/2022] Open
Abstract
The need for time and attention, given by the doctor to the patient, due to the increased volume of medical data to be interpreted and filtered for diagnostic and therapeutic purposes has encouraged the development of the option to support, constructively and effectively, deep learning models. Deep learning (DL) has experienced an exponential development in recent years, with a major impact on interpretations of the medical image. This has influenced the development, diversification and increase of the quality of scientific data, the development of knowledge construction methods and the improvement of DL models used in medical applications. All research papers focus on description, highlighting, classification of one of the constituent elements of deep learning models (DL), used in the interpretation of medical images and do not provide a unified picture of the importance and impact of each constituent in the performance of DL models. The novelty in our paper consists primarily in the unitary approach, of the constituent elements of DL models, namely, data, tools used by DL architectures or specifically constructed DL architecture combinations and highlighting their "key" features, for completion of tasks in current applications in the interpretation of medical images. The use of "key" characteristics specific to each constituent of DL models and the correct determination of their correlations, may be the subject of future research, with the aim of increasing the performance of DL models in the interpretation of medical images.
Collapse
Affiliation(s)
- Tudor Florin Ursuleanu
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
- Department of Surgery VI, “Sf. Spiridon” Hospital, 700111 Iasi, Romania
- Department of Surgery I, Regional Institute of Oncology, 700483 Iasi, Romania
| | - Andreea Roxana Luca
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
- Department Obstetrics and Gynecology, Integrated Ambulatory of Hospital “Sf. Spiridon”, 700106 Iasi, Romania
| | - Liliana Gheorghe
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
- Department of Radiology, “Sf. Spiridon” Hospital, 700111 Iasi, Romania
| | - Roxana Grigorovici
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
| | - Stefan Iancu
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
| | - Maria Hlusneac
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
| | - Cristina Preda
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
- Department of Endocrinology, “Sf. Spiridon” Hospital, 700111 Iasi, Romania
| | - Alexandru Grigorovici
- Faculty of General Medicine, “Grigore T. Popa” University of Medicine and Pharmacy, 700115 Iasi, Romania; (T.F.U.); (R.G.); (S.I.); (M.H.); (C.P.); (A.G.)
- Department of Surgery VI, “Sf. Spiridon” Hospital, 700111 Iasi, Romania
| |
Collapse
|
21
|
Kurmann T, Márquez-Neila P, Allan M, Wolf S, Sznitman R. Mask then classify: multi-instance segmentation for surgical instruments. Int J Comput Assist Radiol Surg 2021; 16:1227-1236. [PMID: 34143374 PMCID: PMC8260538 DOI: 10.1007/s11548-021-02404-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/25/2021] [Accepted: 05/10/2021] [Indexed: 10/26/2022]
Abstract
PURPOSE The detection and segmentation of surgical instruments has been a vital step for many applications in minimally invasive surgical robotics. Previously, the problem was tackled from a semantic segmentation perspective, yet these methods fail to provide good segmentation maps of instrument types and do not contain any information on the instance affiliation of each pixel. We propose to overcome this limitation by using a novel instance segmentation method which first masks instruments and then classifies them into their respective type. METHODS We introduce a novel method for instance segmentation where a pixel-wise mask of each instance is found prior to classification. An encoder-decoder network is used to extract instrument instances, which are then separately classified using the features of the previous stages. Furthermore, we present a method to incorporate instrument priors from surgical robots. RESULTS Experiments are performed on the robotic instrument segmentation dataset of the 2017 endoscopic vision challenge. We perform a fourfold cross-validation and show an improvement of over 18% to the previous state-of-the-art. Furthermore, we perform an ablation study which highlights the importance of certain design choices and observe an increase of 10% over semantic segmentation methods. CONCLUSIONS We have presented a novel instance segmentation method for surgical instruments which outperforms previous semantic segmentation-based methods. Our method further provides a more informative output of instance level information, while retaining a precise segmentation mask. Finally, we have shown that robotic instrument priors can be used to further increase the performance.
Collapse
Affiliation(s)
| | | | - Max Allan
- Intuitive Surgical Inc., Sunnyvale, USA
| | - Sebastian Wolf
- Department of Ophthalmology, Bern University Hospital, Bern, Switzerland
| | | |
Collapse
|
22
|
Robu M, Kadkhodamohammadi A, Luengo I, Stoyanov D. Towards real-time multiple surgical tool tracking. COMPUTER METHODS IN BIOMECHANICS AND BIOMEDICAL ENGINEERING: IMAGING & VISUALIZATION 2021. [DOI: 10.1080/21681163.2020.1835553] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Affiliation(s)
- Maria Robu
- Digital Surgery, Medtronic Company, London, UK
| | | | | | - Danail Stoyanov
- Digital Surgery, Medtronic Company, London, UK
- University College London, London, UK
| |
Collapse
|
23
|
Sahu M, Mukhopadhyay A, Zachow S. Simulation-to-real domain adaptation with teacher-student learning for endoscopic instrument segmentation. Int J Comput Assist Radiol Surg 2021; 16:849-859. [PMID: 33982232 PMCID: PMC8134307 DOI: 10.1007/s11548-021-02383-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2021] [Accepted: 04/16/2021] [Indexed: 02/06/2023]
Abstract
PURPOSE Segmentation of surgical instruments in endoscopic video streams is essential for automated surgical scene understanding and process modeling. However, relying on fully supervised deep learning for this task is challenging because manual annotation occupies valuable time of the clinical experts. METHODS We introduce a teacher-student learning approach that learns jointly from annotated simulation data and unlabeled real data to tackle the challenges in simulation-to-real unsupervised domain adaptation for endoscopic image segmentation. RESULTS Empirical results on three datasets highlight the effectiveness of the proposed framework over current approaches for the endoscopic instrument segmentation task. Additionally, we provide analysis of major factors affecting the performance on all datasets to highlight the strengths and failure modes of our approach. CONCLUSIONS We show that our proposed approach can successfully exploit the unlabeled real endoscopic video frames and improve generalization performance over pure simulation-based training and the previous state-of-the-art. This takes us one step closer to effective segmentation of surgical instrument in the annotation scarce setting.
Collapse
Affiliation(s)
- Manish Sahu
- Zuse Institute Berlin (ZIB), Berlin, Germany
| | | | | |
Collapse
|
24
|
Cho SM, Kim YG, Jeong J, Kim I, Lee HJ, Kim N. Automatic tip detection of surgical instruments in biportal endoscopic spine surgery. Comput Biol Med 2021; 133:104384. [PMID: 33864974 DOI: 10.1016/j.compbiomed.2021.104384] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2020] [Revised: 03/29/2021] [Accepted: 04/05/2021] [Indexed: 11/27/2022]
Abstract
BACKGROUND Recent advances in robotics and deep learning can be used in endoscopic surgeries and can provide numerous advantages by freeing one of the surgeon's hands. This study aims to automatically detect the tip of the instrument, localize a point, and evaluate the detection accuracy in biportal endoscopic spine surgery (BESS). The tip detection could serve as a preliminary study for the development of vision intelligence in robotic endoscopy. METHODS The dataset contains 2310 frames from 9 BESS videos with x and y coordinates of the tip annotated by an expert. We trained two state-of-the-art detectors, RetinaNet and YOLOv2, with bounding boxes centered around the tip annotations with specific margin sizes to determine the optimal margin size for detecting the tip of the instrument and localizing the point. We calculated the recall, precision, and F1-score with a fixed box size for both ground truth tip coordinates and predicted midpoints to compare the performance of the models trained with different margin size bounding boxes. RESULTS For RetinaNet, a margin size of 150 pixels was optimal with a recall of 1.000, precision of 0.733, and F1-score of 0.846. For YOLOv2, a margin size of 150 pixels was optimal with a recall of 0.864, precision of 0.808, F1-score of 0.835. Also, the optimal margin size of 150 pixels of RetinaNet was used to cross-validate its overall robustness. The resulting mean recall, precision, and F1-score were 1.000 ± 0.000, 0.767 ± 0.033, and 0.868 ± 0.022, respectively. CONCLUSIONS In this study, we evaluated an automatic tip detection method for surgical instruments in endoscopic surgery, compared two state-of-the-art detection algorithms, RetinaNet and YOLOv2, and validated the robustness with cross-validation. This method can be applied in different types of endoscopy tip detection.
Collapse
Affiliation(s)
- Sue Min Cho
- Department of Convergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea; Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, United States
| | - Young-Gon Kim
- Department of Convergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea
| | - Jinhoon Jeong
- Department of Convergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea
| | - Inhwan Kim
- Department of Convergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea
| | - Ho-Jin Lee
- Department of Orthopaedic Surgery, Chungnam National University School of Medicine, Seoul, South Korea
| | - Namkug Kim
- Department of Convergence Medicine, Biomedical Engineering Research Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea.
| |
Collapse
|
25
|
Sestini L, Rosa B, De Momi E, Ferrigno G, Padoy N. A Kinematic Bottleneck Approach for Pose Regression of Flexible Surgical Instruments Directly From Images. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3062308] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
26
|
Sun Y, Pan B, Fu Y. Lightweight Deep Neural Network for Real-Time Instrument Semantic Segmentation in Robot Assisted Minimally Invasive Surgery. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3066956] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
27
|
Beyersdorffer P, Kunert W, Jansen K, Miller J, Wilhelm P, Burgert O, Kirschniak A, Rolinger J. Detection of adverse events leading to inadvertent injury during laparoscopic cholecystectomy using convolutional neural networks. ACTA ACUST UNITED AC 2021; 66:413-421. [PMID: 33655738 DOI: 10.1515/bmt-2020-0106] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2020] [Accepted: 02/16/2021] [Indexed: 01/17/2023]
Abstract
Uncontrolled movements of laparoscopic instruments can lead to inadvertent injury of adjacent structures. The risk becomes evident when the dissecting instrument is located outside the field of view of the laparoscopic camera. Technical solutions to ensure patient safety are appreciated. The present work evaluated the feasibility of an automated binary classification of laparoscopic image data using Convolutional Neural Networks (CNN) to determine whether the dissecting instrument is located within the laparoscopic image section. A unique record of images was generated from six laparoscopic cholecystectomies in a surgical training environment to configure and train the CNN. By using a temporary version of the neural network, the annotation of the training image files could be automated and accelerated. A combination of oversampling and selective data augmentation was used to enlarge the fully labeled image data set and prevent loss of accuracy due to imbalanced class volumes. Subsequently the same approach was applied to the comprehensive, fully annotated Cholec80 database. The described process led to the generation of extensive and balanced training image data sets. The performance of the CNN-based binary classifiers was evaluated on separate test records from both databases. On our recorded data, an accuracy of 0.88 with regard to the safety-relevant classification was achieved. The subsequent evaluation on the Cholec80 data set yielded an accuracy of 0.84. The presented results demonstrate the feasibility of a binary classification of laparoscopic image data for the detection of adverse events in a surgical training environment using a specifically configured CNN architecture.
Collapse
Affiliation(s)
| | - Wolfgang Kunert
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| | - Kai Jansen
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| | - Johanna Miller
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| | - Peter Wilhelm
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| | - Oliver Burgert
- Department of Medical Informatics, Reutlingen University, Reutlingen, Germany
| | - Andreas Kirschniak
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| | - Jens Rolinger
- Department of Surgery and Transplantation, Tübingen University Hospital, Tübingen, Germany
| |
Collapse
|
28
|
Marzullo A, Moccia S, Catellani M, Calimeri F, Momi ED. Towards realistic laparoscopic image generation using image-domain translation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 200:105834. [PMID: 33229016 DOI: 10.1016/j.cmpb.2020.105834] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/28/2020] [Accepted: 11/05/2020] [Indexed: 06/11/2023]
Abstract
Background and ObjectivesOver the last decade, Deep Learning (DL) has revolutionized data analysis in many areas, including medical imaging. However, there is a bottleneck in the advancement of DL in the surgery field, which can be seen in a shortage of large-scale data, which in turn may be attributed to the lack of a structured and standardized methodology for storing and analyzing surgical images in clinical centres. Furthermore, accurate annotations manually added are expensive and time consuming. A great help can come from the synthesis of artificial images; in this context, in the latest years, the use of Generative Adversarial Neural Networks (GANs) achieved promising results in obtaining photo-realistic images. MethodsIn this study, a method for Minimally Invasive Surgery (MIS) image synthesis is proposed. To this aim, the generative adversarial network pix2pix is trained to generate paired annotated MIS images by transforming rough segmentation of surgical instruments and tissues into realistic images. An additional regularization term was added to the original optimization problem, in order to enhance realism of surgical tools with respect to the background. Results Quantitative and qualitative (i.e., human-based) evaluations of generated images have been carried out in order to assess the effectiveness of the method. ConclusionsExperimental results show that the proposed method is actually able to translate MIS segmentations to realistic MIS images, which can in turn be used to augment existing data sets and help at overcoming the lack of useful images; this allows physicians and algorithms to take advantage from new annotated instances for their training.
Collapse
Affiliation(s)
- Aldo Marzullo
- Department of Mathematics and Computer Science, University of Calabria, Rende, Italy.
| | - Sara Moccia
- Department of Information Engineering, Unviersitá Politecnica delle Marche, Ancona, Italy; Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Michele Catellani
- Department of urology, European Institute of Oncology, IRCCS, Milan, Italy
| | - Francesco Calimeri
- Department of Mathematics and Computer Science, University of Calabria, Rende, Italy
| | - Elena De Momi
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| |
Collapse
|
29
|
Shimizu T, Hachiuma R, Kajita H, Takatsume Y, Saito H. Hand Motion-Aware Surgical Tool Localization and Classification from an Egocentric Camera. J Imaging 2021; 7:15. [PMID: 34460614 PMCID: PMC8321273 DOI: 10.3390/jimaging7020015] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2020] [Revised: 01/15/2021] [Accepted: 01/18/2021] [Indexed: 11/16/2022] Open
Abstract
Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods.
Collapse
Affiliation(s)
- Tomohiro Shimizu
- Faculty of Science and Technology, Keio University, Yokohama, Kanagawa 223-8852, Japan; (R.H.); (H.S.)
| | - Ryo Hachiuma
- Faculty of Science and Technology, Keio University, Yokohama, Kanagawa 223-8852, Japan; (R.H.); (H.S.)
| | - Hiroki Kajita
- Keio University School of Medicine, Shinjuku-ku 160-8582, Tokyo, Japan; (H.K.); (Y.T.)
| | - Yoshifumi Takatsume
- Keio University School of Medicine, Shinjuku-ku 160-8582, Tokyo, Japan; (H.K.); (Y.T.)
| | - Hideo Saito
- Faculty of Science and Technology, Keio University, Yokohama, Kanagawa 223-8852, Japan; (R.H.); (H.S.)
| |
Collapse
|
30
|
Nogales A, García-Tejedor ÁJ, Monge D, Vara JS, Antón C. A survey of deep learning models in medical therapeutic areas. Artif Intell Med 2021; 112:102020. [PMID: 33581832 DOI: 10.1016/j.artmed.2021.102020] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/18/2020] [Revised: 12/21/2020] [Accepted: 01/10/2021] [Indexed: 12/18/2022]
Abstract
Artificial intelligence is a broad field that comprises a wide range of techniques, where deep learning is presently the one with the most impact. Moreover, the medical field is an area where data both complex and massive and the importance of the decisions made by doctors make it one of the fields in which deep learning techniques can have the greatest impact. A systematic review following the Cochrane recommendations with a multidisciplinary team comprised of physicians, research methodologists and computer scientists has been conducted. This survey aims to identify the main therapeutic areas and the deep learning models used for diagnosis and treatment tasks. The most relevant databases included were MedLine, Embase, Cochrane Central, Astrophysics Data System, Europe PubMed Central, Web of Science and Science Direct. An inclusion and exclusion criteria were defined and applied in the first and second peer review screening. A set of quality criteria was developed to select the papers obtained after the second screening. Finally, 126 studies from the initial 3493 papers were selected and 64 were described. Results show that the number of publications on deep learning in medicine is increasing every year. Also, convolutional neural networks are the most widely used models and the most developed area is oncology where they are used mainly for image analysis.
Collapse
Affiliation(s)
- Alberto Nogales
- CEIEC, Research Institute, Universidad Francisco de Vitoria, Ctra. M-515 Pozuelo-Majadahonda km 1800, 28223, Pozuelo de Alarcón, Spain.
| | - Álvaro J García-Tejedor
- CEIEC, Research Institute, Universidad Francisco de Vitoria, Ctra. M-515 Pozuelo-Majadahonda km 1800, 28223, Pozuelo de Alarcón, Spain.
| | - Diana Monge
- Faculty of Medicine, Research Institute, Universidad Francisco de Vitoria, Ctra. M-515 Pozuelo-Majadahonda km 1800, 28223, Pozuelo de Alarcón, Spain.
| | - Juan Serrano Vara
- CEIEC, Research Institute, Universidad Francisco de Vitoria, Ctra. M-515 Pozuelo-Majadahonda km 1800, 28223, Pozuelo de Alarcón, Spain.
| | - Cristina Antón
- Faculty of Medicine, Research Institute, Universidad Francisco de Vitoria, Ctra. M-515 Pozuelo-Majadahonda km 1800, 28223, Pozuelo de Alarcón, Spain.
| |
Collapse
|
31
|
Fiorentino MC, Moccia S, Capparuccini M, Giamberini S, Frontoni E. A regression framework to head-circumference delineation from US fetal images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 198:105771. [PMID: 33049451 DOI: 10.1016/j.cmpb.2020.105771] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/10/2020] [Accepted: 09/20/2020] [Indexed: 06/11/2023]
Abstract
BACKGROUND AND OBJECTIVES Measuring head-circumference (HC) length from ultrasound (US) images is a crucial clinical task to assess fetus growth. To lower intra- and inter-operator variability in HC length measuring, several computer-assisted solutions have been proposed in the years. Recently, a large number of deep-learning approaches is addressing the problem of HC delineation through the segmentation of the whole fetal head via convolutional neural networks (CNNs). Since the task is a edge-delineation problem, we propose a different strategy based on regression CNNs. METHODS The proposed framework consists of a region-proposal CNN for head localization and centering, and a regression CNN for accurately delineate the HC. The first CNN is trained exploiting transfer learning, while we propose a training strategy for the regression CNN based on distance fields. RESULTS The framework was tested on the HC18 Challenge dataset, which consists of 999 training and 335 testing images. A mean absolute difference of 1.90 ( ± 1.76) mm and a Dice similarity coefficient of 97.75 ( ± 1.32) % were achieved, overcoming approaches in the literature. CONCLUSIONS The experimental results showed the effectiveness of the proposed framework, proving its potential in supporting clinicians during the clinical practice.
Collapse
Affiliation(s)
- Maria Chiara Fiorentino
- Department of Information Engineering, Universita Politecnica delle Marche, Via Brecce Bianche, 12, Ancona 60131, Italy
| | - Sara Moccia
- Department of Information Engineering, Universita Politecnica delle Marche, Via Brecce Bianche, 12, Ancona 60131, Italy; Department of Advanced Robotics, Istituto Italiano di Tecnologia, Via Morego, 30, Genova 16163, Italy.
| | - Morris Capparuccini
- Department of Information Engineering, Universita Politecnica delle Marche, Via Brecce Bianche, 12, Ancona 60131, Italy
| | - Sara Giamberini
- Department of Information Engineering, Universita Politecnica delle Marche, Via Brecce Bianche, 12, Ancona 60131, Italy
| | - Emanuele Frontoni
- Department of Information Engineering, Universita Politecnica delle Marche, Via Brecce Bianche, 12, Ancona 60131, Italy
| |
Collapse
|
32
|
von Atzigen M, Liebmann F, Hoch A, Bauer DE, Snedeker JG, Farshad M, Fürnstahl P. HoloYolo: A proof-of-concept study for marker-less surgical navigation of spinal rod implants with augmented reality and on-device machine learning. Int J Med Robot 2020; 17:1-10. [PMID: 33073908 DOI: 10.1002/rcs.2184] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2020] [Revised: 10/12/2020] [Accepted: 10/14/2020] [Indexed: 12/14/2022]
Abstract
BACKGROUND Existing surgical navigation approaches of the rod bending procedure in spinal fusion rely on optical tracking systems that determine the location of placed pedicle screws using a hand-held marker. METHODS We propose a novel, marker-less surgical navigation proof-of-concept to bending rod implants. Our method combines augmented reality with on-device machine learning to generate and display a virtual template of the optimal rod shape without touching the instrumented anatomy. Performance was evaluated on lumbosacral spine phantoms against a pointer-based navigation benchmark approach and ground truth data obtained from computed tomography. RESULTS Our method achieved a mean error of 1.83 ± 1.10 mm compared to 1.87 ± 1.31 mm measured in the marker-based approach, while only requiring 21.33 ± 8.80 s as opposed to 36.65 ± 7.49 s attained by the pointer-based method. CONCLUSION Our results suggests that the combination of augmented reality and machine learning has the potential to replace conventional pointer-based navigation in the future.
Collapse
Affiliation(s)
- Marco von Atzigen
- Research in Orthopedic Computer Science, Balgrist University Hospital, University of Zurich, Zurich, Switzerland.,Laboratory for Orthopaedic Biomechanics, ETH Zurich, Zurich, Switzerland
| | - Florentin Liebmann
- Research in Orthopedic Computer Science, Balgrist University Hospital, University of Zurich, Zurich, Switzerland.,Laboratory for Orthopaedic Biomechanics, ETH Zurich, Zurich, Switzerland
| | - Armando Hoch
- Research in Orthopedic Computer Science, Balgrist University Hospital, University of Zurich, Zurich, Switzerland.,Orthopaedic Department, Balgrist University Hospital, University of Zurich, Zurich, Switzerland
| | - David E Bauer
- Orthopaedic Department, Balgrist University Hospital, University of Zurich, Zurich, Switzerland
| | - Jess Gerrit Snedeker
- Laboratory for Orthopaedic Biomechanics, ETH Zurich, Zurich, Switzerland.,Orthopaedic Department, Balgrist University Hospital, University of Zurich, Zurich, Switzerland
| | - Mazda Farshad
- Orthopaedic Department, Balgrist University Hospital, University of Zurich, Zurich, Switzerland
| | - Philipp Fürnstahl
- Research in Orthopedic Computer Science, Balgrist University Hospital, University of Zurich, Zurich, Switzerland
| |
Collapse
|
33
|
Chadebecq F, Vasconcelos F, Mazomenos E, Stoyanov D. Computer Vision in the Surgical Operating Room. Visc Med 2020; 36:456-462. [PMID: 33447601 DOI: 10.1159/000511934] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2020] [Accepted: 09/30/2020] [Indexed: 12/20/2022] Open
Abstract
Background Multiple types of surgical cameras are used in modern surgical practice and provide a rich visual signal that is used by surgeons to visualize the clinical site and make clinical decisions. This signal can also be used by artificial intelligence (AI) methods to provide support in identifying instruments, structures, or activities both in real-time during procedures and postoperatively for analytics and understanding of surgical processes. Summary In this paper, we provide a succinct perspective on the use of AI and especially computer vision to power solutions for the surgical operating room (OR). The synergy between data availability and technical advances in computational power and AI methodology has led to rapid developments in the field and promising advances. Key Messages With the increasing availability of surgical video sources and the convergence of technologies around video storage, processing, and understanding, we believe clinical solutions and products leveraging vision are going to become an important component of modern surgical capabilities. However, both technical and clinical challenges remain to be overcome to efficiently make use of vision-based approaches into the clinic.
Collapse
Affiliation(s)
- François Chadebecq
- Department of Computer Science, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, London, United Kingdom
| | - Francisco Vasconcelos
- Department of Computer Science, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, London, United Kingdom
| | - Evangelos Mazomenos
- Department of Computer Science, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, London, United Kingdom
| | - Danail Stoyanov
- Department of Computer Science, Wellcome/EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, London, United Kingdom
| |
Collapse
|
34
|
Yang C, Zhao Z, Hu S. Image-based laparoscopic tool detection and tracking using convolutional neural networks: a review of the literature. Comput Assist Surg (Abingdon) 2020; 25:15-28. [PMID: 32886540 DOI: 10.1080/24699322.2020.1801842] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022] Open
Abstract
Intraoperative detection and tracking of minimally invasive instruments is a prerequisite for computer- and robotic-assisted surgery. Since additional hardware, such as tracking systems or the robot encoders, are cumbersome and lack accuracy, surgical vision is evolving as a promising technique to detect and track the instruments using only endoscopic images. The present paper presents a review of the literature regarding image-based laparoscopic tool detection and tracking using convolutional neural networks (CNNs) and consists of four primary parts: (1) fundamentals of CNN; (2) public datasets; (3) CNN-based methods for the detection and tracking of laparoscopic instruments; and (4) discussion and conclusion. To help researchers quickly understand the various existing CNN-based algorithms, some basic information and a quantitative estimation of several performances are analyzed and compared from the perspective of 'partial CNN approaches' and 'full CNN approaches'. Moreover, we highlight the challenges related to research of CNN-based detection algorithms and provide possible future developmental directions.
Collapse
Affiliation(s)
- Congmin Yang
- School of Control Science and Engineering, Shandong University, Jinan, China
| | - Zijian Zhao
- School of Control Science and Engineering, Shandong University, Jinan, China
| | - Sanyuan Hu
- Department of General surgery, First Affiliated Hospital of Shandong First Medical University, Jinan, China
| |
Collapse
|
35
|
Chu Y, Yang X, Li H, Ai D, Ding Y, Fan J, Song H, Yang J. Multi-level feature aggregation network for instrument identification of endoscopic images. Phys Med Biol 2020; 65:165004. [PMID: 32344381 DOI: 10.1088/1361-6560/ab8dda] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
Identification of surgical instruments is crucial in understanding surgical scenarios and providing an assistive process in endoscopic image-guided surgery. This study proposes a novel multilevel feature-aggregated deep convolutional neural network (MLFA-Net) for identifying surgical instruments in endoscopic images. First, a global feature augmentation layer is created on the top layer of the backbone to improve the localization ability of object identification by boosting the high-level semantic information to the feature flow network. Second, a modified interaction path of cross-channel features is proposed to increase the nonlinear combination of features in the same level and improve the efficiency of information propagation. Third, a multiview fusion branch of features is built to aggregate the location-sensitive information of the same level in different views, increase the information diversity of features, and enhance the localization ability of objects. By utilizing the latent information, the proposed network of multilevel feature aggregation can accomplish multitask instrument identification with a single network. Three tasks are handled by the proposed network, including object detection, which classifies the type of instrument and locates its border; mask segmentation, which detects the instrument shape; and pose estimation, which detects the keypoint of instrument parts. The experiments are performed on laparoscopic images from MICCAI 2017 Endoscopic Vision Challenge, and the mean average precision (AP) and average recall (AR) are utilized to quantify the segmentation and pose estimation results. For the bounding box regression, the AP and AR are 79.1% and 63.2%, respectively, while the AP and AR of mask segmentation are 78.1% and 62.1%, and the AP and AR of the pose estimation achieve 67.1% and 55.7%, respectively. The experiments demonstrate that our method efficiently improves the recognition accuracy of the instrument in endoscopic images, and outperforms the other state-of-the-art methods.
Collapse
Affiliation(s)
- Yakui Chu
- Beijing Engineering Research Center of Mixed Reality and Advanced Display, School of Optics and Photonics, Beijing Institute of Technology, Beijing 100081 People's Republic of China. Authors contribute equally to this article
| | | | | | | | | | | | | | | |
Collapse
|
36
|
Sun Y, Pan B, Guo Y, Fu Y, Niu G. Vision-based hand-eye calibration for robot-assisted minimally invasive surgery. Int J Comput Assist Radiol Surg 2020; 15:2061-2069. [PMID: 32808149 DOI: 10.1007/s11548-020-02245-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2020] [Accepted: 08/07/2020] [Indexed: 11/24/2022]
Abstract
PURPOSE The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand-eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand-eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand-eye transformation is still a critical problem. METHODS We propose a novel hand-eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model. RESULTS Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand-eye calibration without calibration object. CONCLUSION Vision-based hand-eye calibration is developed. We demonstrate the feasibility to perform hand-eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.
Collapse
Affiliation(s)
- Yanwen Sun
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Bo Pan
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China.
| | - Yongchen Guo
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Yili Fu
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China
| | - Guojun Niu
- School of Mechanical Engineering and Automation, Zhejiang Sci-Tech University, Hangzhou, China
| |
Collapse
|
37
|
Zhong F, Wang Z, Chen W, He K, Wang Y, Liu YH. Hand-Eye Calibration of Surgical Instrument for Robotic Surgery Using Interactive Manipulation. IEEE Robot Autom Lett 2020. [DOI: 10.1109/lra.2020.2967685] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
38
|
Ma H, Smal I, Daemen J, Walsum TV. Dynamic coronary roadmapping via catheter tip tracking in X-ray fluoroscopy with deep learning based Bayesian filtering. Med Image Anal 2020; 61:101634. [DOI: 10.1016/j.media.2020.101634] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2019] [Revised: 10/26/2019] [Accepted: 01/02/2020] [Indexed: 10/25/2022]
|
39
|
Moccia R, Iacono C, Siciliano B, Ficuciello F. Vision-Based Dynamic Virtual Fixtures for Tools Collision Avoidance in Robotic Surgery. IEEE Robot Autom Lett 2020. [DOI: 10.1109/lra.2020.2969941] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
40
|
Mikada T, Kanno T, Kawase T, Miyazaki T, Kawashima K. Three-dimensional posture estimation of robot forceps using endoscope with convolutional neural network. Int J Med Robot 2020; 16:e2062. [PMID: 31913577 PMCID: PMC7154714 DOI: 10.1002/rcs.2062] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2019] [Revised: 11/27/2019] [Accepted: 11/28/2019] [Indexed: 11/10/2022]
Abstract
BACKGROUND In recent years, there has been significant developments in surgical robots. Image-based sensing of surgical instruments, without the use of electric sensors, are preferred for easily washable robots. METHODS We propose a method to estimate the three-dimensional posture of the tip of the forceps tip by using an endoscopic image. A convolutional neural network (CNN) receives the image of the tracked markers attached to the forceps as an input and outputs the posture of the forceps. RESULTS The posture estimation results showed that the posture estimated from the image followed the electrical sensor. The estimated results of the external force calculated based on the posture also followed the measured values. CONCLUSION The method which estimates the forceps posture from the image using CNN is effective. The mean absolute error of the estimated external force is smaller than the human detection limit.
Collapse
Affiliation(s)
- Takuto Mikada
- Department of Biomechanics, Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan
| | - Takahiro Kanno
- Department of Biomechanics, Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan
| | - Toshihiro Kawase
- Department of Biomechanics, Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan.,Institute of Innovative Research, Tokyo Institute of Technology, Yokohama, Japan
| | - Tetsuro Miyazaki
- Department of Biomechanics, Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan
| | - Kenji Kawashima
- Department of Biomechanics, Institute of Biomaterials and Bioengineering, Tokyo Medical and Dental University, Tokyo, Japan
| |
Collapse
|
41
|
Moccia S, Romeo L, Migliorelli L, Frontoni E, Zingaretti P. Supervised CNN Strategies for Optical Image Segmentation and Classification in Interventional Medicine. INTELLIGENT SYSTEMS REFERENCE LIBRARY 2020. [DOI: 10.1007/978-3-030-42750-4_8] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
|
42
|
Vercauteren T, Unberath M, Padoy N, Navab N. CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions. PROCEEDINGS OF THE IEEE. INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS 2020; 108:198-214. [PMID: 31920208 PMCID: PMC6952279 DOI: 10.1109/jproc.2019.2946993] [Citation(s) in RCA: 41] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/01/2019] [Revised: 09/12/2019] [Accepted: 10/04/2019] [Indexed: 05/10/2023]
Abstract
Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.
Collapse
Affiliation(s)
- Tom Vercauteren
- School of Biomedical Engineering & Imaging SciencesKing’s College LondonLondonWC2R 2LSU.K.
| | - Mathias Unberath
- Department of Computer ScienceJohns Hopkins UniversityBaltimoreMD21218USA
| | - Nicolas Padoy
- ICube institute, CNRS, IHU Strasbourg, University of Strasbourg67081StrasbourgFrance
| | - Nassir Navab
- Fakultät für InformatikTechnische Universität München80333MunichGermany
| |
Collapse
|
43
|
Moccia S, Migliorelli L, Carnielli V, Frontoni E. Preterm Infants' Pose Estimation With Spatio-Temporal Features. IEEE Trans Biomed Eng 2019; 67:2370-2380. [PMID: 31870974 DOI: 10.1109/tbme.2019.2961448] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECTIVE Preterm infants' limb monitoring in neonatal intensive care units (NICUs) is of primary importance for assessing infants' health status and motor/cognitive development. Herein, we propose a new approach to preterm infants' limb pose estimation that features spatio-temporal information to detect and track limb joints from depth videos with high reliability. METHODS Limb-pose estimation is performed using a deep-learning framework consisting of a detection and a regression convolutional neural network (CNN) for rough and precise joint localization, respectively. The CNNs are implemented to encode connectivity in the temporal direction through 3D convolution. Assessment of the proposed framework is performed through a comprehensive study with sixteen depth videos acquired in the actual clinical practice from sixteen preterm infants (the babyPose dataset). RESULTS When applied to pose estimation, the median root mean square distance, computed among all limbs, between the estimated and the ground-truth pose was 9.06 pixels, overcoming approaches based on spatial features only (11.27 pixels). CONCLUSION Results showed that the spatio-temporal features had a significant influence on the pose-estimation performance, especially in challenging cases (e.g., homogeneous image intensity). SIGNIFICANCE This article significantly enhances the state of art in automatic assessment of preterm infants' health status by introducing the use of spatio-temporal features for limb detection and tracking, and by being the first study to use depth videos acquired in the actual clinical practice for limb-pose estimation. The babyPose dataset has been released as the first annotated dataset for infants' pose estimation.
Collapse
|
44
|
Inter-foetus Membrane Segmentation for TTTS Using Adversarial Networks. Ann Biomed Eng 2019; 48:848-859. [DOI: 10.1007/s10439-019-02424-9] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2019] [Accepted: 11/23/2019] [Indexed: 12/18/2022]
|
45
|
Zhao Z, Cai T, Chang F, Cheng X. Real-time surgical instrument detection in robot-assisted surgery using a convolutional neural network cascade. Healthc Technol Lett 2019; 6:275-279. [PMID: 32038871 PMCID: PMC6952255 DOI: 10.1049/htl.2019.0064] [Citation(s) in RCA: 24] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2019] [Accepted: 10/02/2019] [Indexed: 12/24/2022] Open
Abstract
Surgical instrument detection in robot-assisted surgery videos is an import vision component for these systems. Most of the current deep learning methods focus on single-tool detection and suffer from low detection speed. To address this, the authors propose a novel frame-by-frame detection method using a cascading convolutional neural network (CNN) which consists of two different CNNs for real-time multi-tool detection. An hourglass network and a modified visual geometry group (VGG) network are applied to jointly predict the localisation. The former CNN outputs detection heatmaps representing the location of tool tip areas, and the latter performs bounding-box regression for tool tip areas on these heatmaps stacked with input RGB image frames. The authors’ method is tested on the publicly available EndoVis Challenge dataset and the ATLAS Dione dataset. The experimental results show that their method achieves better performance than mainstream detection methods in terms of detection accuracy and speed.
Collapse
Affiliation(s)
- Zijian Zhao
- School of Control Science and Engineering, Jinan, Shandong, People's Republic of China
| | - Tongbiao Cai
- School of Control Science and Engineering, Jinan, Shandong, People's Republic of China
| | - Faliang Chang
- School of Control Science and Engineering, Jinan, Shandong, People's Republic of China
| | - Xiaolin Cheng
- Laboratory of Laparoscopic Technique and Engineering, Qilu Hospital of Shandong University, Jinan, Shandong, People's Republic of China
| |
Collapse
|
46
|
Colleoni E, Moccia S, Du X, De Momi E, Stoyanov D. Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2019.2917163] [Citation(s) in RCA: 57] [Impact Index Per Article: 11.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
47
|
Funke I, Mees ST, Weitz J, Speidel S. Video-based surgical skill assessment using 3D convolutional neural networks. Int J Comput Assist Radiol Surg 2019; 14:1217-1225. [PMID: 31104257 DOI: 10.1007/s11548-019-01995-1] [Citation(s) in RCA: 66] [Impact Index Per Article: 13.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2019] [Accepted: 05/08/2019] [Indexed: 11/29/2022]
Abstract
PURPOSE A profound education of novice surgeons is crucial to ensure that surgical interventions are effective and safe. One important aspect is the teaching of technical skills for minimally invasive or robot-assisted procedures. This includes the objective and preferably automatic assessment of surgical skill. Recent studies presented good results for automatic, objective skill evaluation by collecting and analyzing motion data such as trajectories of surgical instruments. However, obtaining the motion data generally requires additional equipment for instrument tracking or the availability of a robotic surgery system to capture kinematic data. In contrast, we investigate a method for automatic, objective skill assessment that requires video data only. This has the advantage that video can be collected effortlessly during minimally invasive and robot-assisted training scenarios. METHODS Our method builds on recent advances in deep learning-based video classification. Specifically, we propose to use an inflated 3D ConvNet to classify snippets, i.e., stacks of a few consecutive frames, extracted from surgical video. The network is extended into a temporal segment network during training. RESULTS We evaluate the method on the publicly available JIGSAWS dataset, which consists of recordings of basic robot-assisted surgery tasks performed on a dry lab bench-top model. Our approach achieves high skill classification accuracies ranging from 95.1 to 100.0%. CONCLUSIONS Our results demonstrate the feasibility of deep learning-based assessment of technical skill from surgical video. Notably, the 3D ConvNet is able to learn meaningful patterns directly from the data, alleviating the need for manual feature engineering. Further evaluation will require more annotated data for training and testing.
Collapse
Affiliation(s)
- Isabel Funke
- Division of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), Partner Site Dresden, Dresden, Germany.
| | - Sören Torge Mees
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Jürgen Weitz
- Department of Visceral, Thoracic and Vascular Surgery, Faculty of Medicine and University Hospital Carl Gustav Carus, TU Dresden, Dresden, Germany
| | - Stefanie Speidel
- Division of Translational Surgical Oncology, National Center for Tumor Diseases (NCT), Partner Site Dresden, Dresden, Germany
| |
Collapse
|
48
|
Objective assessment of intraoperative technical skill in capsulorhexis using videos of cataract surgery. Int J Comput Assist Radiol Surg 2019; 14:1097-1105. [PMID: 30977091 DOI: 10.1007/s11548-019-01956-8] [Citation(s) in RCA: 22] [Impact Index Per Article: 4.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2019] [Accepted: 03/27/2019] [Indexed: 10/27/2022]
Abstract
PURPOSE Objective assessment of intraoperative technical skill is necessary for technology to improve patient care through surgical training. Our objective in this study was to develop and validate deep learning techniques for technical skill assessment using videos of the surgical field. METHODS We used a data set of 99 videos of capsulorhexis, a critical step in cataract surgery. One expert surgeon annotated each video for technical skill using a standard structured rating scale, the International Council of Ophthalmology's Ophthalmology Surgical Competency Assessment Rubric:phacoemulsification (ICO-OSCAR:phaco). Using two capsulorhexis indices in this scale (commencement of flap and follow-through, formation and completion), we specified an expert performance when at least one of the indices was 5 and the other index was at least 4, and novice otherwise. In addition, we used scores for capsulorhexis commencement and capsulorhexis formation as separate ground truths (Likert scale of 2 to 5; analyzed as 2/3, 4 and 5). We crowdsourced annotations of instrument tips. We separately modeled instrument trajectories and optical flow using temporal convolutional neural networks to predict a skill class (expert/novice) and score on each item for capsulorhexis in ICO-OSCAR:phaco. We evaluated the algorithms in a five-fold cross-validation and computed accuracy and area under the receiver operating characteristics curve (AUC). RESULTS The accuracy and AUC were 0.848 and 0.863 for instrument tip velocities, and 0.634 and 0.803 for optical flow fields, respectively. CONCLUSIONS Deep neural networks effectively model surgical technical skill in capsulorhexis given structured representation of intraoperative data such as optical flow fields extracted from video or crowdsourced tool localization information.
Collapse
|
49
|
Lin J, Walsted ES, Backer V, Hull JH, Elson DS. Quantification and Analysis of Laryngeal Closure From Endoscopic Videos. IEEE Trans Biomed Eng 2019; 66:1127-1136. [DOI: 10.1109/tbme.2018.2867636] [Citation(s) in RCA: 17] [Impact Index Per Article: 3.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
50
|
Al Hajj H, Lamard M, Conze PH, Roychowdhury S, Hu X, Maršalkaitė G, Zisimopoulos O, Dedmari MA, Zhao F, Prellberg J, Sahu M, Galdran A, Araújo T, Vo DM, Panda C, Dahiya N, Kondo S, Bian Z, Vahdat A, Bialopetravičius J, Flouty E, Qiu C, Dill S, Mukhopadhyay A, Costa P, Aresta G, Ramamurthy S, Lee SW, Campilho A, Zachow S, Xia S, Conjeti S, Stoyanov D, Armaitis J, Heng PA, Macready WG, Cochener B, Quellec G. CATARACTS: Challenge on automatic tool annotation for cataRACT surgery. Med Image Anal 2018; 52:24-41. [PMID: 30468970 DOI: 10.1016/j.media.2018.11.008] [Citation(s) in RCA: 34] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2018] [Revised: 11/13/2018] [Accepted: 11/15/2018] [Indexed: 12/29/2022]
Abstract
Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.
Collapse
Affiliation(s)
| | - Mathieu Lamard
- Inserm, UMR 1101, Brest, F-29200, France; Univ Bretagne Occidentale, Brest, F-29200, France
| | - Pierre-Henri Conze
- Inserm, UMR 1101, Brest, F-29200, France; IMT Atlantique, LaTIM UMR 1101, UBL, Brest, F-29200, France
| | | | - Xiaowei Hu
- Dept. of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | | | | | - Muneer Ahmad Dedmari
- Chair for Computer Aided Medical Procedures, Faculty of Informatics, Technical University of Munich, Garching b. Munich, 85748, Germany
| | - Fenqiang Zhao
- Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, HangZhou, 310000, China
| | - Jonas Prellberg
- Dept. of Informatics, Carl von Ossietzky University, Oldenburg, 26129, Germany
| | - Manish Sahu
- Department of Visual Data Analysis, Zuse Institute Berlin, Berlin, 14195, Germany
| | - Adrian Galdran
- INESC TEC - Instituto de Engenharia de Sistemas e Computadores - Tecnologia e Ciência, Porto, 4200-465, Portugal
| | - Teresa Araújo
- Faculdade de Engenharia, Universidade do Porto, Porto, 4200-465, Portugal; INESC TEC - Instituto de Engenharia de Sistemas e Computadores - Tecnologia e Ciência, Porto, 4200-465, Portugal
| | - Duc My Vo
- Gachon University, 1342 Seongnamdaero, Sujeonggu, Seongnam, 13120, Korea
| | | | - Navdeep Dahiya
- Laboratory of Computational Computer Vision, Georgia Tech, Atlanta, GA, 30332, USA
| | | | | | - Arash Vahdat
- D-Wave Systems Inc., Burnaby, BC, V5G 4M9, Canada
| | | | | | - Chenhui Qiu
- Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, HangZhou, 310000, China
| | - Sabrina Dill
- Department of Visual Data Analysis, Zuse Institute Berlin, Berlin, 14195, Germany
| | - Anirban Mukhopadhyay
- Department of Computer Science, Technische Universität Darmstadt, Darmstadt, 64283, Germany
| | - Pedro Costa
- INESC TEC - Instituto de Engenharia de Sistemas e Computadores - Tecnologia e Ciência, Porto, 4200-465, Portugal
| | - Guilherme Aresta
- Faculdade de Engenharia, Universidade do Porto, Porto, 4200-465, Portugal; INESC TEC - Instituto de Engenharia de Sistemas e Computadores - Tecnologia e Ciência, Porto, 4200-465, Portugal
| | - Senthil Ramamurthy
- Laboratory of Computational Computer Vision, Georgia Tech, Atlanta, GA, 30332, USA
| | - Sang-Woong Lee
- Gachon University, 1342 Seongnamdaero, Sujeonggu, Seongnam, 13120, Korea
| | - Aurélio Campilho
- Faculdade de Engenharia, Universidade do Porto, Porto, 4200-465, Portugal; INESC TEC - Instituto de Engenharia de Sistemas e Computadores - Tecnologia e Ciência, Porto, 4200-465, Portugal
| | - Stefan Zachow
- Department of Visual Data Analysis, Zuse Institute Berlin, Berlin, 14195, Germany
| | - Shunren Xia
- Key Laboratory of Biomedical Engineering of Ministry of Education, Zhejiang University, HangZhou, 310000, China
| | - Sailesh Conjeti
- Chair for Computer Aided Medical Procedures, Faculty of Informatics, Technical University of Munich, Garching b. Munich, 85748, Germany; German Center for Neurodegenrative Diseases (DZNE), Bonn, 53127, Germany
| | - Danail Stoyanov
- Digital Surgery Ltd, EC1V 2QY, London, UK; University College London, Gower Street, WC1E 6BT, London, UK
| | | | - Pheng-Ann Heng
- Dept. of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
| | | | - Béatrice Cochener
- Inserm, UMR 1101, Brest, F-29200, France; Univ Bretagne Occidentale, Brest, F-29200, France; Service d'Ophtalmologie, CHRU Brest, Brest, F-29200, France
| | | |
Collapse
|