1
|
Hoffmann H, Funke I, Peters P, Venkatesh DK, Egger J, Rivoir D, Röhrig R, Hölzle F, Bodenstedt S, Willemer MC, Speidel S, Puladi B. AIxSuture: vision-based assessment of open suturing skills. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03093-3. [PMID: 38526613 DOI: 10.1007/s11548-024-03093-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 02/28/2024] [Indexed: 03/27/2024]
Abstract
PURPOSE Efficient and precise surgical skills are essential in ensuring positive patient outcomes. By continuously providing real-time, data driven, and objective evaluation of surgical performance, automated skill assessment has the potential to greatly improve surgical skill training. Whereas machine learning-based surgical skill assessment is gaining traction for minimally invasive techniques, this cannot be said for open surgery skills. Open surgery generally has more degrees of freedom when compared to minimally invasive surgery, making it more difficult to interpret. In this paper, we present novel approaches for skill assessment for open surgery skills. METHODS We analyzed a novel video dataset for open suturing training. We provide a detailed analysis of the dataset and define evaluation guidelines, using state of the art deep learning models. Furthermore, we present novel benchmarking results for surgical skill assessment in open suturing. The models are trained to classify a video into three skill levels based on the global rating score. To obtain initial results for video-based surgical skill classification, we benchmarked a temporal segment network with both an I3D and a Video Swin backbone on this dataset. RESULTS The dataset is composed of 314 videos of approximately five minutes each. Model benchmarking results are an accuracy and F1 score of up to 75 and 72%, respectively. This is similar to the performance achieved by the individual raters, regarding inter-rater agreement and rater variability. We present the first end-to-end trained approach for skill assessment for open surgery training. CONCLUSION We provide a thorough analysis of a new dataset as well as novel benchmarking results for surgical skill assessment. This opens the doors to new advances in skill assessment by enabling video-based skill assessment for classic surgical techniques with the potential to improve the surgical outcome of patients.
Collapse
Affiliation(s)
- Hanna Hoffmann
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany.
- The Centre for Tactile Internet (CeTI), TUD Dresden University of Technology, Dresden, Germany.
- Faculty of Medicine, University Hospital Carl Gustav Carus, Dresden, Germany.
- BMBF Research Hub 6 G-Life, TUD Dresden University of Technology, Dresden, Germany.
| | - Isabel Funke
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany
- The Centre for Tactile Internet (CeTI), TUD Dresden University of Technology, Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Philipp Peters
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Danush Kumar Venkatesh
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany
- School of Embedded Composite Artificial Intelligence (SECAI), TUD Dresden University of Technology, Dresden, Germany
- Faculty of Medicine, University Hospital Carl Gustav Carus, Dresden, Germany
| | - Jan Egger
- Institute for AI in Medicine, University Hospital Essen (AöR), Essen, Germany
| | - Dominik Rivoir
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany
- The Centre for Tactile Internet (CeTI), TUD Dresden University of Technology, Dresden, Germany
- German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Rainer Röhrig
- Institute of Medical Informatics, University Hospital RWTH Aachen, Aachen, Germany
| | - Frank Hölzle
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
| | - Sebastian Bodenstedt
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany
- The Centre for Tactile Internet (CeTI), TUD Dresden University of Technology, Dresden, Germany
| | - Marie-Christin Willemer
- MITZ, University Hospital Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
- Faculty of Medicine, University Hospital Carl Gustav Carus, Dresden, Germany
| | - Stefanie Speidel
- Department of Translational Surgical Oncology, NCT/UCC Dresden, Dresden, Germany
- The Centre for Tactile Internet (CeTI), TUD Dresden University of Technology, Dresden, Germany
- Faculty of Medicine, University Hospital Carl Gustav Carus, Dresden, Germany
- BMBF Research Hub 6 G-Life, TUD Dresden University of Technology, Dresden, Germany
| | - Behrus Puladi
- Department of Oral and Maxillofacial Surgery, University Hospital RWTH Aachen, Aachen, Germany
- Institute of Medical Informatics, University Hospital RWTH Aachen, Aachen, Germany
| |
Collapse
|
2
|
Venkatesh DK, Rivoir D, Pfeiffer M, Kolbinger F, Distler M, Weitz J, Speidel S. Exploring semantic consistency in unpaired image translation to generate data for surgical applications. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03079-1. [PMID: 38407730 DOI: 10.1007/s11548-024-03079-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2024] [Accepted: 02/14/2024] [Indexed: 02/27/2024]
Abstract
PURPOSE In surgical computer vision applications, data privacy and expert annotation challenges impede the acquisition of labeled training data. Unpaired image-to-image translation techniques have been explored to automatically generate annotated datasets by translating synthetic images into a realistic domain. The preservation of structure and semantic consistency, i.e., per-class distribution during translation, poses a significant challenge, particularly in cases of semantic distributional mismatch. METHOD This study empirically investigates various translation methods for generating data in surgical applications, explicitly focusing on semantic consistency. Through our analysis, we introduce a novel and simple combination of effective approaches, which we call ConStructS. The defined losses within this approach operate on multiple image patches and spatial resolutions during translation. RESULTS Various state-of-the-art models were extensively evaluated on two challenging surgical datasets. With two different evaluation schemes, the semantic consistency and the usefulness of the translated images on downstream semantic segmentation tasks were evaluated. The results demonstrate the effectiveness of the ConStructS method in minimizing semantic distortion, with images generated by this model showing superior utility for downstream training. CONCLUSION In this study, we tackle semantic inconsistency in unpaired image translation for surgical applications with minimal labeled data. The simple model (ConStructS) enhances consistency during translation and serves as a practical way of generating fully labeled and semantically consistent datasets at minimal cost. Our code is available at https://gitlab.com/nct_tso_public/constructs .
Collapse
Affiliation(s)
- Danush Kumar Venkatesh
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany.
- SECAI, TU Dresden, Dresden, Germany.
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany.
| | - Dominik Rivoir
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| | - Micha Pfeiffer
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
| | - Fiona Kolbinger
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
| | - Marius Distler
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
| | - Jürgen Weitz
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| | - Stefanie Speidel
- Department of Translational Surgical Oncology, National Centre for Tumor Diseases(NCT/UCC), Dresden, 01307, Germany
- SECAI, TU Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine, TU Dresden, 01307, Dresden, Germany
- The Centre for Tactile Internet(CeTI), TU Dresden, Dresden, Germany
| |
Collapse
|