1
|
Liu Z, Gao W, Zhu J, Yu Z, Fu Y. Surface deformation tracking in monocular laparoscopic video. Med Image Anal 2023; 86:102775. [PMID: 36848721 DOI: 10.1016/j.media.2023.102775] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2022] [Revised: 02/17/2023] [Accepted: 02/18/2023] [Indexed: 02/23/2023]
Abstract
Image-guided surgery has been proven to enhance the accuracy and safety of minimally invasive surgery (MIS). Nonrigid deformation tracking of soft tissue is one of the main challenges in image-guided MIS owing to the existence of tissue deformation, homogeneous texture, smoke and instrument occlusion, etc. In this paper, we proposed a piecewise affine deformation model-based nonrigid deformation tracking method. A Markov random field based mask generation method is developed to eliminate tracking anomalies. The deformation information vanishes when the regular constraint is invalid, which further deteriorates the tracking accuracy. Atime-series deformation solidification mechanism is introduced to reduce the degradation of the deformation field of the model. For the quantitative evaluation of the proposed method, we synthesized nine laparoscopic videos mimicking instrument occlusion and tissue deformation. Quantitative tracking robustness was evaluated on the synthetic videos. Three real videos of MIS containing challenges of large-scale deformation, large-range smoke, instrument occlusion, and permanent changes in soft tissue texture were also used to evaluate the performance of the proposed method. Experimental results indicate the proposed method outperforms state-of-the-art methods in terms of accuracy and robustness, which shows good performance in image-guided MIS.
Collapse
Affiliation(s)
- Ziteng Liu
- School of Life Science and Technology, Harbin Institute of Technology, 2 Yikuang Str., Nangang District, Harbin, 150080, China
| | - Wenpeng Gao
- School of Life Science and Technology, Harbin Institute of Technology, 2 Yikuang Str., Nangang District, Harbin, 150080, China.
| | - Jiahua Zhu
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, 2 Yikuang Str., Nangang District, Harbin, 150080, China
| | - Zhi Yu
- School of Life Science and Technology, Harbin Institute of Technology, 2 Yikuang Str., Nangang District, Harbin, 150080, China
| | - Yili Fu
- State Key Laboratory of Robotics and System, Harbin Institute of Technology, 2 Yikuang Str., Nangang District, Harbin, 150080, China.
| |
Collapse
|
2
|
Zheng Q, Yang R, Ni X, Yang S, Jiang Z, Wang L, Chen Z, Liu X. Development and validation of a deep learning-based laparoscopic system for improving video quality. Int J Comput Assist Radiol Surg 2023; 18:257-268. [PMID: 36243805 DOI: 10.1007/s11548-022-02777-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2022] [Accepted: 10/05/2022] [Indexed: 02/03/2023]
Abstract
PURPOSE A clear surgical field of view is a prerequisite for successful laparoscopic surgery. Surgical smoke, image blur, and lens fogging can affect the clarity of laparoscopic imaging. We aimed to develop a real-time assistance system (namely LVQIS) for removing these interfering factors during laparoscopic surgery, thereby improving laparoscopic video quality. METHODS LVQIS was developed with generative adversarial networks (GAN) and transfer learning, which included two classification models (ResNet-50), a motion blur removal model (MPRNet), and a smoke/fog removal model (GAN). 136 laparoscopic surgery videos were retrospectively collected in a tripartite dataset for training and validation. A synthetic dataset was simulated using the image enhancement library Albumentations and the 3D rendering software Blender. The objective evaluation results were through PSNR, SSIM and FID, and the subjective evaluation includes the operation pause time and the degree of anxiety of surgeons. RESULTS The synthesized dataset contained 19,245 clear images, 19,245 motion blur images, and 19,245 smoke/fog images. The ResNet-50 CNN model identified whether a single laparoscopic image had motion blur and smoke/fog with an accuracy of over 0.99. The PSNR, SSIM and FID of the de-smoke model were 29.67, 0.9551 and 74.72, respectively, and the PSNR, SSIM and FID of the de-blurring model were 26.78, 0.9020 and 80.10, respectively, which were better than other advanced de-blurring and de-smoke/fog models. In a comparative study of 100 laparoscopic surgeries, the use of LVQIS significantly reduced the operation pause time (P < 0.001) and the anxiety of surgeons (P = 0.004). CONCLUSIONS In this study, LVQIS is an efficient and robust system that can improve the quality of laparoscopic video, reduce surgical pause time and the anxiety of surgeons, and has the potential for real-time application in real clinical settings.
Collapse
Affiliation(s)
- Qingyuan Zheng
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Rui Yang
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Xinmiao Ni
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Song Yang
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Zhengyu Jiang
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Lei Wang
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China
| | - Zhiyuan Chen
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China.
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China.
| | - Xiuheng Liu
- Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People's Republic of China.
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China.
| |
Collapse
|