Real-Time FEM-Based Registration of 3-D to 2.5-D Transrectal Ultrasound Images

A Narrative Review of Image Processing Techniques Related to Prostate Ultrasound

Prostate cancer (PCa) poses a significant threat to men's health, with early diagnosis being crucial for improving prognosis and reducing mortality rates. Transrectal ultrasound (TRUS) plays a vital role in the diagnosis and image-guided intervention of PCa. To facilitate physicians with more accurate and efficient computer-assisted diagnosis and interventions, many image processing algorithms in TRUS have been proposed and achieved state-of-the-art performance in several tasks, including prostate gland segmentation, prostate image registration, PCa classification and detection and interventional needle detection. The rapid development of these algorithms over the past 2 decades necessitates a comprehensive summary. As a consequence, this survey provides a narrative review of this field, outlining the evolution of image processing methods in the context of TRUS image analysis and meanwhile highlighting their relevant contributions. Furthermore, this survey discusses current challenges and suggests future research directions to possibly advance this field further.

A Systematic Review of New Imaging Technologies for Robotic Prostatectomy: From Molecular Imaging to Augmented Reality

New imaging technologies play a pivotal role in the current management of patients with prostate cancer. Robotic assisted radical prostatectomy (RARP) is a standard of care for localized disease and through the already imaging-based console subject of research towards combinations of imaging technologies and RARP as well as their impact on surgical outcomes. Therefore, we aimed to provide a comprehensive analysis of the currently available literature for new imaging technologies for RARP. On 24 January 2023, we performed a systematic review of the current literature on Pubmed, Scopus and Web of Science according to the PRISMA guidelines and Oxford levels of evidence. A total of 46 studies were identified of which 19 studies focus on imaging of the primary tumor, 12 studies on the intraoperative tumor detection of lymph nodes and 15 studies on the training of surgeons. While the feasibility of combined approaches using new imaging technologies including MRI, PSMA-PET CT or intraoperatively applied radioactive and fluorescent dyes has been demonstrated, the prospective confirmation of improvements in surgical outcomes is currently ongoing.

Geometric Mapping Evaluation for Real-Time Local Sensor Simulation

Medical augmented reality and simulated test environments struggle in accurately simulating local sensor measurements across large spatial domains while maintaining the proper resolution of information required and real time capability. Here, a simple method for real-time simulation of intraoperative sensors is presented to aid with medical sensor development and professional training. During a surgical intervention, the interaction between medical sensor systems and tissue leads to mechanical deformation of the tissue. Through the inclusion of detailed finite element simulations in a real-time augmented reality system the method presented will allow for more accurate simulation of intraoperative sensor measurements that are independent of the mechanical state of the tissue. This concept uses a coarse, macro-level deformation mesh to maintain both computational speed and the illusion of reality and a simple geometric point mapping method to include detailed fine mesh information. The resulting system allows for flexible simulation of different types of localized sensor measurement techniques. Preliminary simulation results are provided using a real-time capable simulation environment and prove the feasibility of the method.

Ultrasound image guided and mixed reality-based surgical system with real-time soft tissue deformation computing for robotic cervical pedicle screw placement

Cervical pedicle screw (CPS) placement surgery remains technically demanding due to the complicated anatomy with neurovascular structures. State-of-the-art surgical navigation or robotic systems still suffer from the problem of hand-eye coordination and soft tissue deformation. In this study, we aim at tracking the intraoperative soft tissue deformation and constructing a virtual physical fusion surgical scene, and integrating them into the robotic system for CPS placement surgery. Firstly, we propose a real-time deformation computation method based on the prior shape model and intraoperative partial information acquired from ultrasound images. According to the generated posterior shape, the structure representation of deformed target tissue gets updated continuously. Secondly, a hand tremble compensation method is proposed to improve the accuracy and robustness of the virtual-physical calibration procedure, and a mixed reality based surgical scene is further constructed for CPS placement surgery. Thirdly, we integrate the soft tissue deformation method and virtual-physical fusion method into our previously proposed surgical robotic system, and the surgical workflow for CPS placement surgery is introduced. We conducted phantom and animal experiments to evaluate the feasibility and accuracy of the proposed system. Our system yielded a mean surface distance error of 1.52 ± 0.43 mm for soft tissue deformation computing, and an average distance deviation of 1.04 ± 0.27 mm for CPS placement. Results demonstrated that our system involves tremendous clinical application potential. Our proposed system promotes the efficiency and safety of the CPS placement surgery.

Population-based 3D respiratory motion modelling from convolutional autoencoders for 2D ultrasound-guided radiotherapy

Radiotherapy is a widely used treatment modality for various types of cancers. A challenge for precise delivery of radiation to the treatment site is the management of internal motion caused by the patient's breathing, especially around abdominal organs such as the liver. Current image-guided radiation therapy (IGRT) solutions rely on ionising imaging modalities such as X-ray or CBCT, which do not allow real-time target tracking. Ultrasound imaging (US) on the other hand is relatively inexpensive, portable and non-ionising. Although 2D US can be acquired at a sufficient temporal frequency, it doesn't allow for target tracking in multiple planes, while 3D US acquisitions are not adapted for real-time. In this work, a novel deep learning-based motion modelling framework is presented for ultrasound IGRT. Our solution includes an image similarity-based rigid alignment module combined with a deep deformable motion model. Leveraging the representational capabilities of convolutional autoencoders, our deformable motion model associates complex 3D deformations with 2D surrogate US images through a common learned low dimensional representation. The model is trained on a variety of deformations and anatomies which enables it to generate the 3D motion experienced by the liver of a previously unseen subject. During inference, our framework only requires two pre-treatment 3D volumes of the liver at extreme breathing phases and a live 2D surrogate image representing the current state of the organ. In this study, the presented model is evaluated on a 3D+t US data set of 20 volunteers based on image similarity as well as anatomical target tracking performance. We report results that surpass comparable methodologies in both metric categories with a mean tracking error of 3.5±2.4 mm, demonstrating the potential of this technique for IGRT.

Prostate brachytherapy intraoperative dosimetry using a combination of radiographic seed localization with a C-arm and deformed ultrasound prostate contours

PURPOSE

The purpose of the study was to assess the feasibility of performing intraoperative dosimetry for permanent prostate brachytherapy by combining transrectal ultrasound (TRUS) and fluoroscopy/cone beam CT [CBCT] images and accounting for the effect of prostate deformation.

METHODS AND MATERIALS

13 patients underwent TRUS and multiview two-dimensional fluoroscopic imaging partway through the implant, as well as repeat fluoroscopic imaging with the TRUS probe inserted and retracted, and finally three-dimensional CBCT imaging at the end of the implant. The locations of all the implanted seeds were obtained from the fluoroscopy/CBCT images and were registered to prostate contours delineated on the TRUS images based on a common subset of seeds identified on both image sets. Prostate contours were also deformed, using a finite-element model, to take into account the effect of the TRUS probe pressure. Prostate dosimetry parameters were obtained for fluoroscopic and CBCT-dosimetry approaches and compared with the standard-of-care Day-0 postimplant CT dosimetry.

RESULTS

High linear correlation (R² > 0.8) was observed in the measured values of prostate D_90%, V_100%, and V_150%, between the two intraoperative dosimetry approaches. The prostate D_90% and V_100% obtained from intraoperative dosimetry methods were in agreement with the postimplant CT dosimetry. Only the prostate V_150% was on average 4.1% (p-value <0.05) higher in the CBCT-dosimetry approach and 6.7% (p-value <0.05) higher in postimplant CT dosimetry compared with the fluoroscopic dosimetry approach. Deformation of the prostate by the ultrasound probe appeared to have a minimal effect on prostate dosimetry.

CONCLUSIONS

The results of this study have shown that both of the proposed dosimetric evaluation approaches have potential for real-time intraoperative dosimetry.

A new era: artificial intelligence and machine learning in prostate cancer

Artificial intelligence (AI) - the ability of a machine to perform cognitive tasks to achieve a particular goal based on provided data - is revolutionizing and reshaping our health-care systems. The current availability of ever-increasing computational power, highly developed pattern recognition algorithms and advanced image processing software working at very high speeds has led to the emergence of computer-based systems that are trained to perform complex tasks in bioinformatics, medical imaging and medical robotics. Accessibility to 'big data' enables the 'cognitive' computer to scan billions of bits of unstructured information, extract the relevant information and recognize complex patterns with increasing confidence. Computer-based decision-support systems based on machine learning (ML) have the potential to revolutionize medicine by performing complex tasks that are currently assigned to specialists to improve diagnostic accuracy, increase efficiency of throughputs, improve clinical workflow, decrease human resource costs and improve treatment choices. These characteristics could be especially helpful in the management of prostate cancer, with growing applications in diagnostic imaging, surgical interventions, skills training and assessment, digital pathology and genomics. Medicine must adapt to this changing world, and urologists, oncologists, radiologists and pathologists, as high-volume users of imaging and pathology, need to understand this burgeoning science and acknowledge that the development of highly accurate AI-based decision-support applications of ML will require collaboration between data scientists, computer researchers and engineers.

Marker-less real-time intra-operative camera and hand-eye calibration procedure for surgical augmented reality

Accurate medical Augmented Reality (AR) rendering requires two calibrations, a camera intrinsic matrix estimation and a hand-eye transformation. We present a unified, practical, marker-less, real-time system to estimate both these transformations during surgery. For camera calibration we perform calibrations at multiple distances from the endoscope, pre-operatively, to parametrize the camera intrinsic matrix as a function of distance from the endoscope. Then, we retrieve the camera parameters intra-operatively by estimating the distance of the surgical site from the endoscope in less than 1 s. Unlike in prior work, our method does not require the endoscope to be taken out of the patient; for the hand-eye calibration, as opposed to conventional methods that require the identification of a marker, we make use of a rendered tool-tip in 3D. As the surgeon moves the instrument and observes the offset between the actual and the rendered tool-tip, they can select points of high visual error and manually bring the instrument tip to match the virtual rendered tool tip. To evaluate the hand-eye calibration, 5 subjects carried out the hand-eye calibration procedure on a da Vinci robot. Average Target Registration Error of approximately 7mm was achieved with just three data points.

A partial augmented reality system with live ultrasound and registered preoperative MRI for guiding robot-assisted radical prostatectomy

We propose an image guidance system for robot assisted laparoscopic radical prostatectomy (RALRP). A virtual 3D reconstruction of the surgery scene is displayed underneath the endoscope's feed on the surgeon's console. This scene consists of an annotated preoperative Magnetic Resonance Image (MRI) registered to intraoperative 3D Trans-rectal Ultrasound (TRUS) as well as real-time sagittal 2D TRUS images of the prostate, 3D models of the prostate, the surgical instrument and the TRUS transducer. We display these components with accurate real-time coordinates with respect to the robot system. Since the scene is rendered from the viewpoint of the endoscope, given correct parameters of the camera, an augmented scene can be overlaid on the video output. The surgeon can rotate the ultrasound transducer and determine the position of the projected axial plane in the MRI using one of the registered da Vinci instruments. This system was tested in the laboratory on custom-made agar prostate phantoms. We achieved an average total registration accuracy of 3.2  ±  1.3 mm. We also report on the successful application of this system in the operating room in 12 patients. The average registration error between the TRUS and the da Vinci system for the last 8 patients was 1.4  ±  0.3 mm and average target registration error of 2.1  ±  0.8 mm, resulting in an in vivo overall robot system to MRI mean registration error of 3.5 mm or less, which is consistent with our laboratory studies.

On the feasibility of transperineal 3D ultrasound image guidance for robotic radical prostatectomy

PURPOSE

Prostate cancer is the most prevalent form of male-specific cancers. Robot-assisted laparoscopic radical prostatectomy (RALRP) using the da Vinci surgical robot has become the gold-standard treatment for organ-confined prostate cancer. To improve intraoperative visualization of anatomical structures, many groups have developed techniques integrating transrectal ultrasound (TRUS) into the surgical workflow. TRUS, however, is intrusive and does not provide real-time volumetric imaging.

METHODS

We propose a proof-of-concept system offering an alternative noninvasive transperineal view of the prostate and surrounding structures using 3D ultrasound (US), allowing for full-volume imaging in any anatomical plane desired. The system aims to automatically track da Vinci surgical instruments and display a real-time US image registered to preoperative MRI. We evaluate the approach using a custom prostate phantom, an iU22 (Philips Healthcare, Bothell, WA) US machine with an xMATRIX X6-1 transducer, and a custom probe fixture. A novel registration method between the da Vinci kinematic frame and 3D US is presented. To evaluate the entire registration pipeline, we use a previously developed MRI to US deformable registration algorithm.

RESULTS

Our US calibration technique yielded a registration error of 0.84 mm, compared to 1.76 mm with existing methods. We evaluated overall system error with a prostate phantom, achieving a target registration error of 2.55 mm.

CONCLUSION

Transperineal imaging using 3D US is a promising approach for image guidance during RALRP. Preliminary results suggest this system is comparable to existing guidance systems using TRUS. With further development and testing, we believe our system has the potential to improve patient outcomes by imaging anatomical structures and prostate cancer in real time.