Video-based augmented reality combining CT-scan and instrument position data to microscope view in middle ear surgery

Registration of preoperative temporal bone CT-scan to otoendoscopic video for augmented-reality based on convolutional neural networks

PURPOSE

Patient-to-image registration is a preliminary step required in surgical navigation based on preoperative images. Human intervention and fiducial markers hamper this task as they are time-consuming and introduce potential errors. We aimed to develop a fully automatic 2D registration system for augmented reality in ear surgery.

METHODS

CT-scans and corresponding oto-endoscopic videos were collected from 41 patients (58 ears) undergoing ear examination (vestibular schwannoma before surgery, profound hearing loss requiring cochlear implant, suspicion of perilymphatic fistula, contralateral ears in cases of unilateral chronic otitis media). Two to four images were selected from each case. For the training phase, data from patients (75% of the dataset) and 11 cadaveric specimens were used. Tympanic membranes and malleus handles were contoured on both video images and CT-scans by expert surgeons. The algorithm used a U-Net network for detecting the contours of the tympanic membrane and the malleus on both preoperative CT-scans and endoscopic video frames. Then, contours were processed and registered through an iterative closest point algorithm. Validation was performed on 4 cases and testing on 6 cases. Registration error was measured by overlaying both images and measuring the average and Hausdorff distances.

RESULTS

The proposed registration method yielded a precision compatible with ear surgery with a 2D mean overlay error of 0.65 ± 0.60 mm for the incus and 0.48 ± 0.32 mm for the round window. The average Hausdorff distance for these 2 targets was 0.98 ± 0.60 mm and 0.78 ± 0.34 mm respectively. An outlier case with higher errors (2.3 mm and 1.5 mm average Hausdorff distance for incus and round window respectively) was observed in relation to a high discrepancy between the projection angle of the reconstructed CT-scan and the video image. The maximum duration for the overall process was 18 s.

CONCLUSIONS

A fully automatic 2D registration method based on a convolutional neural network and applied to ear surgery was developed. The method did not rely on any external fiducial markers nor human intervention for landmark recognition. The method was fast and its precision was compatible with ear surgery.

Image-to-Patient Registration in Computer-Assisted Surgery of Head and Neck: State-of-the-Art, Perspectives, and Challenges

Today, image-guided systems play a significant role in improving the outcome of diagnostic and therapeutic interventions. They provide crucial anatomical information during the procedure to decrease the size and the extent of the approach, to reduce intraoperative complications, and to increase accuracy, repeatability, and safety. Image-to-patient registration is the first step in image-guided procedures. It establishes a correspondence between the patient's preoperative imaging and the intraoperative data. When it comes to the head-and-neck region, the presence of many sensitive structures such as the central nervous system or the neurosensory organs requires a millimetric precision. This review allows evaluating the characteristics and the performances of different registration methods in the head-and-neck region used in the operation room from the perspectives of accuracy, invasiveness, and processing times. Our work led to the conclusion that invasive marker-based methods are still considered as the gold standard of image-to-patient registration. The surface-based methods are recommended for faster procedures and applied on the surface tissues especially around the eyes. In the near future, computer vision technology is expected to enhance these systems by reducing human errors and cognitive load in the operating room.

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

Deep learning-based approach for the automatic segmentation of adult and pediatric temporal bone computed tomography images

Background

Automatic segmentation of temporal bone computed tomography (CT) images is fundamental to image-guided otologic surgery and the intelligent analysis of CT images in the field of otology. This study was conducted to test a convolutional neural network (CNN) model that can automatically segment almost all temporal bone anatomy structures in adult and pediatric CT images.

Methods

A dataset comprising 80 annotated CT volumes was collected, of which 40 samples were obtained from adults and 40 from children. A further 60 annotated CT volumes (30 from adults and 30 from children) were used to train the model. The remaining 20 annotated CT volumes were employed to determine the model's generalizability for automatic segmentation. Finally, the Dice coefficient (DC) and average symmetric surface distance (ASSD) were utilized as metrics to evaluate the performance of the CNN model. Two independent-sample t-tests were used to compare the test set results of adults and children.

Results

In the adult test set, the mean DC values of all the structures ranged from 0.714 to 0.912, and the ASSD values were less than 0.24 mm for 11 structures. In the pediatric test set, the mean DC values of all the structures ranged from 0.658 to 0.915, and the ASSD values were less than 0.18 mm for 11 structures. There was no statistically significant difference between the adult and child test sets in most temporal bone structures.

Conclusions

Our CNN model shows excellent automatic segmentation performance and good generalizability for both adult and pediatric temporal bone CT images, which can help to advance otologist education, intelligent imaging diagnosis, surgery simulation, application of augmented reality, and preoperative planning for image-guided otology surgery.

Examining the utility of a photorealistic virtual ear in otologic education

BACKGROUND

Otolaryngology-head and neck surgical (OHNS) trainees' operating exposure is supplemented by a combination of didactic teaching, textbook reading, and cadaveric dissections. Conventional teaching, however, may not adequately equip trainees with an understanding of complex visuospatial relationships of the middle ear. Both face and content validation were assessed of a novel three-dimensional (3D) photorealistic virtual ear simulation tool underwent face and content validation as an educational tool for OHNS trainees.

METHODS

A three-dimensional mesh reconstruction of open access imaging was generated using geometric modeling, which underwent global illumination, subsurface scattering, and texturing to create photorealistic virtual reality (VR) ear models were created from open access imaging and comiled into a educational platform. This was compiled into an educational VR platform which was explored to validate the face and content validity questionnaires in a prospective manner. OHNS post-graduate trainees were recruited from University of Toronto and University of Calgary OHNS programs. Participation was on a voluntary basis.

RESULTS

Total of 23 OHNS post-graduate trainees from the two universities were included in this study. The mean comfort level of otologic anatomy was rated 4.8 (± 2.2) out of 10. Senior residents possessed more otologic surgical experience (P < 0.001) and higher average comfort when compared to junior residents [6.7 (± 0.7) vs. 3.6 (± 1.9); P = 0.001]. Face and content validities were achieved in all respective domains with no significant difference between the two groups. Overall, respondents believed OtoVIS was a useful tool to learn otologic anatomy with a median score of 10.0 (8.3-10.0) and strongly agreed that OtoVIS should be added to OHNS training with a score of 10.0 (9.3-10.0).

CONCLUSIONS

OtoVIS achieved both face and content validity as a photorealistic VR otologic simulator for teaching otologic anatomy in the postgraduate setting. As an immersive learning tool, it may supplement trainees' understanding and residents endorsed its use.

Anatomical Variations of the Human Cochlea Using an Image Analysis Tool

Understanding cochlear anatomy is crucial for developing less traumatic electrode arrays and insertion guidance for cochlear implantation. The human cochlea shows considerable variability in size and morphology. This study analyses 1000+ clinical temporal bone CT images using a web-based image analysis tool. Cochlear size and shape parameters were obtained to determine population statistics and perform regression and correlation analysis. The analysis revealed that cochlear morphology follows Gaussian distribution, while cochlear dimensions A and B are not well-correlated to each other. Additionally, dimension B is more correlated to duct lengths, the wrapping factor and volume than dimension A. The scala tympani size varies considerably among the population, with the size generally decreasing along insertion depth with dimensional jumps through the trajectory. The mean scala tympani radius was 0.32 mm near the 720° insertion angle. Inter-individual variability was four times that of intra-individual variation. On average, the dimensions of both ears are similar. However, statistically significant differences in clinical dimensions were observed between ears of the same patient, suggesting that size and shape are not the same. Harnessing deep learning-based, automated image analysis tools, our results yielded important insights into cochlear morphology and implant development, helping to reduce insertion trauma and preserving residual hearing.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Snake Robots for Surgical Applications: A Review

Although substantial advancements have been achieved in robot-assisted surgery, the blueprint to existing snake robotics predominantly focuses on the preliminary structural design, control, and human–robot interfaces, with features which have not been particularly explored in the literature. This paper aims to conduct a review of planning and operation concepts of hyper-redundant serpentine robots for surgical use, as well as any future challenges and solutions for better manipulation. Current researchers in the field of the manufacture and navigation of snake robots have faced issues, such as a low dexterity of the end-effectors around delicate organs, state estimation and the lack of depth perception on two-dimensional screens. A wide range of robots have been analysed, such as the i²Snake robot, inspiring the use of force and position feedback, visual servoing and augmented reality (AR). We present the types of actuation methods, robot kinematics, dynamics, sensing, and prospects of AR integration in snake robots, whilst addressing their shortcomings to facilitate the surgeon’s task. For a smoother gait control, validation and optimization algorithms such as deep learning databases are examined to mitigate redundancy in module linkage backlash and accidental self-collision. In essence, we aim to provide an outlook on robot configurations during motion by enhancing their material compositions within anatomical biocompatibility standards.

Vision-Based Augmented Reality System for Middle Ear Surgery: Evaluation in Operating Room Environment

HYPOTHESIS

Augmented reality (AR) solely based on image features is achievable in operating room conditions and its precision is compatible with otological surgery.

BACKGROUND

The objective of this work was to evaluate the performance of a vision-based AR system for middle ear surgery in the operating room conditions.

METHODS

Nine adult patients undergoing ossicular procedures were included in this prospective study. AR was obtained by combining real-time video from the operating microscope with the virtual image obtained from the preoperative computed tomography (CT)-scan. Initial registration between the video and the virtual CT image was achieved using manual selection of six points on the tympanic sulcus. Patient-microscope movements during the procedure were tracked using image-feature matching algorithm. The microscope was randomly moved at an approximated speed of 5 mm/s in the three axes of space and rotation for 180 seconds. The accuracy of the system was assessed by calculating the distance between each fiducial point selected on the video image and its corresponding point on the scanner.

RESULTS

AR could be obtained for at least 3 minutes in seven out of nine patients. The overlay fiducial and target registration errors were 0.38 ± 0.23 mm (n = 7) and 0.36 ± 0.15 mm (n = 5) respectively, with a drift error of 1.2 ± 0.5 μm/s. The system was stable throughout the procedure and achieved a refresh rate of 12 fps. Moderate bleeding and introduction of surgical instruments did not compromise the performance of the system.

CONCLUSION

The AR system yielded sub-millimetric accuracy and remained stable throughout the experimental study despite patient-microscope movements and field of view obtrusions.

Robotics, automation, active electrode arrays, and new devices for cochlear implantation: A contemporary review

In the last two decades, cochlear implant surgery has evolved into a minimally invasive, hearing preservation surgical technique. The devices used during surgery have benefited from technological advances that have allowed modification and possible improvement of the surgical technique. Robotics has recently gained popularity in otology as an effective tool to overcome the surgeon's limitations such as tremor, drift and accurate force control feedback in laboratory testing. Cochlear implantation benefits from robotic assistance in several steps during the surgical procedure: (i) during the approach to the middle ear by automated mastoidectomy and posterior tympanotomy or through a tunnel from the postauricular skin to the middle ear (i.e. direct cochlear access); (ii) a minimally invasive cochleostomy by a robot-assisted drilling tool; (iii) alignment of the correct insertion axis on the basal cochlear turn; (iv) insertion of the electrode array with a motorized insertion tool. In recent years, the development of bone-attached parallel robots and image-guided surgical robotic systems has allowed the first successful cochlear implantation procedures in patients via a single hole drilled tunnel. Several other robotic systems, new materials, sensing technologies applied to the electrodes, and smart devices have been developed, tested in experimental models and finally some have been used in patients with the aim of reducing trauma in cochleostomy, and permitting slow and more accurate insertion of the electrodes. Despite the promising results in laboratory tests in terms of minimal invasiveness, reduced trauma and better hearing preservation, so far, no clinical benefits on residual hearing preservation or better speech performance have been demonstrated. Before these devices can become the standard approach for cochlear implantation, several points still need to be addressed, primarily cost and duration of the procedure. One can hope that improvement in the cost/benefit ratio will expand the technology to every cochlear implantation procedure. Laboratory research and clinical studies on patients should continue with the aim of making intracochlear implant insertion an atraumatic and reversible gesture for total preservation of the inner ear structure and physiology.

Augmented Reality Based Transmodiolar Cochlear Implantation

HYPOTHESIS

Transmodiolar auditory implantation via the middle ear cavity could be possible using an augmented reality system (ARS).

BACKGROUND

There is no clear landmark to indicate the cochlear apex or the modiolar axis. The ARS seems to be a promising tool for transmodiolar implantation by combining information from the preprocedure computed tomography scan (CT-scan) images to the real-time video of the surgical field.

METHODS

Eight human temporal bone resin models were included (five adults and three children). The procedure started by the identification of the modiolar axis on the preprocedure CT-scan followed by a 3D reconstruction of the images. Information on modiolar location and navigational guidance was supplemented to the reconstructed model, which was then registered with the surgical video using a point-based approach. Relative movements between the phantom and the microscope were tracked using image feature-based motion tracking. Based on the information provided via the ARS, the surgeon implanted the electrode-array inside the modiolus after drilling the helicothrema. Postprocedure CT-scan images were acquired to evaluate the registration error and the implantation accuracy.

RESULTS

The implantation could be conducted in all cases with a 2D registration error of 0.4 ± 0.24 mm. The mean entry point error was 0.6 ± 1.00 mm and the implant angular error 13.5 ± 8.93 degrees (n = 8), compatible with the procedure requirements.

CONCLUSION

We developed an image-based ARS to identify the extremities and the axis of the cochlear modiolus on intraprocedure videos. The system yielded submillimetric accuracy for implantation and remained stable throughout the experimental study.

Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos

PURPOSE

Visualization of the cochlea is impossible due to the delicate and intricate ear anatomy. Augmented reality may be used to perform auditory nerve implantation by transmodiolar approach in patients with profound hearing loss.

METHODS

We present an augmented reality system for the visualization of the cochlear axis in surgical videos. The system starts with an automatic anatomical landmark detection in preoperative computed tomography images based on deep reinforcement learning. These landmarks are used to register the preoperative geometry with the real-time microscopic video captured inside the auditory canal. Three-dimensional pose of the cochlear axis is determined using the registration projection matrices. In addition, the patient microscope movements are tracked using an image feature-based tracking process.

RESULTS

The landmark detection stage yielded an average localization error of [Formula: see text] mm ([Formula: see text]). The target registration error was [Formula: see text] mm for the cochlear apex and [Formula: see text] for the cochlear axis.

CONCLUSION

We developed an augmented reality system to visualize the cochlear axis in intraoperative videos. The system yielded millimetric accuracy and remained stable throughout the experimental study despite camera movements throughout the procedure in experimental conditions.