Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan

Calibration-free structured-light-based 3D scanning system in laparoscope for robotic surgery

Accurate 3D shape measurement is crucial for surgical support and alignment in robotic surgery systems. Stereo cameras in laparoscopes offer a potential solution; however, their accuracy in stereo image matching diminishes when the target image has few textures. Although stereo matching with deep learning has gained significant attention, supervised learning requires a large dataset of images with depth annotations, which are scarce for laparoscopes. Thus, there is a strong demand to explore alternative methods for depth reconstruction or annotation for laparoscopes. Active stereo techniques are a promising approach for achieving 3D reconstruction without textures. In this study, a 3D shape reconstruction method is proposed using an ultra-small patterned projector attached to a laparoscopic arm to address these issues. The pattern projector emits a structured light with a grid-like pattern that features node-wise modulation for positional encoding. To scan the target object, multiple images are taken while the projector is in motion, and the relative poses of the projector and a camera are auto-calibrated using a differential rendering technique. In the experiment, the proposed method is evaluated by performing 3D reconstruction using images obtained from a surgical robot and comparing the results with a ground-truth shape obtained from X-ray CT.

Deep focus light-field camera for handheld 3D intraoral scanning using crosstalk-free solid immersion microlens arrays

3D in vivo imaging techniques facilitate disease tracking and treatment, but bulky configurations and motion artifacts limit practical clinical applications. Compact light-field cameras with microlens arrays offer a feasible option for rapid volumetric imaging, yet their utilization in clinical practice necessitates an increased depth-of-field for handheld operation. Here, we report deep focus light-field camera (DF-LFC) with crosstalk-free solid immersion microlens arrays (siMLAs), allowing large depth-of-field and high-resolution imaging for handheld 3D intraoral scanning. The siMLAs consist of thin PDMS-coated microlens arrays and a metal-insulator-metal absorber to extend the focal length with low optical crosstalk and specular reflection. The experimental results show that the immersion of MLAs in PDMS increases the focal length by a factor of 2.7 and the transmittance by 5.6%-27%. Unlike conventional MLAs, the siMLAs exhibit exceptionally high f-numbers up to f/6, resulting in a large depth-of-field for light-field imaging. The siMLAs were fully integrated into an intraoral scanner to reconstruct a 3D dental phantom with a distance measurement error of 82 ± 41 μm during handheld operation. The DF-LFC offers a new direction not only for digital dental impressions with high accuracy, simplified workflow, reduced waste, and digital compatibility but also for assorted clinical endoscopy and microscopy.

Single and multi-frame auto-calibration for 3D endoscopy with differential rendering

The use of 3D measurement in endoscopic images offers practicality in cancer diagnosis, computer-assisted interventions, and making annotations for machine learning training data. An effective approach is the implementation of an active stereo system, using a micro-sized pattern projector and an endoscope camera, which has been intensively developed. One open problem for such a system is the necessity of strict and complex calibration of the projector-camera system to precisely recover the shapes. Moreover, since the head of an endoscope should have enough elasticity to avoid harming target objects, the positions of the pattern projector cannot be tightly fixed to the head, resulting in limited accuracy. A straightforward approach to the problem is applying auto-calibration. However, it requires special markers in the pattern or a highly accurate initial position for stable calibration, which is impractical for real operation. In the paper, we propose a novel auto-calibration method based on differential rendering techniques, which are recently proposed and drawing wide attention. To apply the method to an endoscopic system, where a diffractive optical element (DOE) is used, we propose a technique to simultaneously estimate the focal length of the DOE as well as the extrinsic parameters between a projector and a camera. We also propose a multi-frame optimization algorithm to jointly optimize the intrinsic and extrinsic parameters, relative pose between frames, and the entire shape.Clinical relevance- One-shot endoscopic measurement of depth information is a practical solution for cancer diagnosis, computer-assisted interventions, and making annotations for machine learning training data.

The role of computer-assisted systems for upper-endoscopy quality monitoring and assessment of gastric lesions

This article analyses the literature regarding the value of computer-assisted systems in esogastroduodenoscopy-quality monitoring and the assessment of gastric lesions. Current data show promising results in upper-endoscopy quality control and a satisfactory detection accuracy of gastric premalignant and malignant lesions, similar or even exceeding that of experienced endoscopists. Moreover, artificial systems enable the decision for the best treatment strategies in gastric-cancer patient care, namely endoscopic vs surgical resection according to tumor depth. In so doing, unnecessary surgical interventions would be avoided whilst providing a better quality of life and prognosis for these patients. All these performance data have been revealed by numerous studies using different artificial intelligence (AI) algorithms in addition to white-light endoscopy or novel endoscopic techniques that are available in expert endoscopy centers. It is expected that ongoing clinical trials involving AI and the embedding of computer-assisted diagnosis systems into endoscopic devices will enable real-life implementation of AI endoscopic systems in the near future and at the same time will help to overcome the current limits of the computer-assisted systems leading to an improvement in performance. These benefits should lead to better diagnostic and treatment strategies for gastric-cancer patients. Furthermore, the incorporation of AI algorithms in endoscopic tools along with the development of large electronic databases containing endoscopic images might help in upper-endoscopy assistance and could be used for telemedicine purposes and second opinion for difficult cases.

Fully Auto-calibrated Active-stereo-based 3D Endoscopic System using Correspondence Estimation with Graph Convolutional Network

We have developed a series of 3D endoscopic systems where a micro-sized pattern projector is inserted through the instrument channel of the endoscope and shapes are reconstructed by a structured light technique using captured images of the endoscopic camera. One problem of the previous works is that the accuracy of shape reconstruction is low, because the projector cannot be fixed to the endoscope, and thus, the pose of the pattern projector w.r.t. the camera cannot be pre-calibrated. In this paper, we propose a method to auto-calibrate the pose of the projector without using any special devices nor manual process. Since the technique is one-shot, multiple shapes can be reconstructed from an image sequence and a large 3D scene can be recovered by merging them. Experiments are conducted using the real system.