Deep learning with cinematic rendering: fine-tuning deep neural networks using photorealistic medical images

Automatic Detection of Nuclear Spins at Arbitrary Magnetic Fields via Signal-to-Image AI Model

Quantum sensors leverage matter's quantum properties to enable measurements with unprecedented spatial and spectral resolution. Among these sensors, those utilizing nitrogen-vacancy (NV) centers in diamond offer the distinct advantage of operating at room temperature. Nevertheless, signals received from NV centers are often complex, making interpretation challenging. This is especially relevant in low magnetic field scenarios, where standard approximations for modeling the system fail. Additionally, NV signals feature a prominent noise component. In this Letter, we present a signal-to-image deep learning model capable of automatically inferring the number of nuclear spins surrounding a NV sensor and the hyperfine couplings between the sensor and the nuclear spins. Our model is trained to operate effectively across various magnetic field scenarios, requires no prior knowledge of the involved nuclei, and is designed to handle noisy signals, leading to fast characterization of nuclear environments in real experimental conditions. With detailed numerical simulations, we test the performance of our model in scenarios involving varying numbers of nuclei, achieving an average error of less than 2 kHz in the estimated hyperfine constants.

Monocular endoscopy images depth estimation with multi-scale residual fusion

BACKGROUND

Monocular depth estimation plays a fundamental role in clinical endoscopy surgery. However, the coherent illumination, smooth surfaces, and texture-less nature of endoscopy images present significant challenges to traditional depth estimation methods. Existing approaches struggle to accurately perceive depth in such settings.

METHOD

To overcome these challenges, this paper proposes a novel multi-scale residual fusion method for estimating the depth of monocular endoscopy images. Specifically, we address the issue of coherent illumination by leveraging image frequency domain component space transformation, thereby enhancing the stability of the scene's light source. Moreover, we employ an image radiation intensity attenuation model to estimate the initial depth map. Finally, to refine the accuracy of depth estimation, we utilize a multi-scale residual fusion optimization technique.

RESULTS

To evaluate the performance of our proposed method, extensive experiments were conducted on public datasets. The structural similarity measures for continuous frames in three distinct clinical data scenes reached impressive values of 0.94, 0.82, and 0.84, respectively. These results demonstrate the effectiveness of our approach in capturing the intricate details of endoscopy images. Furthermore, the depth estimation accuracy achieved remarkable levels of 89.3 % and 91.2 % for the two models' data, respectively, underscoring the robustness of our method.

CONCLUSIONS

Overall, the promising results obtained on public datasets highlight the significant potential of our method for clinical applications, facilitating reliable depth estimation and enhancing the quality of endoscopy surgical procedures.

Colonoscopy 3D video dataset with paired depth from 2D-3D registration

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD.

Multi-scale, multi-dimensional binocular endoscopic image depth estimation network

During invasive surgery, the use of deep learning techniques to acquire depth information from lesion sites in real-time is hindered by the lack of endoscopic environmental datasets. This work aims to develop a high-accuracy three-dimensional (3D) simulation model for generating image datasets and acquiring depth information in real-time. Here, we proposed an end-to-end multi-scale supervisory depth estimation network (MMDENet) model for the depth estimation of pairs of binocular images. The proposed MMDENet highlights a multi-scale feature extraction module incorporating contextual information to enhance the correspondence precision of poorly exposed regions. A multi-dimensional information-guidance refinement module is also proposed to refine the initial coarse disparity map. Statistical experimentation demonstrated a 3.14% reduction in endpoint error compared to state-of-the-art methods. With a processing time of approximately 30fps, satisfying the requirements of real-time operation applications. In order to validate the performance of the trained MMDENet in actual endoscopic images, we conduct both qualitative and quantitative analysis with 93.38% high precision, which holds great promise for applications in surgical navigation.

A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning

This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.

Implementation of cinematic rendering of gastric masses into clinical practice: a pictorial review

High-resolution CT is the gold standard diagnostic imaging study for staging of gastric malignancies. Cinematic rendering creates a photorealistic evaluation using the standard high-resolution CT volumetric data set. This novel display method offers unique possibilities for the evaluation of gastric masses. Here we present further observations of the role of cinematic rendering in the evaluation of gastric masses at a large tertiary care center. We offer three valuable teaching points for the application of the cinematic rendering for gastric masses with several case examples for each teaching point, discuss potential limitations of cinematic rendering, and review future directions for cinematic rendering in this setting.

Robust weakly supervised learning for COVID-19 recognition using multi-center CT images

The world is currently experiencing an ongoing pandemic of an infectious disease named coronavirus disease 2019 (i.e., COVID-19), which is caused by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Computed Tomography (CT) plays an important role in assessing the severity of the infection and can also be used to identify those symptomatic and asymptomatic COVID-19 carriers. With a surge of the cumulative number of COVID-19 patients, radiologists are increasingly stressed to examine the CT scans manually. Therefore, an automated 3D CT scan recognition tool is highly in demand since the manual analysis is time-consuming for radiologists and their fatigue can cause possible misjudgment. However, due to various technical specifications of CT scanners located in different hospitals, the appearance of CT images can be significantly different leading to the failure of many automated image recognition approaches. The multi-domain shift problem for the multi-center and multi-scanner studies is therefore nontrivial that is also crucial for a dependable recognition and critical for reproducible and objective diagnosis and prognosis. In this paper, we proposed a COVID-19 CT scan recognition model namely coronavirus information fusion and diagnosis network (CIFD-Net) that can efficiently handle the multi-domain shift problem via a new robust weakly supervised learning paradigm. Our model can resolve the problem of different appearance in CT scan images reliably and efficiently while attaining higher accuracy compared to other state-of-the-art methods.

Image-based shading correction for narrow-FOV truncated pelvic CBCT with deep convolutional neural networks and transfer learning

Purpose: Cone beam computed tomography (CBCT) is a standard solution for in‐room image guidance for radiation therapy. It is used to evaluate and compensate for anatomopathological changes between the dose delivery plan and the fraction delivery day. CBCT is a fast and versatile solution, but it suffers from drawbacks like low contrast and requires proper calibration to derive density values. Although these limitations are even more prominent with in‐room customized CBCT systems, strategies based on deep learning have shown potential in improving image quality. As such, this article presents a method based on a convolutional neural network and a novel two‐step supervised training based on the transfer learning paradigm for shading correction in CBCT volumes with narrow field of view (FOV) acquired with an ad hoc in‐room system.

Methods: We designed a U‐Net convolutional neural network, trained on axial slices of corresponding CT/CBCT couples. To improve the generalization capability of the network, we exploited two‐stage learning using two distinct data sets. At first, the network weights were trained using synthetic CBCT scans generated from a public data set, and then only the deepest layers of the network were trained again with real‐world clinical data to fine‐tune the weights. Synthetic data were generated according to real data acquisition parameters. The network takes a single grayscale volume as input and outputs the same volume with corrected shading and improved HU values.

Results: Evaluation was carried out with a leave‐one‐out cross‐validation, computed on 18 unique CT/CBCT pairs from six different patients from a real‐world dataset. Comparing original CBCT to CT and improved CBCT to CT, we obtained an average improvement of 6 dB on peak signal‐to‐noise ratio (PSNR), +2% on structural similarity index measure (SSIM). The median interquartile range (IQR) Hounsfield unit (HU) difference between CBCT and CT improved from 161.37 (162.54) HU to 49.41 (66.70) HU. Region of interest (ROI)‐based HU difference was narrowed by 75% in the spongy bone (femoral head), 89% in the bladder, 85% for fat, and 83% for muscle. The improvement in contrast‐to‐noise ratio for these ROIs was about 67%.

Conclusions: We demonstrated that shading correction obtaining CT‐compatible data from narrow‐FOV CBCTs acquired with a customized in‐room system is possible. Moreover, the transfer learning approach proved particularly beneficial for such a shading correction approach.

DeepWL: Robust EPID based Winston-Lutz analysis using deep learning, synthetic image generation and optical path-tracing

Radiation therapy requires clinical linear accelerators to be mechanically and dosimetrically calibrated to a high standard. One important quality assurance test is the Winston-Lutz test which localises the radiation isocentre of the linac. In the current work we demonstrate a novel method of analysing EPID based Winston-Lutz QA images using a deep learning model trained only on synthetic image data. In addition, we propose a novel method of generating the synthetic WL images and associated 'ground-truth' masks using an optical path-tracing engine to 'fake' mega-voltage EPID images. The model called DeepWL was trained on 1500 synthetic WL images using data augmentation techniques for 180 epochs. The model was built using Keras with a TensorFlow backend on an Intel Core i5-6500T CPU and trained in approximately 15 h. DeepWL was shown to produce ball bearing and multi-leaf collimator field segmentations with a mean dice coefficient of 0.964 and 0.994 respectively on previously unseen synthetic testing data. When DeepWL was applied to WL data measured on an EPID, the predicted mean displacements were shown to be statistically similar to the Canny Edge detection method. However, the DeepWL predictions for the ball bearing locations were shown to correlate better with manual annotations compared with the Canny edge detection algorithm. DeepWL was demonstrated to analyse Winston-Lutz images with an accuracy suitable for routine linac quality assurance with some statistical evidence that it may outperform Canny Edge detection methods in terms of segmentation robustness and the resultant displacement predictions.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Synthetic data in machine learning for medicine and healthcare

The proliferation of synthetic data in artificial intelligence for medicine and healthcare raises concerns about the vulnerabilities of the software and the challenges of current policy.

Cone-beam computed tomography cinematic rendering: clinical, teaching and research applications

Cone-beam computed tomography (CBCT) is an essential imaging method that increases the accuracy of diagnoses, planning and follow-up of endodontic complex cases. Image postprocessing and subsequent visualization relies on software for three-dimensional navigation, and application of indexation tools to provide clinically useful information according to a set of volumetric data. Image postprocessing has a crucial impact on diagnostic quality and various techniques have been employed on computed tomography (CT) and magnetic resonance imaging (MRI) data sets. These include: multiplanar reformations (MPR), maximum intensity projection (MIP) and volume rendering (VR). A recent advance in 3D data visualization is the new cinematic rendering reconstruction method, a technique that generates photorealistic 3D images from conventional CT and MRI data. This review discusses the importance of CBCT cinematic rendering for clinical decision-making, teaching, and research in Endodontics, and a presents series of cases that illustrate the diagnostic value of 3D cinematic rendering in clinical care.

Target-Independent Domain Adaptation for WBC Classification Using Generative Latent Search

Automating the classification of camera-obtained microscopic images of White Blood Cells (WBCs) and related cell subtypes has assumed importance since it aids the laborious manual process of review and diagnosis. Several State-Of-The-Art (SOTA) methods developed using Deep Convolutional Neural Networks suffer from the problem of domain shift - severe performance degradation when they are tested on data (target) obtained in a setting different from that of the training (source). The change in the target data might be caused by factors such as differences in camera/microscope types, lenses, lighting-conditions etc. This problem can potentially be solved using Unsupervised Domain Adaptation (UDA) techniques albeit standard algorithms presuppose the existence of a sufficient amount of unlabelled target data which is not always the case with medical images. In this paper, we propose a method for UDA that is devoid of the need for target data. Given a test image from the target data, we obtain its 'closest-clone' from the source data that is used as a proxy in the classifier. We prove the existence of such a clone given that infinite number of data points can be sampled from the source distribution. We propose a method in which a latent-variable generative model based on variational inference is used to simultaneously sample and find the 'closest-clone' from the source distribution through an optimization procedure in the latent space. We demonstrate the efficacy of the proposed method over several SOTA UDA methods for WBC classification on datasets captured using different imaging modalities under multiple settings.

Deep Adversarial Training for Multi-Organ Nuclei Segmentation in Histopathology Images

Nuclei mymargin segmentation is a fundamental task for various computational pathology applications including nuclei morphology analysis, cell type classification, and cancer grading. Deep learning has emerged as a powerful approach to segmenting nuclei but the accuracy of convolutional neural networks (CNNs) depends on the volume and the quality of labeled histopathology data for training. In particular, conventional CNN-based approaches lack structured prediction capabilities, which are required to distinguish overlapping and clumped nuclei. Here, we present an approach to nuclei segmentation that overcomes these challenges by utilizing a conditional generative adversarial network (cGAN) trained with synthetic and real data. We generate a large dataset of H&E training images with perfect nuclei segmentation labels using an unpaired GAN framework. This synthetic data along with real histopathology data from six different organs are used to train a conditional GAN with spectral normalization and gradient penalty for nuclei segmentation. This adversarial regression framework enforces higher-order spacial-consistency when compared to conventional CNN models. We demonstrate that this nuclei segmentation approach generalizes across different organs, sites, patients and disease states, and outperforms conventional approaches, especially in isolating individual and overlapping nuclei.

Web-based fully automated cephalometric analysis by deep learning

BACKGROUND AND OBJECTIVE

An accurate lateral cephalometric analysis is vital in orthodontic diagnosis. Identification of anatomic landmarks on lateral cephalograms is tedious, and errors may occur depending on the doctor's experience. Several attempts have been made to reduce this time-consuming process by automating the process through machine learning; however, they only dealt with a small amount of data from one institute. This study aims to develop a fully automated cephalometric analysis method using deep learning and a corresponding web-based application that can be used without high-specification hardware.

METHODS

We built our own dataset comprising 2,075 lateral cephalograms and ground truth positions of 23 landmarks from two institutes and trained a two-stage automated algorithm with a stacked hourglass deep learning model specialized for detecting landmarks in images. Additionally, a web-based application with the proposed algorithm for fully automated cephalometric analysis was developed for better accessibility regardless of the user's computer hardware, which is essential for a deep learning-based method.

RESULTS

The algorithm was evaluated with datasets from various devices and institutes, including a widely used open dataset and achieved 1.37 ± 1.79 mm of point-to-point errors with ground truth positions for 23 cephalometric landmarks. Based on the predicted positions, anatomical types of the subjects were automatically classified and compared with the ground truth, and the automated algorithm achieved a successful classification rate of 88.43%.

CONCLUSIONS

We expect that this fully automated cephalometric analysis algorithm and the web-based application can be widely used in various medical environments to save time and effort for manual marking and diagnosis.

Evaluation of the urinary bladder using three-dimensional CT cinematic rendering

Three-dimensional (3D) visualizations of volumetric data from computed tomography (CT) acquisitions can be important adjuncts to interpretation of two-dimensional (2D) reconstructions. Recently, the 3D technique known as cinematic rendering (CR) was introduced, allowing photorealistic images to be created from standard CT acquisitions. CR methodology is under increasing investigation for use in the display of regions of complex anatomy and as a tool for education and preoperative planning. In this article, we will illustrate the potential utility of CR for evaluating the urinary bladder and associated pathology. The urinary bladder is susceptible to a multitude of neoplastic and inflammatory conditions and their sequelae. The intrinsic properties of CR may prove useful for the display of subtle mucosal/luminal irregularities, the simultaneous display of soft tissue detail with high-resolution maps of associated tumor neovasculature, and the improved display of spatial relationships to aid pre-procedural planning. Further refinement of presets for CR image creation and prospective evaluation of urinary bladder CR in real-world settings will be important for widespread clinical adoption.

Cardiac Chamber Segmentation Using Deep Learning on Magnetic Resonance Images from Patients Before and After Atrial Septal Occlusion Surgery

We propose a robust technique for segmenting magnetic resonance images of post-atrial septal occlusion intervention in the cardiac chamber. The technique can be used to determine the surgical outcomes of atrial septal defects before and after implantation of a septal occluder, which intends to provide volume restoration of the right and left atria. A variant of the U-Net architecture is used to perform atrial segmentation via a deep convolutional neural network. The method was evaluated on a dataset containing 550 two-dimensional image slices, outperforming conventional active contouring regarding the Dice similarity coefficient, Jaccard index, and Hausdorff distance, and achieving segmentation in the presence of ghost artifacts that occlude the atrium outline. Moreover, the proposed technique is closer to manual segmentation than the snakes active contour model. After segmentation, we computed the volume ratio of right to left atria, obtaining a smaller ratio that indicates better restoration. Hence, the proposed technique allows to evaluate the surgical success of atrial septal occlusion and may support diagnosis regarding the accurate evaluation of atrial septal defects before and after occlusion procedures.

GANPOP: Generative Adversarial Network Prediction of Optical Properties From Single Snapshot Wide-Field Images

We present a deep learning framework for wide-field, content-aware estimation of absorption and scattering coefficients of tissues, called Generative Adversarial Network Prediction of Optical Properties (GANPOP). Spatial frequency domain imaging is used to obtain ground-truth optical properties at 660 nm from in vivo human hands and feet, freshly resected human esophagectomy samples, and homogeneous tissue phantoms. Images of objects with either flat-field or structured illumination are paired with registered optical property maps and are used to train conditional generative adversarial networks that estimate optical properties from a single input image. We benchmark this approach by comparing GANPOP to a single-snapshot optical property (SSOP) technique, using a normalized mean absolute error (NMAE) metric. In human gastrointestinal specimens, GANPOP with a single structured-light input image estimates the reduced scattering and absorption coefficients with 60% higher accuracy than SSOP while GANPOP with a single flat-field illumination image achieves similar accuracy to SSOP. When applied to both in vivo and ex vivo swine tissues, a GANPOP model trained solely on structured-illumination images of human specimens and phantoms estimates optical properties with approximately 46% improvement over SSOP, indicating adaptability to new, unseen tissue types. Given a training set that appropriately spans the target domain, GANPOP has the potential to enable rapid and accurate wide-field measurements of optical properties.

Initial experience with 3D CT cinematic rendering of acute pancreatitis and associated complications

Inflammation of the pancreas can present with a wide range of imaging findings from mild enlargement of the gland and surrounding infiltrative fat stranding through extensive glandular necrosis. Complications of pancreatitis are varied and include infected fluid collections, pseudocysts, and vascular findings such as pseudoaneurysms and thromboses. Cross-sectional imaging with computed tomography (CT) is one of the mainstays of evaluating patients with pancreatitis. New methods that allow novel visualization volumetric CT data may improve diagnostic yield for the detection of findings that provide prognostic information in pancreatitis patients or can drive new avenues of research such as machine learning. Cinematic rendering (CR) is a photorealistic visualization method for volumetric imaging data that are being investigated for a variety of potential applications including the life-like display of complex anatomy and visual characterization of mass lesions. In this review, we describe the CR appearance of different types of pancreatitis and complications of pancreatitis. We also note possible future directions for research into the utility of CR for pancreatitis.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

DeepLSR: a deep learning approach for laser speckle reduction

Speckle artifacts degrade image quality in virtually all modalities that utilize coherent energy, including optical coherence tomography, reflectance confocal microscopy, ultrasound, and widefield imaging with laser illumination. We present an adversarial deep learning framework for laser speckle reduction, called DeepLSR (https://durr.jhu.edu/DeepLSR), that transforms images from a source domain of coherent illumination to a target domain of speckle-free, incoherent illumination. We apply this method to widefield images of objects and tissues illuminated with a multi-wavelength laser, using light emitting diode-illuminated images as ground truth. In images of gastrointestinal tissues, DeepLSR reduces laser speckle noise by 6.4 dB, compared to a 2.9 dB reduction from optimized non-local means processing, a 3.0 dB reduction from BM3D, and a 3.7 dB reduction from an optical speckle reducer utilizing an oscillating diffuser. Further, DeepLSR can be combined with optical speckle reduction to reduce speckle noise by 9.4 dB. This dramatic reduction in speckle noise may enable the use of coherent light sources in applications that require small illumination sources and high-quality imaging, including medical endoscopy.

3D CT cinematic rendering of the spleen: Potential role in problem solving

Cinematic rendering (CR) is a new 3D visualization methodology for volumetric diagnostic imaging including computed tomography (CT) datasets composed of isotropic voxels. CR produces photorealistic images with enhanced detail relative to other 3D visualization methods and realistic shadowing. In this review, we provide a number of examples of splenic pathology visualized with CR including conditions affecting the splenic vasculature, neoplasms, and accessory spleens. These examples are compared to 2D CT and traditional 3D CT techniques and the potential advantages of CR are highlighted. CR displays textural changes in the splenic parenchyma to particular advantage, and a portion of this review will be devoted to examples of how textural features can help distinguish intrapancreatic accessory spleens from neuroendocrine tumors.