Conversion of single-energy CT to parametric maps of dual-energy CT using convolutional neural network

OBJECTIVES

We propose a deep learning (DL) multitask learning framework using convolutional neural network for a direct conversion of single-energy CT (SECT) to 3 different parametric maps of dual-energy CT (DECT): virtual-monochromatic image (VMI), effective atomic number (EAN), and relative electron density (RED).

METHODS

We propose VMI-Net for conversion of SECT to 70, 120, and 200 keV VMIs. In addition, EAN-Net and RED-Net were also developed to convert SECT to EAN and RED. We trained and validated our model using 67 patients collected between 2019 and 2020. Single-layer CT images with 120 kVp acquired by the DECT (IQon spectral CT; Philips Healthcare, Amsterdam, Netherlands) were used as input, while the VMIs, EAN, and RED acquired by the same device were used as target. The performance of the DL framework was evaluated by absolute difference (AD) and relative difference (RD).

RESULTS

The VMI-Net converted 120 kVp SECT to the VMIs with AD of 9.02 Hounsfield Unit, and RD of 0.41% compared to the ground truth VMIs. The ADs of the converted EAN and RED were 0.29 and 0.96, respectively, while the RDs were 1.99% and 0.50% for the converted EAN and RED, respectively.

CONCLUSIONS

SECT images were directly converted to the 3 parametric maps of DECT (ie, VMIs, EAN, and RED). By using this model, one can generate the parametric information from SECT images without DECT device. Our model can help investigate the parametric information from SECT retrospectively.

ADVANCES IN KNOWLEDGE

DL framework enables converting SECT to various high-quality parametric maps of DECT.

The clinical impact of macrophage polarity after Kasai portoenterostomy in biliary atresia

Introduction

Biliary atresia (BA) is a cholestatic hepatopathy caused by fibrosing destruction of intrahepatic and extrahepatic bile ducts, and its etiology has not been clearly revealed. In BA, liver fibrosis progression is often observed even after Kasai portoenterostomy (KPE), and more than half of cases require liver transplantation in their lifetime in Japan. Macrophages play an important role in liver fibrosis progression and are classically divided into proinflammatory (M1) and fibrotic macrophages (M2), whose phenotypic transformation is called "macrophage polarity." The polarity has been reported to reflect the tissue microenvironment. In this study, we examined the relationship between macrophage polarity and the post-KPE clinical course.

Materials and methods

Thirty BA patients who underwent KPE in our institution from 2000 to 2020 were recruited. Multiple immunostainings for CD68, CD163, CK19, and α-SMA were carried out on liver biopsy specimens obtained at KPE. ROC curves were calculated based on each clinical event, and the correlation with the clinical data was analyzed.

Results and discussion

The M2 ratio, defined as the proportion of M2 macrophages (CD163-positive cells), was correlated inversely with the occurrence of postoperative cholangitis (AUC: 0.7602). The patients were classified into M2 high (n = 19) and non-high (n = 11) groups based on an M2 ratio value obtained from the Youden index ( = 0.918). As a result, pathological evaluations (Metavir score, αSMA area fraction, and CK19 area fraction) were not significantly different between these groups. In mild liver fibrosis cases (Metavir score = 0-2), the M2 non-high group had a significantly lower native liver survival rate than the high group (p = 0.02). Moreover, 4 out of 8 cases in the M2 non-high group underwent early liver transplantation within 2 years after KPE.

Conclusions

Non-M2 macrophages, including M1 macrophages, may be correlated with postoperative cholangitis, and the M2 non-high group in mild liver fibrosis cases had a significantly lower native liver survival rate than the high group, requiring early liver transplantation in this study. Preventing advanced liver fibrosis is a key factor in improving native liver survival for BA patients, and liver macrophages may play important roles in liver homeostasis and the promotion of inflammation and fibrosis.

Recent Advances in Infrared Face Analysis and Recognition with Deep Learning

Besides the many advances made in the facial detection and recognition fields, face recognition applied to visual images (VIS-FR) has received increasing interest in recent years, especially in the field of communication, identity authentication, public safety and to address the risk of terrorism and crime. These systems however encounter important problems in the presence of variations in pose, expression, age, occlusion, disguise, and lighting as these factors significantly reduce the recognition accuracy. To prevent problems in the visible spectrum, several researchers have recommended the use of infrared images. This paper provides an updated overview of deep infrared (IR) approaches in face recognition (FR) and analysis. First, we present the most widely used databases, both public and private, and the various metrics and loss functions that have been proposed and used in deep infrared techniques. We then review deep face analysis and recognition/identification methods proposed in recent years. In this review, we show that infrared techniques have given interesting results for face recognition, solving some of the problems encountered with visible spectrum techniques. We finally identify some weaknesses of current infrared FR approaches as well as many future research directions to address the IR FR limitations.

A deep learning model for drug screening and evaluation in bladder cancer organoids

Three-dimensional cell tissue culture, which produces biological structures termed organoids, has rapidly promoted the progress of biological research, including basic research, drug discovery, and regenerative medicine. However, due to the lack of algorithms and software, analysis of organoid growth is labor intensive and time-consuming. Currently it requires individual measurements using software such as ImageJ, leading to low screening efficiency when used for a high throughput screen. To solve this problem, we developed a bladder cancer organoid culture system, generated microscopic images, and developed a novel automatic image segmentation model, AU2Net (Attention and Cross U2Net). Using a dataset of two hundred images from growing organoids (day1 to day 7) and organoids with or without drug treatment, our model applies deep learning technology for image segmentation. To further improve the accuracy of model prediction, a variety of methods are integrated to improve the model's specificity, including adding Grouping Cross Merge (GCM) modules at the model's jump joints to strengthen the model's feature information. After feature information acquisition, a residual attentional gate (RAG) is added to suppress unnecessary feature propagation and improve the precision of organoids segmentation by establishing rich context-dependent models for local features. Experimental results show that each optimization scheme can significantly improve model performance. The sensitivity, specificity, and F1-Score of the ACU2Net model reached 94.81%, 88.50%, and 91.54% respectively, which exceed those of U-Net, Attention U-Net, and other available network models. Together, this novel ACU2Net model can provide more accurate segmentation results from organoid images and can improve the efficiency of drug screening evaluation using organoids.

Computer-aided classification of indirect immunofluorescence patterns on esophagus and split skin for the detection of autoimmune dermatoses

Autoimmune bullous dermatoses (AIBD) are rare diseases that affect human skin and mucous membranes. Clinically, they are characterized by blister formation and/or erosions. Depending on the structures involved and the depth of blister formation, they are grouped into pemphigus diseases, pemphigoid diseases, and dermatitis herpetiformis. Classification of AIBD into their sub-entities is crucial to guide treatment decisions. One of the most sensitive screening methods for initial differentiation of AIBD is the indirect immunofluorescence (IIF) microscopy on tissue sections of monkey esophagus and primate salt-split skin, which are used to detect disease-specific autoantibodies. Interpretation of IIF patterns requires a detailed examination of the image by trained professionals automating this process is a challenging task with these highly complex tissue substrates, but offers the great advantage of an objective result. Here, we present computer-aided classification of esophagus and salt-split skin IIF images. We show how deep networks can be adapted to the specifics and challenges of IIF image analysis by incorporating segmentation of relevant regions into the prediction process, and demonstrate their high accuracy. Using this semi-automatic extension can reduce the workload of professionals when reading tissue sections in IIF testing. Furthermore, these results on highly complex tissue sections show that further integration of semi-automated workflows into the daily workflow of diagnostic laboratories is promising.

Deep-Learning Pipeline for Statistical Quantification of Amorphous Two-Dimensional Materials

Aberration-corrected transmission electron microscopy enables imaging of two-dimensional (2D) materials with atomic resolution. However, dissecting the short-range-ordered structures in radiation-sensitive and amorphous 2D materials remains a significant challenge due to low atomic contrast and laborious manual evaluation. Here, we imaged carbon-based 2D materials with strong contrast, which is enabled by chromatic and spherical aberration correction at a low acceleration voltage. By constructing a deep-learning pipeline, atomic registry in amorphous 2D materials can be precisely determined, providing access to a full spectrum of quantitative data sets, including bond length/angle distribution, pair distribution function, and real-space polygon mapping. Accurate segmentation of micropores and surface contamination, together with robustness against background inhomogeneity, guaranteed the quantification validity in complex experimental images. The automated image analysis provides quantitative metrics with high efficiency and throughput, which may shed light on the structural understanding of short-range-ordered structures. In addition, the convolutional neural network can be readily generalized to crystalline materials, allowing for automatic defect identification and strain mapping.

A novel deep learning segmentation model for organoid-based drug screening

Organoids are self-organized three-dimensional in vitro cell cultures derived from stem cells. They can recapitulate organ development, tissue regeneration, and disease progression and, hence, have broad applications in drug discovery. However, the lack of effective graphic algorithms for organoid growth analysis has slowed the development of organoid-based drug screening. In this study, we take advantage of a bladder cancer organoid system and develop a deep learning model, the res-double dynamic conv attention U-Net (RDAU-Net) model, to improve the efficiency and accuracy of organoid-based drug screenings. In this RDAU-Net model, the dynamic convolution and attention modules are integrated. The feature-extracting capability of the encoder and the utilization of multi-scale information are substantially enhanced, and the semantic gap caused by skip connections has been filled, which substantially improved its anti-interference ability. A total of 200 images of bladder cancer organoids on culture days 1, 3, 5, and 7, with or without drug treatment, were employed for training and testing. Compared with the other variations of the U-Net model, the segmentation indicators, such as Intersection over Union and dice similarity coefficient, in the RDAU-Net model have been improved. In addition, this algorithm effectively prevented false identification and missing identification, while maintaining a smooth edge contour of segmentation results. In summary, we proposed a novel method based on a deep learning model which could significantly improve the efficiency and accuracy of high-throughput drug screening and evaluation using organoids.

Deep Learning-Based Framework for Fast and Accurate Acoustic Hologram Generation

Acoustic holography has been gaining attention for various applications, such as noncontact particle manipulation, noninvasive neuromodulation, and medical imaging. However, only a few studies on how to generate acoustic holograms have been conducted, and even conventional acoustic hologram algorithms show limited performance in the fast and accurate generation of acoustic holograms, thus hindering the development of novel applications. We here propose a deep learning-based framework to achieve fast and accurate acoustic hologram generation. The framework has an autoencoder-like architecture; thus, the unsupervised training is realized without any ground truth. For the framework, we demonstrate a newly developed hologram generator network, the holographic ultrasound generation network (HU-Net), which is suitable for unsupervised learning of hologram generation, and a novel loss function that is devised for energy-efficient holograms. Furthermore, for considering various hologram devices (i.e., ultrasound transducers), we propose a physical constraint (PC) layer. Simulation and experimental studies were carried out for two different hologram devices, such as a 3-D printed lens, attached to a single element transducer, and a 2-D ultrasound array. The proposed framework was compared with the iterative angular spectrum approach (IASA) and the state-of-the-art (SOTA) iterative optimization method, Diff-PAT. In the simulation study, our framework showed a few hundred times faster generation speed, along with comparable or even better reconstruction quality, than those of IASA and Diff-PAT. In the experimental study, the framework was validated with 3-D printed lenses fabricated based on different methods, and the physical effect of the lenses on the reconstruction quality was discussed. The outcomes of the proposed framework in various cases (i.e., hologram generator networks, loss functions, and hologram devices) suggest that our framework may become a very useful alternative tool for other existing acoustic hologram applications, and it can expand novel medical applications.

Computer-aided diagnosis of prostate cancer based on deep neural networks from multi-parametric magnetic resonance imaging

Objectives: To evaluate a new deep neural network (DNN)–based computer-aided diagnosis (CAD) method, namely, a prostate cancer localization network and an integrated multi-modal classification network, to automatically localize prostate cancer on multi-parametric magnetic resonance imaging (mp-MRI) and classify prostate cancer and non-cancerous tissues.

Materials and methods: The PROSTAREx database consists of a “training set” (330 suspected lesions from 204 cases) and a “test set” (208 suspected lesions from 104 cases). Sequences include T2-weighted, diffusion-weighted, Ktrans, and apparent diffusion coefficient (ADC) images. For the task of abnormal localization, inspired by V-net, we designed a prostate cancer localization network with mp-MRI data as input to achieve automatic localization of prostate cancer. Combining the concepts of multi-modal learning and ensemble learning, the integrated multi-modal classification network is based on the combination of mp-MRI data as input to distinguish prostate cancer from non-cancerous tissues through a series of operations such as convolution and pooling. The performance of each network in predicting prostate cancer was examined using the receiver operating curve (ROC), and the area under the ROC curve (AUC), sensitivity (TPR), specificity (TNR), accuracy, and Dice similarity coefficient (DSC) were calculated.

Results: The prostate cancer localization network exhibited excellent performance in localizing prostate cancer, with an average error of only 1.64 mm compared to the labeled results, an error of about 6%. On the test dataset, the network had a sensitivity of 0.92, specificity of 0.90, PPV of 0.91, NPV of 0.93, and DSC of 0.84. Compared with multi-modal classification networks, the performance of single-modal classification networks is slightly inadequate. The integrated multi-modal classification network performed best in classifying prostate cancer and non-cancerous tissues with a TPR of 0.95, TNR of 0.82, F1-Score of 0.8920, AUC of 0.912, and accuracy of 0.885, which fully confirmed the feasibility of the ensemble learning approach.

Conclusion: The proposed DNN-based prostate cancer localization network and integrated multi-modal classification network yielded high performance in experiments, demonstrating that the prostate cancer localization network and integrated multi-modal classification network can be used for computer-aided diagnosis (CAD) of prostate cancer localization and classification.

Fear memory-associated synaptic and mitochondrial changes revealed by deep learning-based processing of electron microscopy data

Serial section electron microscopy (ssEM) can provide comprehensive 3D ultrastructural information of the brain with exceptional computational cost. Targeted reconstruction of subcellular structures from ssEM datasets is less computationally demanding but still highly informative. We thus developed a region-CNN-based deep learning method to identify, segment, and reconstruct synapses and mitochondria to explore the structural plasticity of synapses and mitochondria in the auditory cortex of mice subjected to fear conditioning. Upon reconstructing over 135,000 mitochondria and 160,000 synapses, we find that fear conditioning significantly increases the number of mitochondria but decreases their size and promotes formation of multi-contact synapses, comprising a single axonal bouton and multiple postsynaptic sites from different dendrites. Modeling indicates that such multi-contact configuration increases the information storage capacity of new synapses by over 50%. With high accuracy and speed in reconstruction, our method yields structural and functional insight into cellular plasticity associated with fear learning.

Delineating mechanisms and design principles of Caenorhabditis elegans embryogenesis using in toto high-resolution imaging data and computational modeling

The nematode (roundworm) Caenorhabditis elegans is one of the most popular animal models for the study of developmental biology, as its invariant development and transparent body enable in toto cellular-resolution fluorescence microscopy imaging of developmental processes at 1-min intervals. This has led to the development of various computational tools for the systematic and automated analysis of imaging data to delineate the molecular and cellular processes throughout the embryogenesis of C. elegans, such as those associated with cell lineage, cell migration, cell morphology, and gene activity. In this review, we first introduce C. elegans embryogenesis and the development of techniques for tracking cell lineage and reconstructing cell morphology during this process. We then contrast the developmental modes of C. elegans and the customized technologies used for studying them with the ones of other animal models, highlighting its advantage for studying embryogenesis with exceptional spatial and temporal resolution. This is followed by an examination of the physical models that have been devised—based on accurate determinations of developmental processes afforded by analyses of imaging data—to interpret the early embryonic development of C. elegans from subcellular to intercellular levels of multiple cells, which focus on two key processes: cell polarization and morphogenesis. We subsequently discuss how quantitative data-based theoretical modeling has improved our understanding of the mechanisms of C. elegans embryogenesis. We conclude by summarizing the challenges associated with the acquisition of C. elegans embryogenesis data, the construction of algorithms to analyze them, and the theoretical interpretation.

NO Synthesis but Not Apoptosis, Mitosis or Inflammation Can Explain Correlations between Flow Directionality and Paracellular Permeability of Cultured Endothelium

Haemodynamic wall shear stress varies from site to site within the arterial system and is thought to cause local variation in endothelial permeability to macromolecules. Our aim was to investigate mechanisms underlying the changes in paracellular permeability caused by different patterns of shear stress in long-term culture. We used the swirling well system and a substrate-binding tracer that permits visualisation of transport at the cellular level. Permeability increased in the centre of swirled wells, where flow is highly multidirectional, and decreased towards the edge, where flow is more uniaxial, compared to static controls. Overall, there was a reduction in permeability. There were also decreases in early- and late-stage apoptosis, proliferation and mitosis, and there were significant correlations between the first three and permeability when considering variation from the centre to the edge under flow. However, data from static controls did not fit the same relation, and a cell-by-cell analysis showed that <5% of uptake under shear was associated with each of these events. Nuclear translocation of NF-κB p65 increased and then decreased with the duration of applied shear, as did permeability, but the spatial correlation between them was not significant. Application of an NO synthase inhibitor abolished the overall decrease in permeability caused by chronic shear and the difference in permeability between the centre and the edge of the well. Hence, shear and paracellular permeability appear to be linked by NO synthesis and not by apoptosis, mitosis or inflammation. The effect was mediated by an increase in transport through tricellular junctions.

Surface-Related and Internal Multiple Elimination Using Deep Learning

Multiple elimination has always been a key, challenge, and hotspot in the field of hydrocarbon exploration. However, each multiple elimination method comes with one or more limitations at present. The efficiency and success of each approach strongly depend on their corresponding prior assumptions, in particular for seismic data acquired from complex geological regions. The multiple elimination approach using deep learning encodes the input seismic data to multiple levels of abstraction and decodes those levels to reconstruct the primaries without multiples. In this study, we employ a classic convolution neural network (CNN) with a U-shaped architecture which uses extremely few seismic data for end-to-end training, strongly increasing the neural network speed. Then, we apply the trained network to predict all seismic data, which solves the problem of difficult elimination of global multiples, avoids the regularization of seismic data, and reduces massive amounts of calculation in traditional methods. Several synthetic and field experiments are conducted to validate the advantages of the trained network model. The results indicate that the model has the powerful generalization ability and high calculation efficiency for removing surface-related multiples and internal multiples effectively.

Applications of deep learning in electron microscopy

We review the growing use of machine learning in electron microscopy (EM) driven in part by the availability of fast detectors operating at kiloHertz frame rates leading to large data sets that cannot be processed using manually implemented algorithms. We summarize the various network architectures and error metrics that have been applied to a range of EM-related problems including denoising and inpainting. We then provide a review of the application of these in both physical and life sciences, highlighting how conventional networks and training data have been specifically modified for EM.

Person Localization Model Based on a Fusion of Acoustic and Visual Inputs

PLEA is an interactive, biomimetic robotic head with non-verbal communication capabilities. PLEA reasoning is based on a multimodal approach combining video and audio inputs to determine the current emotional state of a person. PLEA expresses emotions using facial expressions generated in real-time, which are projected onto a 3D face surface. In this paper, a more sophisticated computation mechanism is developed and evaluated. The model for audio-visual person separation can locate a talking person in a crowded place by combining input from the ResNet network with input from a hand-crafted algorithm. The first input is used to find human faces in the room, and the second input is used to determine the direction of the sound and to focus attention on a single person. After an information fusion procedure is performed, the face of the person speaking is matched with the corresponding sound direction. As a result of this procedure, the robot could start an interaction with the person based on non-verbal signals. The model was tested and evaluated under laboratory conditions by interaction with users. The results suggest that the methodology can be used efficiently to focus a robot’s attention on a localized person.

A Deep Learning Framework for Segmenting Brain Tumors Using MRI and Synthetically Generated CT Images

Multi-modal three-dimensional (3-D) image segmentation is used in many medical applications, such as disease diagnosis, treatment planning, and image-guided surgery. Although multi-modal images provide information that no single image modality alone can provide, integrating such information to be used in segmentation is a challenging task. Numerous methods have been introduced to solve the problem of multi-modal medical image segmentation in recent years. In this paper, we propose a solution for the task of brain tumor segmentation. To this end, we first introduce a method of enhancing an existing magnetic resonance imaging (MRI) dataset by generating synthetic computed tomography (CT) images. Then, we discuss a process of systematic optimization of a convolutional neural network (CNN) architecture that uses this enhanced dataset, in order to customize it for our task. Using publicly available datasets, we show that the proposed method outperforms similar existing methods.

Assessing the Impact of the Loss Function, Architecture and Image Type for Deep Learning-Based Wildfire Segmentation

Wildfires stand as one of the most relevant natural disasters worldwide, particularly more so due to the effect of climate change and its impact on various societal and environmental levels. In this regard, a significant amount of research has been done in order to address this issue, deploying a wide variety of technologies and following a multi-disciplinary approach. Notably, computer vision has played a fundamental role in this regard. It can be used to extract and combine information from several imaging modalities in regard to fire detection, characterization and wildfire spread forecasting. In recent years, there has been work pertaining to Deep Learning (DL)-based fire segmentation, showing very promising results. However, it is currently unclear whether the architecture of a model, its loss function, or the image type employed (visible, infrared, or fused) has the most impact on the fire segmentation results. In the present work, we evaluate different combinations of state-of-the-art (SOTA) DL architectures, loss functions, and types of images to identify the parameters most relevant to improve the segmentation results. We benchmark them to identify the top-performing ones and compare them to traditional fire segmentation techniques. Finally, we evaluate if the addition of attention modules on the best performing architecture can further improve the segmentation results. To the best of our knowledge, this is the first work that evaluates the impact of the architecture, loss function, and image type in the performance of DL-based wildfire segmentation models.