Deep Sequential Segmentation of Organs in Volumetric Medical Scans

Challenges and solutions of deep learning-based automated liver segmentation: A systematic review

The liver is one of the vital organs in the body. Precise liver segmentation in medical images is essential for liver disease treatment. The deep learning-based liver segmentation process faces several challenges. This research aims to analyze the challenges of liver segmentation in prior studies and identify the modifications made to network models and other enhancements implemented by researchers to tackle each challenge. In total, 88 articles from Scopus and ScienceDirect databases published between January 2016 and January 2022 have been studied. The liver segmentation challenges are classified into five main categories, each containing some subcategories. For each challenge, the proposed technique to overcome the challenge is investigated. The provided report details the authors, publication years, dataset types, imaging technologies, and evaluation metrics of all references for comparison. Additionally, a summary table outlines the challenges and solutions.

Prior-Knowledge Embedded U-Net-Based Fully Automatic Vessel Wall Volume Measurement of the Carotid Artery in 3D Ultrasound Image

The vessel-wall-volume (VWV) measured based on three-dimensional (3D) carotid artery (CA) ultrasound (US) images can help to assess carotid atherosclerosis and manage patients at risk for stroke. Manual involvement for measurement work is subjective and requires well-trained operators, and fully automatic measurement tools are not yet available. Thereby, we proposed a fully automatic VWV measurement framework (Auto-VWV) using a CA prior-knowledge embedded U-Net (CAP-UNet) to measure the VWV from 3D CA US images without manual intervention. The Auto-VWV framework is designed to improve the repeated VWV measuring consistency, which resulted in the first fully automatic framework for VWV measurement. CAP-UNet is developed to improve segmentation accuracy on the whole CA, which composed of a U-Net type backbone and three additional prior-knowledge learning modules. Specifically, a continuity learning module is used to learn the spatial continuity of the arteries in a sequence of image slices. A voxel evolution learning module was designed to learn the evolution of the artery in adjacent slices, and a topology learning module was used to learn the unique topology of the carotid artery. In two 3D CA US datasets, CAP-UNet architecture achieved state-of-the-art performance compared to eight competing models. Furthermore, CAP-UNet-based Auto-VWV achieved better accuracy and consistency than Auto-VWV based on competing models in the simulated repeated measurement. Finally, using 10 pairs of real repeatedly scanned samples, Auto-VWV achieved better VWV measurement reproducibility than intra- and inter-operator manual measurements. The code is available at https://github.com/Yue9603/Auto-VWV.

Semantic Segmentation of CT Liver Structures: A Systematic Review of Recent Trends and Bibliometric Analysis : Neural Network-based Methods for Liver Semantic Segmentation

The use of artificial intelligence (AI) in the segmentation of liver structures in medical images has become a popular research focus in the past half-decade. The performance of AI tools in screening for this task may vary widely and has been tested in the literature in various datasets. However, no scientometric report has provided a systematic overview of this scientific area. This article presents a systematic and bibliometric review of recent advances in neuronal network modeling approaches, mainly of deep learning, to outline the multiple research directions of the field in terms of algorithmic features. Therefore, a detailed systematic review of the most relevant publications addressing fully automatic semantic segmenting liver structures in Computed Tomography (CT) images in terms of algorithm modeling objective, performance benchmark, and model complexity is provided. The review suggests that fully automatic hybrid 2D and 3D networks are the top performers in the semantic segmentation of the liver. In the case of liver tumor and vasculature segmentation, fully automatic generative approaches perform best. However, the reported performance benchmark indicates that there is still much to be improved in segmenting such small structures in high-resolution abdominal CT scans.

Effect of childhood atropine treatment on adult choroidal thickness using sequential deep learning-enabled segmentation

PURPOSE

To describe choroidal thickness measurements using a sequential deep learning segmentation in adults who received childhood atropine treatment for myopia control.

DESIGN

Prospective, observational study.

METHODS

Choroidal thickness was measured by swept-source optical coherence tomography in adults who received childhood atropine, segmented using a sequential deep learning approach.

RESULTS

Of 422 eyes, 94 (22.3 %) had no previous exposure to atropine treatment, while 328 (77.7 %) had received topical atropine during childhood. After adjusting for age, sex, and axial length, childhood atropine exposure was associated with a thicker choroid by 32.1 μm (95 % CI, 9.2-55.0; P = 0.006) in the inner inferior, 23.5 μm (95 % CI, 1.9-45.1; P = 0.03) in the outer inferior, 21.8 μm (95 % CI, 0.76-42.9; P = 0.04) in the inner nasal, and 21.8 μm (95 % CI, 2.6-41.0; P = 0.03) in the outer nasal. Multivariable analysis, adjusted for age, sex, atropine use, and axial length, showed an independent association between central subfield choroidal thickness and the incidence of tessellated fundus (P < 0.001; OR, 0.97; 95 % CI, 0.96-0.98).

CONCLUSIONS

This study demonstrated that short-term (2-4 years) atropine treatment during childhood was associated with an increase in choroidal thickness of 20-40 μm in adulthood (10-20 years later), after adjusting for age, sex, and axial length. We also observed an independent association between eyes with thicker central choroidal measurements and reduced incidence of tessellated fundus. Our study suggests that childhood exposure to atropine treatment may affect choroidal thickness in adulthood.

Automatic segmentation and classification of frontal sinuses for sex determination from CBCT scans using a two-stage anatomy-guided attention network

Sex determination is essential for identifying unidentified individuals, particularly in forensic contexts. Traditional methods for sex determination involve manual measurements of skeletal features on CBCT scans. However, these manual measurements are labor-intensive, time-consuming, and error-prone. The purpose of this study was to automatically and accurately determine sex on a CBCT scan using a two-stage anatomy-guided attention network (SDetNet). SDetNet consisted of a 2D frontal sinus segmentation network (FSNet) and a 3D anatomy-guided attention network (SDNet). FSNet segmented frontal sinus regions in the CBCT images and extracted regions of interest (ROIs) near them. Then, the ROIs were fed into SDNet to predict sex accurately. To improve sex determination performance, we proposed multi-channel inputs (MSIs) and an anatomy-guided attention module (AGAM), which encouraged SDetNet to learn differences in the anatomical context of the frontal sinus between males and females. SDetNet showed superior sex determination performance in the area under the receiver operating characteristic curve, accuracy, Brier score, and specificity compared with the other 3D CNNs. Moreover, the results of ablation studies showed a notable improvement in sex determination with the embedding of both MSI and AGAM. Consequently, SDetNet demonstrated automatic and accurate sex determination by learning the anatomical context information of the frontal sinus on CBCT scans.

Multi-residual 2D network integrating spatial correlation for whole heart segmentation

Whole heart segmentation (WHS) has significant clinical value for cardiac anatomy, modeling, and analysis of cardiac function. This study aims to address the WHS accuracy on cardiac CT images, as well as the fast inference speed and low graphics processing unit (GPU) memory consumption required by practical clinical applications. Thus, we propose a multi-residual two-dimensional (2D) network integrating spatial correlation for WHS. The network performs slice-by-slice segmentation on three-dimensional cardiac CT images in a 2D encoder-decoder manner. In the network, a convolutional long short-term memory skip connection module is designed to perform spatial correlation feature extraction on the feature maps at different resolutions extracted by the sub-modules of the pre-trained ResNet-based encoder. Moreover, a decoder based on the multi-residual module is designed to analyze the extracted features from the perspectives of multi-scale and channel attention, thereby accurately delineating the various substructures of the heart. The proposed method is verified on a dataset of the multi-modality WHS challenge, an in-house WHS dataset, and a dataset of the abdominal organ segmentation challenge. The dice, Jaccard, average symmetric surface distance, Hausdorff distance, inference time, and maximum GPU memory of the WHS are 0.914, 0.843, 1.066 mm, 15.778 mm, 9.535 s, and 1905 MB, respectively. The proposed network has high accuracy, fast inference speed, minimal GPU memory consumption, strong robustness, and good generalization. It can be deployed to clinical practical applications for WHS and can be effectively extended and applied to other multi-organ segmentation fields. The source code is publicly available at https://github.com/nancy1984yan/MultiResNet-SC.

Deep learning to overcome Zernike phase-contrast nanoCT artifacts for automated micro-nano porosity segmentation in bone

Bone material contains a hierarchical network of micro- and nano-cavities and channels, known as the lacuna-canalicular network (LCN), that is thought to play an important role in mechanobiology and turnover. The LCN comprises micrometer-sized lacunae, voids that house osteocytes, and submicrometer-sized canaliculi that connect bone cells. Characterization of this network in three dimensions is crucial for many bone studies. To quantify X-ray Zernike phase-contrast nanotomography data, deep learning is used to isolate and assess porosity in artifact-laden tomographies of zebrafish bones. A technical solution is proposed to overcome the halo and shade-off domains in order to reliably obtain the distribution and morphology of the LCN in the tomographic data. Convolutional neural network (CNN) models are utilized with increasing numbers of images, repeatedly validated by `error loss' and `accuracy' metrics. U-Net and Sensor3D CNN models were trained on data obtained from two different synchrotron Zernike phase-contrast transmission X-ray microscopes, the ANATOMIX beamline at SOLEIL (Paris, France) and the P05 beamline at PETRA III (Hamburg, Germany). The Sensor3D CNN model with a smaller batch size of 32 and a training data size of 70 images showed the best performance (accuracy 0.983 and error loss 0.032). The analysis procedures, validated by comparison with human-identified ground-truth images, correctly identified the voids within the bone matrix. This proposed approach may have further application to classify structures in volumetric images that contain non-linear artifacts that degrade image quality and hinder feature identification.

A Deep Convolutional Gated Recurrent Unit for CT Image Reconstruction

Computed tomography (CT) is one of the most important medical imaging technologies in use today. Most commercial CT products use a technique known as the filtered backprojection (FBP) that is fast and can produce decent image quality when an X-ray dose is high. However, the FBP is not good enough on low-dose X-ray CT imaging because the CT image reconstruction problem becomes more stochastic. A more effective reconstruction technique proposed recently and implemented in a limited number of CT commercial products is an iterative reconstruction (IR). The IR technique is based on a Bayesian formulation of the CT image reconstruction problem with an explicit model of the CT scanning, including its stochastic nature, and a prior model that incorporates our knowledge about what a good CT image should look like. However, constructing such prior knowledge is more complicated than it seems. In this article, we propose a novel neural network for CT image reconstruction. The network is based on the IR formulation and constructed with a recurrent neural network (RNN). Specifically, we transform the gated recurrent unit (GRU) into a neural network performing CT image reconstruction. We call it "GRU reconstruction." This neural network conducts concurrent dual-domain learning. Many deep learning (DL)-based methods in medical imaging are single-domain learning, but dual-domain learning performs better because it learns from both the sinogram and the image domain. In addition, we propose backpropagation through stage (BPTS) as a new RNN backpropagation algorithm. It is similar to the backpropagation through time (BPTT) of an RNN; however, it is tailored for iterative optimization. Results from extensive experiments indicate that our proposed method outperforms conventional model-based methods, single-domain DL methods, and state-of-the-art DL techniques in terms of the root mean squared error (RMSE), the peak signal-to-noise ratio (PSNR), and the structure similarity (SSIM) and in terms of visual appearance.

MS-FANet: Multi-scale feature attention network for liver tumor segmentation

Accurate segmentation of liver tumors is a prerequisite for early diagnosis of liver cancer. Segmentation networks extract features continuously at the same scale, which cannot adapt to the variation of liver tumor volume in computed tomography (CT). Hence, a multi-scale feature attention network (MS-FANet) for liver tumor segmentation is proposed in this paper. The novel residual attention (RA) block and multi-scale atrous downsampling (MAD) are introduced in the encoder of MS-FANet to sufficiently learn variable tumor features and extract tumor features at different scales simultaneously. The dual-path feature (DF) filter and dense upsampling (DU) are introduced in the feature reduction process to reduce effective features for the accurate segmentation of liver tumors. On the public LiTS dataset and 3DIRCADb dataset, MS-FANet achieved 74.2% and 78.0% of average Dice, respectively, outperforming most state-of-the-art networks, this strongly proves the excellent liver tumor segmentation performance and the ability to learn features at different scales.

The Synergy between Deep Learning and Organs-on-Chips for High-Throughput Drug Screening: A Review

Organs-on-chips (OoCs) are miniature microfluidic systems that have arguably become a class of advanced in vitro models. Deep learning, as an emerging topic in machine learning, has the ability to extract a hidden statistical relationship from the input data. Recently, these two areas have become integrated to achieve synergy for accelerating drug screening. This review provides a brief description of the basic concepts of deep learning used in OoCs and exemplifies the successful use cases for different types of OoCs. These microfluidic chips are of potential to be assembled as highly potent human-on-chips with complex physiological or pathological functions. Finally, we discuss the future supply with perspectives and potential challenges in terms of combining OoCs and deep learning for image processing and automation designs.

Deep Learning-Based Medical Images Segmentation of Musculoskeletal Anatomical Structures: A Survey of Bottlenecks and Strategies

By leveraging the recent development of artificial intelligence algorithms, several medical sectors have benefited from using automatic segmentation tools from bioimaging to segment anatomical structures. Segmentation of the musculoskeletal system is key for studying alterations in anatomical tissue and supporting medical interventions. The clinical use of such tools requires an understanding of the proper method for interpreting data and evaluating their performance. The current systematic review aims to present the common bottlenecks for musculoskeletal structures analysis (e.g., small sample size, data inhomogeneity) and the related strategies utilized by different authors. A search was performed using the PUBMED database with the following keywords: deep learning, musculoskeletal system, segmentation. A total of 140 articles published up until February 2022 were obtained and analyzed according to the PRISMA framework in terms of anatomical structures, bioimaging techniques, pre/post-processing operations, training/validation/testing subset creation, network architecture, loss functions, performance indicators and so on. Several common trends emerged from this survey; however, the different methods need to be compared and discussed based on each specific case study (anatomical region, medical imaging acquisition setting, study population, etc.). These findings can be used to guide clinicians (as end users) to better understand the potential benefits and limitations of these tools.

A deep residual attention-based U-Net with a biplane joint method for liver segmentation from CT scans

Liver tumours are diseases with high morbidity and high deterioration probabilities, and accurate liver area segmentation from computed tomography (CT) scans is a prerequisite for quick tumour diagnosis. While 2D network segmentation methods can perform segmentation with lower device performance requirements, they often discard the rich 3D spatial information contained in CT scans, limiting their segmentation accuracy. Hence, a deep residual attention-based U-shaped network (DRAUNet) with a biplane joint method for liver segmentation is proposed in this paper, where the biplane joint method introduces coronal CT slices to assist the transverse slices with segmentation, incorporating more 3D spatial information into the segmentation results to improve the segmentation performance of the network. Additionally, a novel deep residual block (DR block) and dual-effect attention module (DAM) are introduced in DRAUNet, where the DR block has deeper layers and two shortcut paths. The DAM efficiently combines the correlations of feature channels and the spatial locations of feature maps. The DRAUNet with the biplane joint method is tested on three datasets, Liver Tumour Segmentation (LiTS), 3D Image Reconstruction for Comparison of Algorithms Database (3DIRCADb), and Segmentation of the Liver Competition 2007 (Sliver07), and it achieves 97.3%, 97.4%, and 96.9% Dice similarity coefficients (DSCs) for liver segmentation, respectively, outperforming most state-of-the-art networks; this strongly demonstrates the segmentation performance of DRAUNet and the ability of the biplane joint method to obtain 3D spatial information from 3D images.

Multi-Organ Segmentation Using a Low-Resource Architecture

Since their inception, deep-learning architectures have shown promising results for automatic segmentation. However, despite the technical advances introduced by fully convolutional networks, generative adversarial networks or recurrent neural networks, and their usage in hybrid architectures, automatic segmentation in the medical field is still not used at scale. One main reason is related to data scarcity and quality, which in turn generates a lack of annotated data that hinder the generalization of the models. The second main issue refers to challenges in training deep models. This process uses large amounts of GPU memory (that might exceed current hardware limitations) and requires high training times. In this article, we want to prove that despite these issues, good results can be obtained even when using a lower resource architecture, thus opening the way for more researchers to employ and use deep neural networks. In achieving the multi-organ segmentation, we are employing modern pre-processing techniques, a smart model design and fusion between several models trained on the same dataset. Our architecture is compared against state-of-the-art methods employed in a publicly available challenge and the notable results prove the effectiveness of our method.

Canal-Net for automatic and robust 3D segmentation of mandibular canals in CBCT images using a continuity-aware contextual network

The purpose of this study was to propose a continuity-aware contextual network (Canal-Net) for the automatic and robust 3D segmentation of the mandibular canal (MC) with high consistent accuracy throughout the entire MC volume in cone-beam CT (CBCT) images. The Canal-Net was designed based on a 3D U-Net with bidirectional convolutional long short-term memory (ConvLSTM) under a multi-task learning framework. Specifically, the Canal-Net learned the 3D anatomical context information of the MC by incorporating spatio-temporal features from ConvLSTM, and also the structural continuity of the overall MC volume under a multi-task learning framework using multi-planar projection losses complementally. The Canal-Net showed higher segmentation accuracies in 2D and 3D performance metrics (p < 0.05), and especially, a significant improvement in Dice similarity coefficient scores and mean curve distance (p < 0.05) throughout the entire MC volume compared to other popular deep learning networks. As a result, the Canal-Net achieved high consistent accuracy in 3D segmentations of the entire MC in spite of the areas of low visibility by the unclear and ambiguous cortical bone layer. Therefore, the Canal-Net demonstrated the automatic and robust 3D segmentation of the entire MC volume by improving structural continuity and boundary details of the MC in CBCT images.

Practical utility of liver segmentation methods in clinical surgeries and interventions

Clinical imaging (e.g., magnetic resonance imaging and computed tomography) is a crucial adjunct for clinicians, aiding in the diagnosis of diseases and planning of appropriate interventions. This is especially true in malignant conditions such as hepatocellular carcinoma (HCC), where image segmentation (such as accurate delineation of liver and tumor) is the preliminary step taken by the clinicians to optimize diagnosis, staging, and treatment planning and intervention (e.g., transplantation, surgical resection, radiotherapy, PVE, embolization, etc). Thus, segmentation methods could potentially impact the diagnosis and treatment outcomes. This paper comprehensively reviews the literature (during the year 2012-2021) for relevant segmentation methods and proposes a broad categorization based on their clinical utility (i.e., surgical and radiological interventions) in HCC. The categorization is based on the parameters such as precision, accuracy, and automation.

The value of longitudinal clinical data and paired CT scans in predicting the deterioration of COVID-19 revealed by an artificial intelligence system

The respective value of clinical data and CT examinations in predicting COVID-19 progression is unclear, because the CT scans and clinical data previously used are not synchronized in time. To address this issue, we collected 119 COVID-19 patients with 341 longitudinal CT scans and paired clinical data, and we developed an AI system for the prediction of COVID-19 deterioration. By combining features extracted from CT and clinical data with our system, we can predict whether a patient will develop severe symptoms during hospitalization. Complementary to clinical data, CT examinations show significant add-on values for the prediction of COVID-19 progression in the early stage of COVID-19, especially in the 6^th to 8^th day after the symptom onset, indicating that this is the ideal time window for the introduction of CT examinations. We release our AI system to provide clinicians with additional assistance to optimize CT usage in the clinical workflow.

•

COVID-19 patients with 341 longitudinal CT scans and paired clinical data included

•

A new AI model for the prediction of COVID-19 progression was developed

•

CT scans show significant add-on value over clinical data for the prediction

•

Day 6–8 after the onset of COVID-19 symptoms is an ideal time window for a CT scan

Advanced characterization of 3D structure and porosity of ordinary portland cement (OPC) mortar using plasma focused ion beam tomography and X-ray computed tomography

The visualization and quantification of pore networks and main phases have been critical research topics in cementitious materials as many critical mechanical and chemical properties and infrastructure reliability rely on these 3-D characteristics. In this study, we realized the mesoscale serial sectioning and analysis up to ∼80 μm by ∼90 μm by ∼60 μm on portland cement mortar using plasma focused ion beam (PFIB) for the first time. The workflow of working with mortar and PFIB was established applying a prepositioned hard silicon mask to reduce curtaining. Segmentation with minimal human interference was performed using a trained neural network, in which multiple types of segmentation models were compared. Combining PFIB analysis at microscale with X-ray micro-computed tomography, the analysis of capillary pores and air voids ranging from hundreds of nanometers (nm) to millimeters (mm) can be conducted. The volume fraction of large capillary pores and air voids are 11.5% and 12.7%, respectively. Moreover, the skeletonization of connected capillary pores clearly shows fluid transport pathways, which is a key factor determining durability performance of concrete in aggressive environments. Another interesting aspect of the FIB tomography is the reconstruction of anhydrous phases, which could enable direct study of hydration kinetics of individual cement phases. This article is protected by copyright. All rights reserved.

Using a Convolutional Neural Network and Convolutional Long Short-term Memory to Automatically Detect Aneurysms on 2D Digital Subtraction Angiography Images: Framework Development and Validation

BACKGROUND

It is hard to distinguish cerebral aneurysms from overlapping vessels in 2D digital subtraction angiography (DSA) images due to these images' lack of spatial information.

OBJECTIVE

The aims of this study were to (1) construct a deep learning diagnostic system to improve the ability to detect posterior communicating artery aneurysms on 2D DSA images and (2) validate the efficiency of the deep learning diagnostic system in 2D DSA aneurysm detection.

METHODS

We proposed a 2-stage detection system. First, we established the region localization stage to automatically locate specific detection regions of raw 2D DSA sequences. Second, in the intracranial aneurysm detection stage, we constructed a bi-input+RetinaNet+convolutional long short-term memory (C-LSTM) framework to compare its performance for aneurysm detection with that of 3 existing frameworks. Each of the frameworks had a 5-fold cross-validation scheme. The receiver operating characteristic curve, the area under the curve (AUC) value, mean average precision, sensitivity, specificity, and accuracy were used to assess the abilities of different frameworks.

RESULTS

A total of 255 patients with posterior communicating artery aneurysms and 20 patients without aneurysms were included in this study. The best AUC values of the RetinaNet, RetinaNet+C-LSTM, bi-input+RetinaNet, and bi-input+RetinaNet+C-LSTM frameworks were 0.95, 0.96, 0.92, and 0.97, respectively. The mean sensitivities of the RetinaNet, RetinaNet+C-LSTM, bi-input+RetinaNet, and bi-input+RetinaNet+C-LSTM frameworks and human experts were 89% (range 67.02%-98.43%), 88% (range 65.76%-98.06%), 87% (range 64.53%-97.66%), 89% (range 67.02%-98.43%), and 90% (range 68.30%-98.77%), respectively. The mean specificities of the RetinaNet, RetinaNet+C-LSTM, bi-input+RetinaNet, and bi-input+RetinaNet+C-LSTM frameworks and human experts were 80% (range 56.34%-94.27%), 89% (range 67.02%-98.43%), 86% (range 63.31%-97.24%), 93% (range 72.30%-99.56%), and 90% (range 68.30%-98.77%), respectively. The mean accuracies of the RetinaNet, RetinaNet+C-LSTM, bi-input+RetinaNet, and bi-input+RetinaNet+C-LSTM frameworks and human experts were 84.50% (range 69.57%-93.97%), 88.50% (range 74.44%-96.39%), 86.50% (range 71.97%-95.22%), 91% (range 77.63%-97.72%), and 90% (range 76.34%-97.21%), respectively.

CONCLUSIONS

According to our results, more spatial and temporal information can help improve the performance of the frameworks. Therefore, the bi-input+RetinaNet+C-LSTM framework had the best performance when compared to that of the other frameworks. Our study demonstrates that our system can assist physicians in detecting intracranial aneurysms on 2D DSA images.

X-ray microscopy enables multiscale high-resolution 3D imaging of plant cells, tissues, and organs

Capturing complete internal anatomies of plant organs and tissues within their relevant morphological context remains a key challenge in plant science. While plant growth and development are inherently multiscale, conventional light, fluorescence, and electron microscopy platforms are typically limited to imaging of plant microstructure from small flat samples that lack a direct spatial context to, and represent only a small portion of, the relevant plant macrostructures. We demonstrate technical advances with a lab-based X-ray microscope (XRM) that bridge the imaging gap by providing multiscale high-resolution three-dimensional (3D) volumes of intact plant samples from the cell to the whole plant level. Serial imaging of a single sample is shown to provide sub-micron 3D volumes co-registered with lower magnification scans for explicit contextual reference. High-quality 3D volume data from our enhanced methods facilitate sophisticated and effective computational segmentation. Advances in sample preparation make multimodal correlative imaging workflows possible, where a single resin-embedded plant sample is scanned via XRM to generate a 3D cell-level map, and then used to identify and zoom in on sub-cellular regions of interest for high-resolution scanning electron microscopy. In total, we present the methodologies for use of XRM in the multiscale and multimodal analysis of 3D plant features using numerous economically and scientifically important plant systems.

Robust Deep 3D Blood Vessel Segmentation Using Structural Priors

Deep learning has enabled significant improvements in the accuracy of 3D blood vessel segmentation. Open challenges remain in scenarios where labeled 3D segmentation maps for training are severely limited, as is often the case in practice, and in ensuring robustness to noise. Inspired by the observation that 3D vessel structures project onto 2D image slices with informative and unique edge profiles, we propose a novel deep 3D vessel segmentation network guided by edge profiles. Our network architecture comprises a shared encoder and two decoders that learn segmentation maps and edge profiles jointly. 3D context is mined in both the segmentation and edge prediction branches by employing bidirectional convolutional long-short term memory (BCLSTM) modules. 3D features from the two branches are concatenated to facilitate learning of the segmentation map. As a key contribution, we introduce new regularization terms that: a) capture the local homogeneity of 3D blood vessel volumes in the presence of biomarkers; and b) ensure performance robustness to domain-specific noise by suppressing false positive responses. Experiments on benchmark datasets with ground truth labels reveal that the proposed approach outperforms state-of-the-art techniques on standard measures such as DICE overlap and mean Intersection-over-Union. The performance gains of our method are even more pronounced when training is limited. Furthermore, the computational cost of our network inference is among the lowest compared with state-of-the-art.

Hybrid-attention densely connected U-Net with GAP for extracting livers from CT volumes

PURPOSE

Liver segmentation is an important step in the clinical treatment of liver cancer, and accurate and automatic liver segmentation methods are extremely important. U-Net has been used as the benchmark for many medical segmentation networks, but it cannot fully utilize low-resolution information and global contextual information. To solve these problems, we propose a new network architecture named the hybrid-attention densely connected U-Net (HDU-Net).

METHODS

The proposed HDU-Net has three main changes relative to U-Net, as follows: (1) It uses a densely connected structure and dilated convolution to achieve feature reuse and avoid information loss. (2) A global average pooling block is proposed to further augment the receptive field and improve the segmentation accuracy of the network for small or disconnected liver regions. (3) By combining the spatial attention and channel attention mechanisms, a hybrid attention structure is proposed to replace the skip connection component to filter and integrate low-resolution information.

RESULTS

Experiments conducted on the LITS2017, 3Dircadb and Sliver07 datasets show that the proposed model can segment the liver accurately and effectively. Dice scores reach 96.5%, 96.18%, and 97.57% on these datasets, respectively, constituting results that are superior to many previously proposed methods.

CONCLUSIONS

The experimental liver segmentation results have demonstrated that our proposed network provides improved segmentation performance in comparison with other networks. The experimental results without postprocessing confirmed that our network solves the oversegmentation and undersegmentation problems to some extent. The proposed model is effective, robust, and efficient in terms of liver segmentation without requiring extensive training time or a very large dataset.

Multi-task learning approach for volumetric segmentation and reconstruction in 3D OCT images

The choroid is the vascular layer of the eye that supplies photoreceptors with oxygen. Changes in the choroid are associated with many pathologies including myopia where the choroid progressively thins due to axial elongation. To quantize these changes, there is a need to automatically and accurately segment the choroidal layer from optical coherence tomography (OCT) images. In this paper, we propose a multi-task learning approach to segment the choroid from three-dimensional OCT images. Our proposed architecture aggregates the spatial context from adjacent cross-sectional slices to reconstruct the central slice. Spatial context learned by this reconstruction mechanism is then fused with a U-Net based architecture for segmentation. The proposed approach was evaluated on volumetric OCT scans of 166 myopic eyes acquired with a commercial OCT system, and achieved a cross-validation Intersection over Union (IoU) score of 94.69% which significantly outperformed (p<0.001) the other state-of-the-art methods on the same data set. Choroidal thickness maps generated by our approach also achieved a better structural similarity index (SSIM) of 72.11% with respect to the groundtruth. In particular, our approach performs well for highly challenging eyes with thinner choroids. Compared to other methods, our proposed approach also requires lesser processing time and has lower computational requirements. The results suggest that our proposed approach could potentially be used as a fast and reliable method for automated choroidal segmentation.

Number of cells in parathyroid tissue in primary hyperparathyroidism cases and its relationship with serum calcium value

BACKGROUND

The relationship between serum calcium (Ca) level to serum parathyroid hormone (PTH), phosphorus (P) levels and tissue properties of the parathyroid gland is unknown in primary hyperparathyroidism cases. Revealing this relationship may be useful for understanding the etiopathogenesis of primary hyperparathyroidism and determining the time of treatment.

METHODS

Ninety patients (71 females, 19 males, age range; 27-73 years, average age; 46) who underwent single gland excision with the diagnosis of primary hyperparathyroidism were studied. The patients were divided into 2 groups as serum Ca level <12 and serum Ca level ≥12. Age and sex of the patients, mean cell number of the gland, mean volume of the gland, serum levels of PTH, P, and histopathologic type of hyperplasia were evaluated.

RESULTS

The mean cell number per cubic centimeter is 22.9 (10-220 range) million in all glands. Serum Ca level was <12 in 82 (91.1%) of the patients, and ≥12 in 8 (8.9%) cases. Mean cell number of the gland, mean volume of the gland, existence of cystic hyperplasia of the gland, serum levels of PTH and P were statistically significant between the 2 groups (P < .001, P < .001, P < .05, P < .001, P < .05 respectively).

CONCLUSION

In primary hyperparathyroidism cases serum Ca level is not related to age and sex but directly related to proportionals to the cell number and volume of the gland and serum levels of PTH, inversely related to cystic hyperplasia and serum levels of P. Early surgical intervention should be planned since the serum Ca level will be high in large adenomas with a noncystic radiological appearance.

IFSS-Net: Interactive Few-Shot Siamese Network for Faster Muscle Segmentation and Propagation in Volumetric Ultrasound

We present an accurate, fast and efficient method for segmentation and muscle mask propagation in 3D freehand ultrasound data, towards accurate volume quantification. A deep Siamese 3D Encoder-Decoder network that captures the evolution of the muscle appearance and shape for contiguous slices is deployed. We use it to propagate a reference mask annotated by a clinical expert. To handle longer changes of the muscle shape over the entire volume and to provide an accurate propagation, we devise a Bidirectional Long Short Term Memory module. Also, to train our model with a minimal amount of training samples, we propose a strategy combining learning from few annotated 2D ultrasound slices with sequential pseudo-labeling of the unannotated slices. We introduce a decremental update of the objective function to guide the model convergence in the absence of large amounts of annotated data. After training with a few volumes, the decremental update strategy switches from a weak supervised training to a few-shot setting. Finally, to handle the class-imbalance between foreground and background muscle pixels, we propose a parametric Tversky loss function that learns to penalize adaptively the false positives and the false negatives. We validate our approach for the segmentation, label propagation, and volume computation of the three low-limb muscles on a dataset of 61600 images from 44 subjects. We achieve a Dice score coefficient of over 95% and a volumetric error of 1.6035 ± 0.587%.

Interactive contouring through contextual deep learning

PURPOSE

To investigate a deep learning approach that enables three-dimensional (3D) segmentation of an arbitrary structure of interest given a user provided two-dimensional (2D) contour for context. Such an approach could decrease delineation times and improve contouring consistency, particularly for anatomical structures for which no automatic segmentation tools exist.

METHODS

A series of deep learning segmentation models using a Recurrent Residual U-Net with attention gates was trained with a successively expanding training set. Contextual information was provided to the models, using a previously contoured slice as an input, in addition to the slice to be contoured. In total, 6 models were developed, and 19 different anatomical structures were used for training and testing. Each of the models was evaluated for all 19 structures, even if they were excluded from the training set, in order to assess the model's ability to segment unseen structures of interest. Each model's performance was evaluated using the Dice similarity coefficient (DSC), Hausdorff distance, and relative added path length (APL).

RESULTS

The segmentation performance for seen and unseen structures improved when the training set was expanded by addition of structures previously excluded from the training set. A model trained exclusively on heart structures achieved a DSC of 0.33, HD of 44 mm, and relative APL of 0.85 when segmenting the spleen, whereas a model trained on a diverse set of structures, but still excluding the spleen, achieved a DSC of 0.80, HD of 13 mm, and relative APL of 0.35. Iterative prediction performed better compared to direct prediction when considering unseen structures.

CONCLUSIONS

Training a contextual deep learning model on a diverse set of structures increases the segmentation performance for the structures in the training set, but importantly enables the model to generalize and make predictions even for unseen structures that were not represented in the training set. This shows that user-provided context can be incorporated into deep learning contouring to facilitate semi-automatic segmentation of CT images for any given structure. Such an approach can enable faster de-novo contouring in clinical practice.

Robust and Accurate Mandible Segmentation on Dental CBCT Scans Affected by Metal Artifacts Using a Prior Shape Model

Accurate mandible segmentation is significant in the field of maxillofacial surgery to guide clinical diagnosis and treatment and develop appropriate surgical plans. In particular, cone-beam computed tomography (CBCT) images with metal parts, such as those used in oral and maxillofacial surgery (OMFS), often have susceptibilities when metal artifacts are present such as weak and blurred boundaries caused by a high-attenuation material and a low radiation dose in image acquisition. To overcome this problem, this paper proposes a novel deep learning-based approach (SASeg) for automated mandible segmentation that perceives overall mandible anatomical knowledge. SASeg utilizes a prior shape feature extractor (PSFE) module based on a mean mandible shape, and recurrent connections maintain the continuity structure of the mandible. The effectiveness of the proposed network is substantiated on a dental CBCT dataset from orthodontic treatment containing 59 patients. The experiments show that the proposed SASeg can be easily used to improve the prediction accuracy in a dental CBCT dataset corrupted by metal artifacts. In addition, the experimental results on the PDDCA dataset demonstrate that, compared with the state-of-the-art mandible segmentation models, our proposed SASeg can achieve better segmentation performance.

Automatic liver segmentation using 3D convolutional neural networks with a hybrid loss function

PURPOSE

Automatic liver segmentation from abdominal computed tomography (CT) images is a fundamental task in computer-assisted liver surgery programs. Many liver segmentation algorithms are very sensitive to fuzzy boundaries and heterogeneous pathologies, especially when the data are scarce. To solve these problems, we propose an automatic liver segmentation framework based on three-dimensional (3D) convolutional neural networks with a hybrid loss function.

METHODS

Two networks are incorporated in our method with the first being a liver shape autoencoder that is trained to obtain compressed codes of liver shapes, and the second being a liver segmentation network that is trained with a hybrid loss function. The design of the hybrid loss function is comprised of three parts. The first part is an adaptively weighted cross-entropy loss, which pays more attention to misclassified pixels. The second part is an edge-preserving smoothness loss, which guarantees that the adjacent pixels with the same label have similar outputs, while dissimilar for pixels with different labels. The third part of the loss is a shape constraint to model high-level structural differences based on the learned shape codes. Both networks use 3D operations for data processing. In our experiments, data augmentation is performed at both the training and the test stage.

RESULTS

We extensively evaluated our method on two datasets: the Segmentation of the Liver Competition 2007 (Sliver07), and the Combined (CT-MR) Healthy Abdominal Organ Segmentation (CHAOS) Challenge. Finally, with only 20 training scans, we achieved the best score of 82.55 on the Sliver07 challenge, and a score of 83.02 on the CHAOS challenge.

CONCLUSIONS

In this study, we proposed a novel hybrid loss to overcome the difficulties in liver segmentation. The quantitative and qualitative results demonstrate that our method is highly suited for pathological liver segmentation, even when trained with a small dataset.

Evaluation of multislice inputs to convolutional neural networks for medical image segmentation

PURPOSE

When using convolutional neural networks (CNNs) for segmentation of organs and lesions in medical images, the conventional approach is to work with inputs and outputs either as single slice [two-dimensional (2D)] or whole volumes [three-dimensional (3D)]. One common alternative, in this study denoted as pseudo-3D, is to use a stack of adjacent slices as input and produce a prediction for at least the central slice. This approach gives the network the possibility to capture 3D spatial information, with only a minor additional computational cost.

METHODS

In this study, we systematically evaluate the segmentation performance and computational costs of this pseudo-3D approach as a function of the number of input slices, and compare the results to conventional end-to-end 2D and 3D CNNs, and to triplanar orthogonal 2D CNNs. The standard pseudo-3D method regards the neighboring slices as multiple input image channels. We additionally design and evaluate a novel, simple approach where the input stack is a volumetric input that is repeatably convolved in 3D to obtain a 2D feature map. This 2D map is in turn fed into a standard 2D network. We conducted experiments using two different CNN backbone architectures and on eight diverse data sets covering different anatomical regions, imaging modalities, and segmentation tasks.

RESULTS

We found that while both pseudo-3D methods can process a large number of slices at once and still be computationally much more efficient than fully 3D CNNs, a significant improvement over a regular 2D CNN was only observed with two of the eight data sets. triplanar networks had the poorest performance of all the evaluated models. An analysis of the structural properties of the segmentation masks revealed no relations to the segmentation performance with respect to the number of input slices. A post hoc rank sum test which combined all metrics and data sets yielded that only our newly proposed pseudo-3D method with an input size of 13 slices outperformed almost all methods.

CONCLUSION

In the general case, multislice inputs appear not to improve segmentation results over using 2D or 3D CNNs. For the particular case of 13 input slices, the proposed novel pseudo-3D method does appear to have a slight advantage across all data sets compared to all other methods evaluated in this work.

Volumetric Choroidal Segmentation Using Sequential Deep Learning Approach in High Myopia Subjects

Many ocular diseases are associated with choroidal changes. Therefore, it is crucial to be able to segment the choroid to study its properties. Previous methods for choroidal segmentation have focused on single cross-sectional scans. Volumetric choroidal segmentation has yet to be widely reported. In this paper, we propose a sequential segmentation approach using a variation of U-Net with a bidirectional C-LSTM(Convolutional Long Short Term Memory) module in the bottleneck region. The model is evaluated on volumetric scans from 40 high myopia subjects, obtained using SS-OCT(Swept Source Optical Coherence Tomography). A comparison with other U-Net-based variants is also presented. The results demonstrate that volumetric segmentation of the choroid can be achieved with an accuracy of IoU(Intersection over Union) 0.92.Clinical relevance- This deep learning approach can automatically segment the choroidal volume, which can enable better evaluation and monitoring at ocular diseases.

Stacked Bidirectional Convolutional LSTMs for Deriving 3D Non-Contrast CT From Spatiotemporal 4D CT

The imaging workup in acute stroke can be simplified by deriving non-contrast CT (NCCT) from CT perfusion (CTP) images. This results in reduced workup time and radiation dose. To achieve this, we present a stacked bidirectional convolutional LSTM (C-LSTM) network to predict 3D volumes from 4D spatiotemporal data. Several parameterizations of the C-LSTM network were trained on a set of 17 CTP-NCCT pairs to learn to derive a NCCT from CTP and were subsequently quantitatively evaluated on a separate cohort of 16 cases. The results show that the C-LSTM network clearly outperforms the baseline and competitive convolutional neural network methods. We show good scalability and performance of the method by continued training and testing on an independent dataset which includes pathology of 80 and 83 CTP-NCCT pairs, respectively. C-LSTM is, therefore, a promising general deep learning approach to learn from high-dimensional spatiotemporal medical images.

Deep Learning-Based Automatic Segmentation of Lumbosacral Nerves on CT for Spinal Intervention: A Translational Study

BACKGROUND AND PURPOSE

3D reconstruction of a targeted area ("safe" triangle and Kambin triangle) may benefit the viability assessment of transforaminal epidural steroid injection, especially at the L5/S1 level. However, manual segmentation of lumbosacral nerves for 3D reconstruction is time-consuming. The aim of this study was to investigate the feasibility of deep learning-based segmentation of lumbosacral nerves on CT and the reconstruction of the safe triangle and Kambin triangle.

MATERIALS AND METHODS

A total of 50 cases of spinal CT were manually labeled for lumbosacral nerves and bones using Slicer 4.8. The ratio of training/validation/testing was 32:8:10. A 3D U-Net was adopted to build the model SPINECT for automatic segmentations of lumbosacral structures. The Dice score, pixel accuracy, and Intersection over Union were computed to assess the segmentation performance of SPINECT. The areas of Kambin and safe triangles were measured to validate the 3D reconstruction.

RESULTS

The results revealed successful segmentation of lumbosacral bone and nerve on CT. The average pixel accuracy for bone was 0.940, and for nerve, 0.918. The average Intersection over Union for bone was 0.897 and for nerve, 0.827. The Dice score for bone was 0.945, and for nerve, it was 0.905. There were no significant differences in the quantified Kambin triangle or safe triangle between manually segmented images and automatically segmented images (P > .05).

CONCLUSIONS

Deep learning-based automatic segmentation of lumbosacral structures (nerves and bone) on routine CT is feasible, and SPINECT-based 3D reconstruction of safe and Kambin triangles is also validated.