Automatic Multi-Organ Segmentation on Abdominal CT With Dense V-Networks

Going to Extremes: Weakly Supervised Medical Image Segmentation

Medical image annotation is a major hurdle for developing precise and robust machine-learning models. Annotation is expensive, time-consuming, and often requires expert knowledge, particularly in the medical field. Here, we suggest using minimal user interaction in the form of extreme point clicks to train a segmentation model which, in effect, can be used to speed up medical image annotation. An initial segmentation is generated based on the extreme points using the random walker algorithm. This initial segmentation is then used as a noisy supervision signal to train a fully convolutional network that can segment the organ of interest, based on the provided user clicks. Through experimentation on several medical imaging datasets, we show that the predictions of the network can be refined using several rounds of training with the prediction from the same weakly annotated data. Further improvements are shown using the clicked points within a custom-designed loss and attention mechanism. Our approach has the potential to speed up the process of generating new training datasets for the development of new machine-learning and deep-learning-based models for, but not exclusively, medical image analysis.

Deep Learning-Based Automated Abdominal Organ Segmentation in the UK Biobank and German National Cohort Magnetic Resonance Imaging Studies

PURPOSE

The aims of this study were to train and evaluate deep learning models for automated segmentation of abdominal organs in whole-body magnetic resonance (MR) images from the UK Biobank (UKBB) and German National Cohort (GNC) MR imaging studies and to make these models available to the scientific community for analysis of these data sets.

METHODS

A total of 200 T1-weighted MR image data sets of healthy volunteers each from UKBB and GNC (400 data sets in total) were available in this study. Liver, spleen, left and right kidney, and pancreas were segmented manually on all 400 data sets, providing labeled ground truth data for training of a previously described U-Net-based deep learning framework for automated medical image segmentation (nnU-Net). The trained models were tested on all data sets using a 4-fold cross-validation scheme. Qualitative analysis of automated segmentation results was performed visually; performance metrics between automated and manual segmentation results were computed for quantitative analysis. In addition, interobserver segmentation variability between 2 human readers was assessed on a subset of the data.

RESULTS

Automated abdominal organ segmentation was performed with high qualitative and quantitative accuracy on UKBB and GNC data. In more than 90% of data sets, no or only minor visually detectable qualitative segmentation errors occurred. Mean Dice scores of automated segmentations compared with manual reference segmentations were well higher than 0.9 for the liver, spleen, and kidneys on UKBB and GNC data and around 0.82 and 0.89 for the pancreas on UKBB and GNC data, respectively. Mean average symmetric surface distance was between 0.3 and 1.5 mm for the liver, spleen, and kidneys and between 2 and 2.2 mm for pancreas segmentation. The quantitative accuracy of automated segmentation was comparable with the agreement between 2 human readers for all organs on UKBB and GNC data.

CONCLUSION

Automated segmentation of abdominal organs is possible with high qualitative and quantitative accuracy on whole-body MR imaging data acquired as part of UKBB and GNC. The results obtained and deep learning models trained in this study can be used as a foundation for automated analysis of thousands of MR data sets of UKBB and GNC and thus contribute to tackling topical and original scientific questions.

Cascaded Regression Neural Nets for Kidney Localization and Segmentation-free Volume Estimation

Kidney volume is an essential biomarker for a number of kidney disease diagnoses, for example, chronic kidney disease. Existing total kidney volume estimation methods often rely on an intermediate kidney segmentation step. On the other hand, automatic kidney localization in volumetric medical images is a critical step that often precedes subsequent data processing and analysis. Most current approaches perform kidney localization via an intermediate classification or regression step. This paper proposes an integrated deep learning approach for (i) kidney localization in computed tomography scans and (ii) segmentation-free renal volume estimation. Our localization method uses a selection-convolutional neural network that approximates the kidney inferior-superior span along the axial direction. Cross-sectional (2D) slices from the estimated span are subsequently used in a combined sagittal-axial Mask-RCNN that detects the organ bounding boxes on the axial and sagittal slices, the combination of which produces a final 3D organ bounding box. Furthermore, we use a fully convolutional network to estimate the kidney volume that skips the segmentation procedure. We also present a mathematical expression to approximate the 'volume error' metric from the 'Sørensen-Dice coefficient.' We accessed 100 patients' CT scans from the Vancouver General Hospital records and obtained 210 patients' CT scans from the 2019 Kidney Tumor Segmentation Challenge database to validate our method. Our method produces a kidney boundary wall localization error of ~2.4mm and a mean volume estimation error of ~5%.

3D Deep Learning for Anatomical Structure Segmentation in Multiple Imaging Modalities

Accurate, quantitative segmentation of anatomical structures in radiological scans, such as Magnetic Resonance Imaging (MRI) and Computer Tomography (CT), can produce significant biomarkers and can be integrated into computer-aided assisted diagnosis (CADx) systems to support the interpretation of medical images from multi-protocol scanners. However, there are serious challenges towards developing robust automated segmentation techniques, including high variations in anatomical structure and size, the presence of edge-based artefacts, and heavy un-controlled breathing that can produce blurred motion-based artefacts. This paper presents a novel computing approach for automatic organ and muscle segmentation in medical images from multiple modalities by harnessing the advantages of deep learning techniques in a two-part process. (1) a 3D encoder-decoder, Rb-UNet, builds a localisation model and a 3D Tiramisu network generates a boundary-preserving segmentation model for each target structure; (2) the fully trained Rb-UNet predicts a 3D bounding box encapsulating the target structure of interest, after which the fully trained Tiramisu model performs segmentation to reveal detailed organ or muscle boundaries. The proposed approach is evaluated on six different datasets, including MRI, Dynamic Contrast Enhanced (DCE) MRI and CT scans targeting the pancreas, liver, kidneys and psoas-muscle and achieves quantitative measures of mean Dice similarity coefficient (DSC) that surpass or are comparable with the state-of-the-art. A qualitative evaluation performed by two independent radiologists verified the preservation of detailed organ and muscle boundaries.

Artificial intelligence: a critical review of current applications in pancreatic imaging

The applications of artificial intelligence (AI), including machine learning and deep learning, in the field of pancreatic disease imaging are rapidly expanding. AI can be used for the detection of pancreatic ductal adenocarcinoma and other pancreatic tumors but also for pancreatic lesion characterization. In this review, the basic of radiomics, recent developments and current results of AI in the field of pancreatic tumors are presented. Limitations and future perspectives of AI are discussed.

3D IFPN: Improved Feature Pyramid Network for Automatic Segmentation of Gastric Tumor

Automatic segmentation of gastric tumor not only provides image-guided clinical diagnosis but also assists radiologists to read images and improve the diagnostic accuracy. However, due to the inhomogeneous intensity distribution of gastric tumors in CT scans, the ambiguous/missing boundaries, and the highly variable shapes of gastric tumors, it is quite challenging to develop an automatic solution. This study designs a novel 3D improved feature pyramidal network (3D IFPN) to automatically segment gastric tumors in computed tomography (CT) images. To meet the challenges of this extremely difficult task, the proposed 3D IFPN makes full use of the complementary information within the low and high layers of deep convolutional neural networks, which is equipped with three types of feature enhancement modules: 3D adaptive spatial feature fusion (ASFF) module, single-level feature refinement (SLFR) module, and multi-level feature refinement (MLFR) module. The 3D ASFF module adaptively suppresses the feature inconsistency in different levels and hence obtains the multi-level features with high feature invariance. Then, the SLFR module combines the adaptive features and previous multi-level features at each level to generate the multi-level refined features by skip connection and attention mechanism. The MLFR module adaptively recalibrates the channel-wise and spatial-wise responses by adding the attention operation, which improves the prediction capability of the network. Furthermore, a stage-wise deep supervision (SDS) mechanism and a hybrid loss function are also embedded to enhance the feature learning ability of the network. CT volumes dataset collected in three Chinese medical centers was used to evaluate the segmentation performance of the proposed 3D IFPN model. Experimental results indicate that our method outperforms state-of-the-art segmentation networks in gastric tumor segmentation. Moreover, to explore the generalization for other segmentation tasks, we also extend the proposed network to liver tumor segmentation in CT images of the MICCAI 2017 Liver Tumor Segmentation Challenge.

Iterative confidence relabeling with deep ConvNets for organ segmentation with partial labels

Training deep ConvNets requires large labeled datasets. However, collecting pixel-level labels for medical image segmentation is very expensive and requires a high level of expertise. In addition, most existing segmentation masks provided by clinical experts focus on specific anatomical structures. In this paper, we propose a method dedicated to handle such partially labeled medical image datasets. We propose a strategy to identify pixels for which labels are correct, and to train Fully Convolutional Neural Networks with a multi-label loss adapted to this context. In addition, we introduce an iterative confidence self-training approach inspired by curriculum learning to relabel missing pixel labels, which relies on selecting the most confident prediction with a specifically designed confidence network that learns an uncertainty measure which is leveraged in our relabeling process. Our approach, INERRANT for Iterative coNfidencE Relabeling of paRtial ANnoTations, is thoroughly evaluated on two public datasets (TCAI and LITS), and one internal dataset with seven abdominal organ classes. We show that INERRANT robustly deals with partial labels, performing similarly to a model trained on all labels even for large missing label proportions. We also highlight the importance of our iterative learning scheme and the proposed confidence measure for optimal performance. Finally we show a practical use case where a limited number of completely labeled data are enriched by publicly available but partially labeled data.

Comparison of metrics for the evaluation of medical segmentations using prostate MRI dataset

Nine previously proposed segmentation evaluation metrics, targeting medical relevance, accounting for holes, and added regions or differentiating over- and under-segmentation, were compared with 24 traditional metrics to identify those which better capture the requirements for clinical segmentation evaluation. Evaluation was first performed using 2D synthetic shapes to highlight features and pitfalls of the metrics with known ground truths (GTs) and machine segmentations (MSs). Clinical evaluation was then performed using publicly-available prostate images of 20 subjects with MSs generated by 3 different deep learning networks (DenseVNet, HighRes3DNet, and ScaleNet) and GTs drawn by 2 readers. The same readers also performed the 2D visual assessment of the MSs using a dual negative-positive grading of -5 to 5 to reflect over- and under-estimation. Nine metrics that correlated well with visual assessment were selected for further evaluation using 3 different network ranking methods - based on a single metric, normalizing the metric using 2 GTs, and ranking the network based on a metric then averaging, including leave-one-out evaluation. These metrics yielded consistent ranking with HighRes3DNet ranked first then DenseVNet and ScaleNet using all ranking methods. Relative volume difference yielded the best positivity-agreement and correlation with dual visual assessment, and thus is better for providing over- and under-estimation. Interclass Correlation yielded the strongest correlation with the absolute visual assessment (0-5). Symmetric-boundary dice consistently yielded good discrimination of the networks for all three ranking methods with relatively small variations within network. Good rank discrimination may be an additional metric feature required for better network performance evaluation.

Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks

Abdominal anatomy segmentation is crucial for numerous applications from computer-assisted diagnosis to image-guided surgery. In this context, we address fully-automated multi-organ segmentation from abdominal CT and MR images using deep learning. The proposed model extends standard conditional generative adversarial networks. Additionally to the discriminator which enforces the model to create realistic organ delineations, it embeds cascaded partially pre-trained convolutional encoder-decoders as generator. Encoder fine-tuning from a large amount of non-medical images alleviates data scarcity limitations. The network is trained end-to-end to benefit from simultaneous multi-level segmentation refinements using auto-context. Employed for healthy liver, kidneys and spleen segmentation, our pipeline provides promising results by outperforming state-of-the-art encoder-decoder schemes. Followed for the Combined Healthy Abdominal Organ Segmentation (CHAOS) challenge organized in conjunction with the IEEE International Symposium on Biomedical Imaging 2019, it gave us the first rank for three competition categories: liver CT, liver MR and multi-organ MR segmentation. Combining cascaded convolutional and adversarial networks strengthens the ability of deep learning pipelines to automatically delineate multiple abdominal organs, with good generalization capability. The comprehensive evaluation provided suggests that better guidance could be achieved to help clinicians in abdominal image interpretation and clinical decision making.

A deep learning-based auto-segmentation system for organs-at-risk on whole-body computed tomography images for radiation therapy

BACKGROUND AND PURPOSE

Delineating organs at risk (OARs) on computed tomography (CT) images is an essential step in radiation therapy; however, it is notoriously time-consuming and prone to inter-observer variation. Herein, we report a deep learning-based automatic segmentation (AS) algorithm (WBNet) that can accurately and efficiently delineate all major OARs in the entire body directly on CT scans.

MATERIALS AND METHODS

We collected 755 CT scans of the head and neck, thorax, abdomen, and pelvis and manually delineated 50 OARs on the CT images. The CT images with contours were split into training and test sets consisting of 505 and 250 cases, respectively, to develop and validate WBNet. The volumetric Dice similarity coefficient (DSC) and 95th-percentile Hausdorff distance (95% HD) were calculated to evaluate delineation quality for each OAR. We compared the performance of WBNet with three AS algorithms: one commercial multi-atlas-based automatic segmentation (ABAS) software, and two deep learning-based AS algorithms, namely, AnatomyNet and nnU-Net. We have also evaluated the time saving and dose accuracy of WBNet.

RESULTS

WBNet achieved average DSCs of 0.84 and 0.81 on in-house and public datasets, respectively, which outperformed ABAS, AnatomyNet, and nnU-Net. WBNet could reduce the delineation time significantly and perform well in treatment planning, with clinically acceptable dose differences compared with those in manual delineation.

CONCLUSION

This study shows the feasibility and benefits of using WBNet in clinical practice.

Fully Automatic Volume Measurement of the Spleen at CT Using Deep Learning

Purpose

To develop a fully automated algorithm for spleen segmentation and to assess the performance of this algorithm in a large dataset.

Materials and Methods

In this retrospective study, a three-dimensional deep learning network was developed to segment the spleen on thorax-abdomen CT scans. Scans were extracted from patients undergoing oncologic treatment from 2014 to 2017. A total of 1100 scans from 1100 patients were used in this study, and 400 were selected for development of the algorithm. For testing, a dataset of 50 scans was annotated to assess the segmentation accuracy and was compared against the splenic index equation. In a qualitative observer experiment, an enriched set of 100 scan-pairs was used to evaluate whether the algorithm could aid a radiologist in assessing splenic volume change. The reference standard was set by the consensus of two other independent radiologists. A Mann-Whitney U test was conducted to test whether there was a performance difference between the algorithm and the independent observer.

Results

The algorithm and the independent observer obtained comparable Dice scores (P = .834) on the test set of 50 scans of 0.962 and 0.964, respectively. The radiologist had an agreement with the reference standard in 81% (81 of 100) of the cases after a visual classification of volume change, which increased to 92% (92 of 100) when aided by the algorithm.

Conclusion

A segmentation method based on deep learning can accurately segment the spleen on CT scans and may help radiologists to detect abnormal splenic volumes and splenic volume changes.Supplemental material is available for this article.© RSNA, 2020.

A review of deep learning based methods for medical image multi-organ segmentation

Deep learning has revolutionized image processing and achieved the-state-of-art performance in many medical image segmentation tasks. Many deep learning-based methods have been published to segment different parts of the body for different medical applications. It is necessary to summarize the current state of development for deep learning in the field of medical image segmentation. In this paper, we aim to provide a comprehensive review with a focus on multi-organ image segmentation, which is crucial for radiotherapy where the tumor and organs-at-risk need to be contoured for treatment planning. We grouped the surveyed methods into two broad categories which are 'pixel-wise classification' and 'end-to-end segmentation'. Each category was divided into subgroups according to their network design. For each type, we listed the surveyed works, highlighted important contributions and identified specific challenges. Following the detailed review, we discussed the achievements, shortcomings and future potentials of each category. To enable direct comparison, we listed the performance of the surveyed works that used thoracic and head-and-neck benchmark datasets.

Comparison of image quality and lesion diagnosis in abdominopelvic unenhanced CT between reduced-dose CT using deep learning post-processing and standard-dose CT using iterative reconstruction: A prospective study

PURPOSE

To compare image quality and lesion diagnosis between reduced-dose abdominopelvic unenhanced computed tomography (CT) using deep learning (DL) post-processing and standard-dose CT using iterative reconstruction (IR).

METHOD

Totally 251 patients underwent two consecutive abdominopelvic unenhanced CT scans of the same range, including standard and reduced doses, respectively. In group A, standard-dose data were reconstructed by (blend 30 %) IR. In group B, reduced-dose data were reconstructed by filtered back projection reconstruction to obtain group B1 images, and post-processed using the DL algorithm (NeuAI denosing, Neusoft medical, Shenyang, China) with 50 % and 100 % weights to obtain group B2 and B3 images, respectively. Then, CT values of the liver, the second lumbar vertebral centrum, the erector spinae and abdominal subcutaneous fat were measured. CT values, noise levels, signal-to-noise ratios (SNRs), contrast-to-noise ratios (CNRs), radiation doses and subjective scores of image quality were compared. Subjective evaluations of low-density liver lesions were compared by diagnostic results from enhanced CT or Magnetic Resonance Imaging.

RESULTS

Groups B3 and B1 showed the lowest and highest noise levels, respectively (P < 0.001). The SNR and CNR in group B3 were highest (P < 0.001). The radiation dose in group B was reduced by 71.5 % on average compared to group A. Subjective scores in groups A and B2 were highest (P < 0.001). Diagnostic sensitivity and confidence for liver metastases in groups A and B2 were highest (P < 0.001).

CONCLUSIONS

Reduced-dose abdominopelvic unenhanced CT combined with DL post-processing could ensure image quality and satisfy diagnostic needs.

Liver tumor segmentation using 2.5D UV-Net with multi-scale convolution

Liver tumor segmentation networks are generally based on U-shaped encoder-decoder network with 2D or 3D structure. However, 2D networks lose the inter-layer information of continuous slices and 3D networks might introduce unacceptable parameters for GPU memory. As a result, 2.5D networks were proposed to balance the memory consumption and 3D context. Different from the canonical 2.5D design, which utilizes a 2D network combined with RNN, we propose a new 2.5D design called UV-Net to encode the inter-layer information in the context of 3D convolution, and reconstruct the high-resolution results with 2D deconvolution. At the same time, the multi-scale convolution structure enables multi-scale feature extraction without extra computational cost, which effectively mines structured information, reduces information redundancy, strengthens independent features, and makes feature dimension sparse, to enhance network capacity and efficiency. Combined with the proposed preprocessing method of removing mean energy, UV-Net significantly outperforms the existing methods in liver tumor segmentation and especially improves the segmentation accuracy of small objects on the LiTS2017 dataset.

Deep learning to segment pelvic bones: large-scale CT datasets and baseline models

PURPOSE

Pelvic bone segmentation in CT has always been an essential step in clinical diagnosis and surgery planning of pelvic bone diseases. Existing methods for pelvic bone segmentation are either hand-crafted or semi-automatic and achieve limited accuracy when dealing with image appearance variations due to the multi-site domain shift, the presence of contrasted vessels, coprolith and chyme, bone fractures, low dose, metal artifacts, etc. Due to the lack of a large-scale pelvic CT dataset with annotations, deep learning methods are not fully explored.

METHODS

In this paper, we aim to bridge the data gap by curating a large pelvic CT dataset pooled from multiple sources, including 1184 CT volumes with a variety of appearance variations. Then, we propose for the first time, to the best of our knowledge, to learn a deep multi-class network for segmenting lumbar spine, sacrum, left hip, and right hip, from multiple-domain images simultaneously to obtain more effective and robust feature representations. Finally, we introduce a post-processor based on the signed distance function (SDF).

RESULTS

Extensive experiments on our dataset demonstrate the effectiveness of our automatic method, achieving an average Dice of 0.987 for a metal-free volume. SDF post-processor yields a decrease of 15.1% in Hausdorff distance compared with traditional post-processor.

CONCLUSION

We believe this large-scale dataset will promote the development of the whole community and open source the images, annotations, codes, and trained baseline models at https://github.com/ICT-MIRACLE-lab/CTPelvic1K .

Efficiently compressing 3D medical images for teleinterventions via CNNs and anisotropic diffusion

PURPOSE

Efficient compression of images while preserving image quality has the potential to be a major enabler of effective remote clinical diagnosis and treatment, since poor Internet connection conditions are often the primary constraint in such services. This paper presents a framework for organ-specific image compression for teleinterventions based on a deep learning approach and anisotropic diffusion filter.

METHODS

The proposed method, deep learning and anisotropic diffusion (DLAD), uses a convolutional neural network architecture to extract a probability map for the organ of interest; this probability map guides an anisotropic diffusion filter that smooths the image except at the location of the organ of interest. Subsequently, a compression method, such as BZ2 and HEVC-visually lossless, is applied to compress the image. We demonstrate the proposed method on three-dimensional (3D) CT images acquired for radio frequency ablation (RFA) of liver lesions. We quantitatively evaluate the proposed method on 151 CT images using peak-signal-to-noise ratio ( PSNR ), structural similarity ( SSIM ), and compression ratio ( CR ) metrics. Finally, we compare the assessments of two radiologists on the liver lesion detection and the liver lesion center annotation using 33 sets of the original images and the compressed images.

RESULTS

The results show that the method can significantly improve CR of most well-known compression methods. DLAD combined with HEVC-visually lossless achieves the highest average CR of 6.45, which is 36% higher than that of the original HEVC and outperforms other state-of-the-art lossless medical image compression methods. The means of PSNR and SSIM are 70 dB and 0.95, respectively. In addition, the compression effects do not statistically significantly affect the assessments of the radiologists on the liver lesion detection and the lesion center annotation.

CONCLUSIONS

We thus conclude that the method has a high potential to be applied in teleintervention applications.

Three-dimensional image volumes from two-dimensional digitally reconstructed radiographs: A deep learning approach in lower limb CT scans

PURPOSE

Three-dimensional (3D) reconstructions of the human anatomy have been available for surgery planning or diagnostic purposes for a few years now. The different image modalities usually rely on several consecutive two-dimensional (2D) acquisitions in order to reconstruct the 3D volume. Hence, such acquisitions are expensive, time-demanding and often expose the patient to an undesirable amount of radiation. For such reasons, along the most recent years, several studies have been proposed that extrapolate 3D anatomical features from merely 2D exams such as x rays for implant templating in total knee or hip arthroplasties.

METHOD

The presented study shows an adaptation of a deep learning-based convolutional neural network to reconstruct 3D volumes from a mere 2D digitally reconstructed radiograph from one of the most extensive lower limb computed tomography datasets available. This novel approach is based on an encoder-decoder architecture with skip connections and a multidimensional Gaussian filter as data augmentation technique.

RESULTS

The results achieved promising values when compared against the ground truth volumes, quantitatively yielding an average of 0.77 ± 0.05 structured similarity index.

CONCLUSIONS

A novel deep learning-based approach to reconstruct 3D medical image volumes from a single x-ray image was shown in the present study. The network architecture was validated against the original scans presenting SSIM values of 0.77 ± 0.05 and 0.78 ± 0.06, respectively for the knee and the hip crop.

Loss odyssey in medical image segmentation

The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing. In this paper, we present a comprehensive review of segmentation loss functions in an organized manner. We also conduct the first large-scale analysis of 20 general loss functions on four typical 3D segmentation tasks involving six public datasets from 10+ medical centers. The results show that none of the losses can consistently achieve the best performance on the four segmentation tasks, but compound loss functions (e.g. Dice with TopK loss, focal loss, Hausdorff distance loss, and boundary loss) are the most robust losses. Our code and segmentation results are publicly available and can serve as a loss function benchmark. We hope this work will also provide insights on new loss function development for the community.

Automated approach for segmenting gross tumor volumes for lung cancer stereotactic body radiation therapy using CT-based dense V-networks

The aim of this study was to develop an automated segmentation approach for small gross tumor volumes (GTVs) in 3D planning computed tomography (CT) images using dense V-networks (DVNs) that offer more advantages in segmenting smaller structures than conventional V-networks. Regions of interest (ROI) with dimensions of 50 × 50 × 6-72 pixels in the planning CT images were cropped based on the GTV centroids when applying stereotactic body radiotherapy (SBRT) to patients. Segmentation accuracy of GTV contours for 192 lung cancer patients [with the following tumor types: 118 solid, 53 part-solid types and 21 pure ground-glass opacity (pure GGO)], who underwent SBRT, were evaluated based on a 10-fold cross-validation test using Dice's similarity coefficient (DSC) and Hausdorff distance (HD). For each case, 11 segmented GTVs consisting of three single outputs, four logical AND outputs, and four logical OR outputs from combinations of two or three outputs from DVNs were obtained by three runs with different initial weights. The AND output (combination of three outputs) achieved the highest values of average 3D-DSC (0.832 ± 0.074) and HD (4.57 ± 2.44 mm). The average 3D DSCs from the AND output for solid, part-solid and pure GGO types were 0.838 ± 0.074, 0.822 ± 0.078 and 0.819 ± 0.059, respectively. This study suggests that the proposed approach could be useful in segmenting GTVs for planning lung cancer SBRT.

CT based automatic clinical target volume delineation using a dense-fully connected convolution network for cervical Cancer radiation therapy

BACKGROUND

It is very important to accurately delineate the CTV on the patient's three-dimensional CT image in the radiotherapy process. Limited to the scarcity of clinical samples and the difficulty of automatic delineation, the research of automatic delineation of cervical cancer CTV based on CT images for new patients is slow. This study aimed to assess the value of Dense-Fully Connected Convolution Network (Dense V-Net) in predicting Clinical Target Volume (CTV) pre-delineation in cervical cancer patients for radiotherapy.

METHODS

In this study, we used Dense V-Net, a dense and fully connected convolutional network with suitable feature learning in small samples to automatically pre-delineate the CTV of cervical cancer patients based on computed tomography (CT) images and then we assessed the outcome. The CT data of 133 patients with stage IB and IIA postoperative cervical cancer with a comparable delineation scope was enrolled in this study. One hundred and thirteen patients were randomly designated as the training set to adjust the model parameters. Twenty cases were used as the test set to assess the network performance. The 8 most representative parameters were also used to assess the pre-sketching accuracy from 3 aspects: sketching similarity, sketching offset, and sketching volume difference.

RESULTS

The results presented that the DSC, DC/mm, HD/cm, MAD/mm, ∆V, SI, IncI and JD of CTV were 0.82 ± 0.03, 4.28 ± 2.35, 1.86 ± 0.48, 2.52 ± 0.40, 0.09 ± 0.05, 0.84 ± 0.04, 0.80 ± 0.05, and 0.30 ± 0.04, respectively, and the results were greater than those with a single network.

CONCLUSIONS

Dense V-Net can correctly predict CTV pre-delineation of cervical cancer patients and can be applied in clinical practice after completing simple modifications.

Anterior Mediastinal Lesion Segmentation Based on Two-Stage 3D ResUNet With Attention Gates and Lung Segmentation

OBJECTIVES

Anterior mediastinal disease is a common disease in the chest. Computed tomography (CT), as an important imaging technology, is widely used in the diagnosis of mediastinal diseases. Doctors find it difficult to distinguish lesions in CT images because of image artifact, intensity inhomogeneity, and their similarity with other tissues. Direct segmentation of lesions can provide doctors a method to better subtract the features of the lesions, thereby improving the accuracy of diagnosis.

METHOD

As the trend of image processing technology, deep learning is more accurate in image segmentation than traditional methods. We employ a two-stage 3D ResUNet network combined with lung segmentation to segment CT images. Given that the mediastinum is between the two lungs, the original image is clipped through the lung mask to remove some noises that may affect the segmentation of the lesion. To capture the feature of the lesions, we design a two-stage network structure. In the first stage, the features of the lesion are learned from the low-resolution downsampled image, and the segmentation results under a rough scale are obtained. The results are concatenated with the original image and encoded into the second stage to capture more accurate segmentation information from the image. In addition, attention gates are introduced in the upsampling of the network, and these gates can focus on the lesion and play a role in filtering the features. The proposed method has achieved good results in the segmentation of the anterior mediastinal.

RESULTS

The proposed method was verified on 230 patients, and the anterior mediastinal lesions were well segmented. The average Dice coefficient reached 87.73%. Compared with the model without lung segmentation, the model with lung segmentation greatly improved the accuracy of lesion segmentation by approximately 9%. The addition of attention gates slightly improved the segmentation accuracy.

CONCLUSION

The proposed automatic segmentation method has achieved good results in clinical data. In clinical application, automatic segmentation of lesions can assist doctors in the diagnosis of diseases and may facilitate the automated diagnosis of illnesses in the future.

Marginal loss and exclusion loss for partially supervised multi-organ segmentation

Annotating multiple organs in medical images is both costly and time-consuming; therefore, existing multi-organ datasets with labels are often low in sample size and mostly partially labeled, that is, a dataset has a few organs labeled but not all organs. In this paper, we investigate how to learn a single multi-organ segmentation network from a union of such datasets. To this end, we propose two types of novel loss function, particularly designed for this scenario: (i) marginal loss and (ii) exclusion loss. Because the background label for a partially labeled image is, in fact, a 'merged' label of all unlabelled organs and 'true' background (in the sense of full labels), the probability of this 'merged' background label is a marginal probability, summing the relevant probabilities before merging. This marginal probability can be plugged into any existing loss function (such as cross entropy loss, Dice loss, etc.) to form a marginal loss. Leveraging the fact that the organs are non-overlapping, we propose the exclusion loss to gauge the dissimilarity between labeled organs and the estimated segmentation of unlabelled organs. Experiments on a union of five benchmark datasets in multi-organ segmentation of liver, spleen, left and right kidneys, and pancreas demonstrate that using our newly proposed loss functions brings a conspicuous performance improvement for state-of-the-art methods without introducing any extra computation.

Automatic segmentation of pulmonary lobes on low-dose computed tomography using deep learning

Background

To develop and validate a fully automated deep learning-based segmentation algorithm to segment pulmonary lobe on low-dose computed tomography (LDCT) images.

Methods

This study presents an automatic segmentation of pulmonary lobes using a fully convolutional neural network named dense V-network (DenseVNet) on lung cancer screening LDCT images. A total of 160 LDCT cases for lung cancer screening (100 in the training set, 10 in the validation set, and 50 in the test set) was included in this study. Specifically, the template of pulmonary lobes (the right lung consists of three lobes, and the left lung consists of two lobes) were obtained from pixel-level annotations by semiautomatic segmentation platform. Then, the model was trained under the supervision of the LDCT training set. Finally, the trained model was used to segment the LDCT in the test set. Dice coefficient, Jaccard coefficient, and Hausdorff distance were adopted as evaluation metrics to verify the performance of our segmentation model.

Results

In this study, the model achieved the accurate segmentation of each pulmonary lobe in seconds without the intervention of researchers. The testing set consisted 50 LDCT cases were used to evaluate the performance of the segmentation model. The all-lobes Dice coefficient of the test set was 0.944, the Jaccard coefficient was 0.896, and the Hausdorff distance was 92.908 mm.

Conclusions

The segmentation model based on LDCT can automatically and robustly and efficiently segment pulmonary lobes. It will provide effective location information and contour constraints for pulmonary nodule detection on LDCT images for lung cancer screening, which may have potential clinical application.

Automated Pipeline to Generate Anatomically Accurate Patient-Specific Biomechanical Models of Healthy and Pathological FSUs

State-of-the-art preoperative biomechanical analysis for the planning of spinal surgery not only requires the generation of three-dimensional patient-specific models but also the accurate biomechanical representation of vertebral joints. The benefits offered by computational models suitable for such purposes are still outweighed by the time and effort required for their generation, thus compromising their applicability in a clinical environment. In this work, we aim to ease the integration of computerized methods into patient-specific planning of spinal surgery. We present the first pipeline combining deep learning and finite element methods that allows a completely automated model generation of functional spine units (FSUs) of the lumbar spine for patient-specific FE simulations (FEBio). The pipeline consists of three steps: (a) multiclass segmentation of cropped 3D CT images containing lumbar vertebrae using the DenseVNet network, (b) automatic landmark-based mesh fitting of statistical shape models onto 3D semantic segmented meshes of the vertebral models, and (c) automatic generation of patient-specific FE models of lumbar segments for the simulation of flexion-extension, lateral bending, and axial rotation movements. The automatic segmentation of FSUs was evaluated against the gold standard (manual segmentation) using 10-fold cross-validation. The obtained Dice coefficient was 93.7% on average, with a mean surface distance of 0.88 mm and a mean Hausdorff distance of 11.16 mm (N = 150). Automatic generation of finite element models to simulate the range of motion (ROM) was successfully performed for five healthy and five pathological FSUs. The results of the simulations were evaluated against the literature and showed comparable ROMs in both healthy and pathological cases, including the alteration of ROM typically observed in severely degenerated FSUs. The major intent of this work is to automate the creation of anatomically accurate patient-specific models by a single pipeline allowing functional modeling of spinal motion in healthy and pathological FSUs. Our approach reduces manual efforts to a minimum and the execution of the entire pipeline including simulations takes approximately 2 h. The automation, time-efficiency and robustness level of the pipeline represents a first step toward its clinical integration.

Liver segmentation in abdominal CT images via auto-context neural network and self-supervised contour attention

OBJECTIVE

Accurate image segmentation of the liver is a challenging problem owing to its large shape variability and unclear boundaries. Although the applications of fully convolutional neural networks (CNNs) have shown groundbreaking results, limited studies have focused on the performance of generalization. In this study, we introduce a CNN for liver segmentation on abdominal computed tomography (CT) images that focus on the performance of generalization and accuracy.

METHODS

To improve the generalization performance, we initially propose an auto-context algorithm in a single CNN. The proposed auto-context neural network exploits an effective high-level residual estimation to obtain the shape prior. Identical dual paths are effectively trained to represent mutual complementary features for an accurate posterior analysis of a liver. Further, we extend our network by employing a self-supervised contour scheme. We trained sparse contour features by penalizing the ground-truth contour to focus more contour attentions on the failures.

RESULTS

We used 180 abdominal CT images for training and validation. Two-fold cross-validation is presented for a comparison with the state-of-the-art neural networks. The experimental results show that the proposed network results in better accuracy when compared to the state-of-the-art networks by reducing 10.31% of the Hausdorff distance. Novel multiple N-fold cross-validations are conducted to show the best performance of generalization of the proposed network.

CONCLUSION AND SIGNIFICANCE

The proposed method minimized the error between training and test images more than any other modern neural networks. Moreover, the contour scheme was successfully employed in the network by introducing a self-supervising metric.

Artificial intelligence in radiology

The interest in artificial intelligence (AI) has ballooned within radiology in the past few years primarily due to notable successes of deep learning. With the advances brought by deep learning, AI has the potential to recognize and localize complex patterns from different radiological imaging modalities, many of which even achieve comparable performance to human decision-making in recent applications. In this chapter, we review several AI applications in radiology for different anatomies: chest, abdomen, pelvis, as well as general lesion detection/identification that is not limited to specific anatomies. For each anatomy site, we focus on introducing the tasks of detection, segmentation, and classification with an emphasis on describing the technology development pathway with the aim of providing the reader with an understanding of what AI can do in radiology and what still needs to be done for AI to better fit in radiology. Combining with our own research experience of AI in medicine, we elaborate how AI can enrich knowledge discovery, understanding, and decision-making in radiology, rather than replacing the radiologist.

Models Genesis

Transfer learning from natural image to medical image has been established as one of the most practical paradigms in deep learning for medical image analysis. To fit this paradigm, however, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information, thereby inevitably compromising its performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learnt by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch and existing pre-trained 3D models in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D/2.5D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated and recurrent anatomy in medical images can serve as strong yet free supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all codes and pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Fully end-to-end deep-learning-based diagnosis of pancreatic tumors

Artificial intelligence can facilitate clinical decision making by considering massive amounts of medical imaging data. Various algorithms have been implemented for different clinical applications. Accurate diagnosis and treatment require reliable and interpretable data. For pancreatic tumor diagnosis, only 58.5% of images from the First Affiliated Hospital and the Second Affiliated Hospital, Zhejiang University School of Medicine are used, increasing labor and time costs to manually filter out images not directly used by the diagnostic model.

Methods: This study used a training dataset of 143,945 dynamic contrast-enhanced CT images of the abdomen from 319 patients. The proposed model contained four stages: image screening, pancreas location, pancreas segmentation, and pancreatic tumor diagnosis.

Results: We established a fully end-to-end deep-learning model for diagnosing pancreatic tumors and proposing treatment. The model considers original abdominal CT images without any manual preprocessing. Our artificial-intelligence-based system achieved an area under the curve of 0.871 and a F1 score of 88.5% using an independent testing dataset containing 107,036 clinical CT images from 347 patients. The average accuracy for all tumor types was 82.7%, and the independent accuracies of identifying intraductal papillary mucinous neoplasm and pancreatic ductal adenocarcinoma were 100% and 87.6%, respectively. The average test time per patient was 18.6 s, compared with at least 8 min for manual reviewing. Furthermore, the model provided a transparent and interpretable diagnosis by producing saliency maps highlighting the regions relevant to its decision.

Conclusions: The proposed model can potentially deliver efficient and accurate preoperative diagnoses that could aid the surgical management of pancreatic tumor.

[Artificial Intelligence in Radiology - Definition, Potential and Challenges]

Artificial Intelligence in Radiology - Definition, Potential and Challenges Abstract. Artificial Intelligence (AI) is omnipresent. It has neatly permeated our daily life, even if we are not always fully aware of its ubiquitous presence. The healthcare sector in particular is experiencing a revolution which will change our daily routine considerably in the near future. Due to its advanced digitization and its historical technical affinity radiology is especially prone to these developments. But what exactly is AI and what makes AI so potent that established medical disciplines such as radiology worry about their future job perspectives? What are the assets of AI in radiology today - and what are the major challenges? This review article tries to give some answers to these questions.

Multi-stage learning for segmentation of aortic dissections using a prior aortic anatomy simplification

Aortic dissection (AD) is a life-threatening cardiovascular disease with a high mortality rate. The accurate and generalized 3-D reconstruction of AD from CT-angiography can effectively assist clinical procedures and surgery plans, however, is clinically unavaliable due to the lacking of efficient tools. In this study, we presented a novel multi-stage segmentation framework for type B AD to extract true lumen (TL), false lumen (FL) and all branches (BR) as different classes. Two cascaded neural networks were used to segment the aortic trunk and branches and to separate the dual lumen, respectively. An aortic straightening method was designed based on the prior vascular anatomy of AD, simplifying the curved aortic shape before the second network. The straightening-based method achieved the mean Dice scores of 0.96, 0.95 and 0.89 for TL, FL, and BR on a multi-center dataset involving 120 patients, outperforming the end-to-end multi-class methods and the multi-stage methods without straightening on the dual-lumen segmentation, even using different network architectures. Both the global volumetric features of the aorta and the local characteristics of the primary tear could be better identified and quantified based on the straightening. Comparing to previous deep learning methods dealing with AD segmentations, the proposed framework presented advantages in segmentation accuracy.

PWD-3DNet: A Deep Learning-Based Fully-Automated Segmentation of Multiple Structures on Temporal Bone CT Scans

The temporal bone is a part of the lateral skull surface that contains organs responsible for hearing and balance. Mastering surgery of the temporal bone is challenging because of this complex and microscopic three-dimensional anatomy. Segmentation of intra-temporal anatomy based on computed tomography (CT) images is necessary for applications such as surgical training and rehearsal, amongst others. However, temporal bone segmentation is challenging due to the similar intensities and complicated anatomical relationships among critical structures, undetectable small structures on standard clinical CT, and the amount of time required for manual segmentation. This paper describes a single multi-class deep learning-based pipeline as the first fully automated algorithm for segmenting multiple temporal bone structures from CT volumes, including the sigmoid sinus, facial nerve, inner ear, malleus, incus, stapes, internal carotid artery and internal auditory canal. The proposed fully convolutional network, PWD-3DNet, is a patch-wise densely connected (PWD) three-dimensional (3D) network. The accuracy and speed of the proposed algorithm was shown to surpass current manual and semi-automated segmentation techniques. The experimental results yielded significantly high Dice similarity scores and low Hausdorff distances for all temporal bone structures with an average of 86% and 0.755 millimeter (mm), respectively. We illustrated that overlapping in the inference sub-volumes improves the segmentation performance. Moreover, we proposed augmentation layers by using samples with various transformations and image artefacts to increase the robustness of PWD-3DNet against image acquisition protocols, such as smoothing caused by soft tissue scanner settings and larger voxel sizes used for radiation reduction. The proposed algorithm was tested on low-resolution CTs acquired by another center with different scanner parameters than the ones used to create the algorithm and shows potential for application beyond the particular training data used in the study.

[Volume Measurements of Post-transplanted Liver of Pediatric Recipients Using Workstations and Deep Learning]

PURPOSE

The purpose of this study was to propose a method for segmentation and volume measurement of graft liver and spleen of pediatric transplant recipients on digital imaging and communications in medicine (DICOM) -format images using U-Net and three-dimensional (3-D) workstations (3DWS) .

METHOD

For segmentation accuracy assessments, Dice coefficients were calculated for the graft liver and spleen. After verifying that the created DICOM-format images could be imported using the existing 3DWS, accuracy rates between the ground truth and segmentation images were calculated via mask processing.

RESULT

As per the verification results, Dice coefficients for the test data were as follows: graft liver, 0.758 and spleen, 0.577. All created DICOM-format images were importable using the 3DWS, with accuracy rates of 87.10±4.70% and 80.27±11.29% for the graft liver and spleen, respectively.

CONCLUSION

The U-Net could be used for graft liver and spleen segmentations, and volume measurement using 3DWS was simplified by this method.

CT-ORG, a new dataset for multiple organ segmentation in computed tomography

Despite the relative ease of locating organs in the human body, automated organ segmentation has been hindered by the scarcity of labeled training data. Due to the tedium of labeling organ boundaries, most datasets are limited to either a small number of cases or a single organ. Furthermore, many are restricted to specific imaging conditions unrepresentative of clinical practice. To address this need, we developed a diverse dataset of 140 CT scans containing six organ classes: liver, lungs, bladder, kidney, bones and brain. For the lungs and bones, we expedited annotation using unsupervised morphological segmentation algorithms, which were accelerated by 3D Fourier transforms. Demonstrating the utility of the data, we trained a deep neural network which requires only 4.3 s to simultaneously segment all the organs in a case. We also show how to efficiently augment the data to improve model generalization, providing a GPU library for doing so. We hope this dataset and code, available through TCIA, will be useful for training and evaluating organ segmentation models.

Multi-Organ Segmentation Over Partially Labeled Datasets With Multi-Scale Feature Abstraction

Shortage of fully annotated datasets has been a limiting factor in developing deep learning based image segmentation algorithms and the problem becomes more pronounced in multi-organ segmentation. In this paper, we propose a unified training strategy that enables a novel multi-scale deep neural network to be trained on multiple partially labeled datasets for multi-organ segmentation. In addition, a new network architecture for multi-scale feature abstraction is proposed to integrate pyramid input and feature analysis into a U-shape pyramid structure. To bridge the semantic gap caused by directly merging features from different scales, an equal convolutional depth mechanism is introduced. Furthermore, we employ a deep supervision mechanism to refine the outputs in different scales. To fully leverage the segmentation features from all the scales, we design an adaptive weighting layer to fuse the outputs in an automatic fashion. All these mechanisms together are integrated into a Pyramid Input Pyramid Output Feature Abstraction Network (PIPO-FAN). Our proposed method was evaluated on four publicly available datasets, including BTCV, LiTS, KiTS and Spleen, where very promising performance has been achieved. The source code of this work is publicly shared at https://github.com/DIAL-RPI/PIPO-FAN to facilitate others to reproduce the work and build their own models using the introduced mechanisms.

CS2-Net: Deep learning segmentation of curvilinear structures in medical imaging

Automated detection of curvilinear structures, e.g., blood vessels or nerve fibres, from medical and biomedical images is a crucial early step in automatic image interpretation associated to the management of many diseases. Precise measurement of the morphological changes of these curvilinear organ structures informs clinicians for understanding the mechanism, diagnosis, and treatment of e.g. cardiovascular, kidney, eye, lung, and neurological conditions. In this work, we propose a generic and unified convolution neural network for the segmentation of curvilinear structures and illustrate in several 2D/3D medical imaging modalities. We introduce a new curvilinear structure segmentation network (CS²-Net), which includes a self-attention mechanism in the encoder and decoder to learn rich hierarchical representations of curvilinear structures. Two types of attention modules - spatial attention and channel attention - are utilized to enhance the inter-class discrimination and intra-class responsiveness, to further integrate local features with their global dependencies and normalization, adaptively. Furthermore, to facilitate the segmentation of curvilinear structures in medical images, we employ a 1×3 and a 3×1 convolutional kernel to capture boundary features. Besides, we extend the 2D attention mechanism to 3D to enhance the network's ability to aggregate depth information across different layers/slices. The proposed curvilinear structure segmentation network is thoroughly validated using both 2D and 3D images across six different imaging modalities. Experimental results across nine datasets show the proposed method generally outperforms other state-of-the-art algorithms in various metrics.

Evaluation of Multimodal Algorithms for the Segmentation of Multiparametric MRI Prostate Images

Prostate segmentation in multiparametric magnetic resonance imaging (mpMRI) can help to support prostate cancer diagnosis and therapy treatment. However, manual segmentation of the prostate is subjective and time-consuming. Many deep learning monomodal networks have been developed for automatic whole prostate segmentation from T2-weighted MR images. We aimed to investigate the added value of multimodal networks in segmenting the prostate into the peripheral zone (PZ) and central gland (CG). We optimized and evaluated monomodal DenseVNet, multimodal ScaleNet, and monomodal and multimodal HighRes3DNet, which yielded dice score coefficients (DSC) of 0.875, 0.848, 0.858, and 0.890 in WG, respectively. Multimodal HighRes3DNet and ScaleNet yielded higher DSC with statistical differences in PZ and CG only compared to monomodal DenseVNet, indicating that multimodal networks added value by generating better segmentation between PZ and CG regions but did not improve the WG segmentation. No significant difference was observed in the apex and base of WG segmentation between monomodal and multimodal networks, indicating that the segmentations at the apex and base were more affected by the general network architecture. The number of training data was also varied for DenseVNet and HighRes3DNet, from 20 to 120 in steps of 20. DenseVNet was able to yield DSC of higher than 0.65 even for special cases, such as TURP or abnormal prostate, whereas HighRes3DNet's performance fluctuated with no trend despite being the best network overall. Multimodal networks did not add value in segmenting special cases but generally reduced variations in segmentation compared to the same matched monomodal network.

Predicting Chemotherapeutic Response for Far-advanced Gastric Cancer by Radiomics with Deep Learning Semi-automatic Segmentation

Purpose: To build a dual-energy computed tomography (DECT) delta radiomics model to predict chemotherapeutic response for far-advanced gastric cancer (GC) patients. A semi-automatic segmentation method based on deep learning was designed, and its performance was compared with that of manual segmentation. Methods: This retrospective study included 86 patients with far-advanced GC treated with chemotherapy from September 2016 to December 2017 (66 and 20 in the training and testing cohorts, respectively). Delta radiomics features between the baseline and first follow-up DECT were modeled by random forest to predict the chemotherapeutic response evaluated by the second follow-up DECT. Nine feature subsets from confounding factors and delta radiomics features were used to choose the best model with 10-fold cross-validation in the training cohort. A semi-automatic segmentation method based on deep learning was developed to predict the chemotherapeutic response and compared with manual segmentation in the testing cohort, which was further validated in an independent validation cohort of 30 patients. Results: The best model, constructed by confounding factors and texture features, reached an average AUC of 0.752 in the training cohort. Our proposed semi-automatic segmentation method was more time-effective than manual segmentation, with average saving-time of 11.2333 ± 6.3989 minutes and 9.9889 ±5.5086 minutes in the testing cohort and the independent validation cohort, respectively (both p < 0.05). The predictive ability of the semi-automatic segmentation was also better than that of the manual segmentation both in the testing cohort and the independent validation cohort (AUC: 0.728 vs. 0.687 and 0.828 vs. 0.749, respectively). Conclusion: DECT delta radiomics serves as a promising biomarker for predicting chemotherapeutic response for far-advanced GC. Semi-automatic segmentation based on deep learning shows the potential for clinical use with increased reproducibility and decreased labor costs compared to the manual version.

Pairwise learning for medical image segmentation

Fully convolutional networks (FCNs) trained with abundant labeled data have been proven to be a powerful and efficient solution for medical image segmentation. However, FCNs often fail to achieve satisfactory results due to the lack of labelled data and significant variability of appearance in medical imaging. To address this challenging issue, this paper proposes a conjugate fully convolutional network (CFCN) where pairwise samples are input for capturing a rich context representation and guide each other with a fusion module. To avoid the overfitting problem introduced by intra-class heterogeneity and boundary ambiguity with a small number of training samples, we propose to explicitly exploit the prior information from the label space, termed as proxy supervision. We further extend the CFCN to a compact conjugate fully convolutional network (C²FCN), which just has one head for fitting the proxy supervision without incurring two additional branches of decoders fitting ground truth of the input pairs compared to CFCN. In the test phase, the segmentation probability is inferred by the learned logical relation implied in the proxy supervision. Quantitative evaluation on the Liver Tumor Segmentation (LiTS) and Combined (CT-MR) Healthy Abdominal Organ Segmentation (CHAOS) datasets shows that the proposed framework achieves a significant performance improvement on both binary segmentation and multi-category segmentation, especially with a limited amount of training data. The source code is available at https://github.com/renzhenwang/pairwise_segmentation.

Potentials and caveats of AI in hybrid imaging

State-of-the-art patient management frequently mandates the investigation of both anatomy and physiology of the patients. Hybrid imaging modalities such as the PET/MRI, PET/CT and SPECT/CT have the ability to provide both structural and functional information of the investigated tissues in a single examination. With the introduction of such advanced hardware fusion, new problems arise such as the exceedingly large amount of multi-modality data that requires novel approaches of how to extract a maximum of clinical information from large sets of multi-dimensional imaging data. Artificial intelligence (AI) has emerged as one of the leading technologies that has shown promise in facilitating highly integrative analysis of multi-parametric data. Specifically, the usefulness of AI algorithms in the medical imaging field has been heavily investigated in the realms of (1) image acquisition and reconstruction, (2) post-processing and (3) data mining and modelling. Here, we aim to provide an overview of the challenges encountered in hybrid imaging and discuss how AI algorithms can facilitate potential solutions. In addition, we highlight the pitfalls and challenges in using advanced AI algorithms in the context of hybrid imaging and provide suggestions for building robust AI solutions that enable reproducible and transparent research.

Complete abdomen and pelvis segmentation using U-net variant architecture

PURPOSE

Organ segmentation of computed tomography (CT) imaging is essential for radiotherapy treatment planning. Treatment planning requires segmentation not only of the affected tissue, but nearby healthy organs-at-risk, which is laborious and time-consuming. We present a fully automated segmentation method based on the three-dimensional (3D) U-Net convolutional neural network (CNN) capable of whole abdomen and pelvis segmentation into 33 unique organ and tissue structures, including tissues that may be overlooked by other automated segmentation approaches such as adipose tissue, skeletal muscle, and connective tissue and vessels. Whole abdomen segmentation is capable of quantifying exposure beyond a handful of organs-at-risk to all tissues within the abdomen.

METHODS

Sixty-six (66) CT examinations of 64 individuals were included in the training and validation sets and 18 CT examinations from 16 individuals were included in the test set. All pixels in each examination were segmented by image analysts (with physician correction) and assigned one of 33 labels. Segmentation was performed with a 3D U-Net variant architecture which included residual blocks, and model performance was quantified on 18 test cases. Human interobserver variability (using semiautomated segmentation) was also reported on two scans, and manual interobserver variability of three individuals was reported on one scan. Model performance was also compared to several of the best models reported in the literature for multiple organ segmentation.

RESULTS

The accuracy of the 3D U-Net model ranges from a Dice coefficient of 0.95 in the liver, 0.93 in the kidneys, 0.79 in the pancreas, 0.69 in the adrenals, and 0.51 in the renal arteries. Model accuracy is within 5% of human segmentation in eight of 19 organs and within 10% accuracy in 13 of 19 organs.

CONCLUSIONS

The CNN approaches the accuracy of human tracers and on certain complex organs displays more consistent prediction than human tracers. Fully automated deep learning-based segmentation of CT abdomen has the potential to improve both the speed and accuracy of radiotherapy dose prediction for organs-at-risk.

Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering. We demonstrate the usefulness of this architecture on the example of 2D/3D registration between radiographs and CT scans. Specifically, we show that our transformer enables end-to-end learning of an image processing and projection model that approximates an image similarity function that is convex with respect to the pose parameters, and can thus be optimized effectively using conventional gradient descent. To the best of our knowledge, we are the first to describe the spatial transformers in the context of projective transmission imaging, including rendering and pose estimation. We hope that our developments will benefit related 3D research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

Deep Q-CapsNet Reinforcement Learning Framework for Intrauterine Cavity Segmentation in TTTS Fetal Surgery Planning

Fetoscopic laser photocoagulation is the most effective treatment for Twin-to-Twin Transfusion Syndrome, a condition affecting twin pregnancies in which there is a deregulation of blood circulation through the placenta, that can be fatal to both babies. For the purposes of surgical planning, we design the first automatic approach to detect and segment the intrauterine cavity from axial, sagittal and coronal MRI stacks. Our methodology relies on the ability of capsule networks to successfully capture the part-whole interdependency of objects in the scene, particularly for unique class instances (i.e., intrauterine cavity). The presented deep Q-CapsNet reinforcement learning framework is built upon a context-adaptive detection policy to generate a bounding box of the womb. A capsule architecture is subsequently designed to segment (or refine) the whole intrauterine cavity. This network is coupled with a strided nnU-Net feature extractor, which encodes discriminative feature maps to construct strong primary capsules. The method is robustly evaluated with and without the localization stage using 13 performance measures, and directly compared with 15 state-of-the-art deep neural networks trained on 71 singleton and monochorionic twin pregnancies. An average Dice score above 0.91 is achieved for all ablations, revealing the potential of our approach to be used in clinical practice.