Regression forests for efficient anatomy detection and localization in computed tomography scans

Multi-task global optimization-based method for vascular landmark detection

Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.

Feature Aggregation and Refinement Network for 2D Anatomical Landmark Detection

Localization of anatomical landmarks is essential for clinical diagnosis, treatment planning, and research. This paper proposes a novel deep network named feature aggregation and refinement network (FARNet) for automatically detecting anatomical landmarks. FARNet employs an encoder-decoder structure architecture. To alleviate the problem of limited training data in the medical domain, we adopt a backbone network pre-trained on natural images as the encoder. The decoder includes a multi-scale feature aggregation module for multi-scale feature fusion and a feature refinement module for high-resolution heatmap regression. Coarse-to-fine supervisions are applied to the two modules to facilitate end-to-end training. We further propose a novel loss function named Exponential Weighted Center loss for accurate heatmap regression, which focuses on the losses from the pixels near landmarks and suppresses the ones from far away. We evaluate FARNet on three publicly available anatomical landmark detection datasets, including cephalometric, hand, and spine radiographs. Our network achieves state-of-the-art performances on all three datasets. Code is available at https://github.com/JuvenileInWind/FARNet .

Using a New Deep Learning Method for 3D Cephalometry in Patients With Cleft Lip and Palate

Deep learning algorithms based on automatic 3-dimensional (D) cephalometric marking points about people without craniomaxillofacial deformities has achieved good results. However, there has been no previous report about cleft lip and palate. The purpose of this study is to apply a new deep learning method based on a 3D point cloud graph convolutional neural network to predict and locate landmarks in patients with cleft lip and palate based on the relationships between points. The authors used the PointNet++ model to investigate the automatic 3D cephalometric marking points. And the mean distance error of the center coordinate position and the success detection rate (SDR) were used to evaluate the accuracy of systematic labeling. A total of 150 patients were enrolled. The mean distance error for all 27 landmarks was 1.33 mm, and 9 landmarks (30%) showed SDRs at 2 mm over 90%, and 3 landmarks (35%) showed SDRs at 2 mm under 70%. The automatic 3D cephalometric marking points take 16 seconds per dataset. In summary, our training sets were derived from the cleft lip with/without palate computed tomography to achieve accurate results. The 3D cephalometry system based on the graph convolutional neural network algorithm may be suitable for 3D cephalometry system in cleft lip and palate cases. More accurate results may be obtained if the cleft lip and palate training set is expanded in the future.

Combining natural and artificial intelligence for robust automatic anatomy segmentation: Application in neck and thorax auto-contouring

BACKGROUND

Automatic segmentation of 3D objects in computed tomography (CT) is challenging. Current methods, based mainly on artificial intelligence (AI) and end-to-end deep learning (DL) networks, are weak in garnering high-level anatomic information, which leads to compromised efficiency and robustness. This can be overcome by incorporating natural intelligence (NI) into AI methods via computational models of human anatomic knowledge.

PURPOSE

We formulate a hybrid intelligence (HI) approach that integrates the complementary strengths of NI and AI for organ segmentation in CT images and illustrate performance in the application of radiation therapy (RT) planning via multisite clinical evaluation.

METHODS

The system employs five modules: (i) body region recognition, which automatically trims a given image to a precisely defined target body region; (ii) NI-based automatic anatomy recognition object recognition (AAR-R), which performs object recognition in the trimmed image without DL and outputs a localized fuzzy model for each object; (iii) DL-based recognition (DL-R), which refines the coarse recognition results of AAR-R and outputs a stack of 2D bounding boxes (BBs) for each object; (iv) model morphing (MM), which deforms the AAR-R fuzzy model of each object guided by the BBs output by DL-R; and (v) DL-based delineation (DL-D), which employs the object containment information provided by MM to delineate each object. NI from (ii), AI from (i), (iii), and (v), and their combination from (iv) facilitate the HI system.

RESULTS

The HI system was tested on 26 organs in neck and thorax body regions on CT images obtained prospectively from 464 patients in a study involving four RT centers. Data sets from one separate independent institution involving 125 patients were employed in training/model building for each of the two body regions, whereas 104 and 110 data sets from the 4 RT centers were utilized for testing on neck and thorax, respectively. In the testing data sets, 83% of the images had limitations such as streak artifacts, poor contrast, shape distortion, pathology, or implants. The contours output by the HI system were compared to contours drawn in clinical practice at the four RT centers by utilizing an independently established ground-truth set of contours as reference. Three sets of measures were employed: accuracy via Dice coefficient (DC) and Hausdorff boundary distance (HD), subjective clinical acceptability via a blinded reader study, and efficiency by measuring human time saved in contouring by the HI system. Overall, the HI system achieved a mean DC of 0.78 and 0.87 and a mean HD of 2.22 and 4.53 mm for neck and thorax, respectively. It significantly outperformed clinical contouring in accuracy and saved overall 70% of human time over clinical contouring time, whereas acceptability scores varied significantly from site to site for both auto-contours and clinically drawn contours.

CONCLUSIONS

The HI system is observed to behave like an expert human in robustness in the contouring task but vastly more efficiently. It seems to use NI help where image information alone will not suffice to decide, first for the correct localization of the object and then for the precise delineation of the boundary.

Localization of Craniomaxillofacial Landmarks on CBCT Images Using 3D Mask R-CNN and Local Dependency Learning

Cephalometric analysis relies on accurate detection of craniomaxillofacial (CMF) landmarks from cone-beam computed tomography (CBCT) images. However, due to the complexity of CMF bony structures, it is difficult to localize landmarks efficiently and accurately. In this paper, we propose a deep learning framework to tackle this challenge by jointly digitalizing 105 CMF landmarks on CBCT images. By explicitly learning the local geometrical relationships between the landmarks, our approach extends Mask R-CNN for end-to-end prediction of landmark locations. Specifically, we first apply a detection network on a down-sampled 3D image to leverage global contextual information to predict the approximate locations of the landmarks. We subsequently leverage local information provided by higher-resolution image patches to refine the landmark locations. On patients with varying non-syndromic jaw deformities, our method achieves an average detection accuracy of 1.38± 0.95mm, outperforming a related state-of-the-art method.

Machine learning to predict dynamic changes of pathogenic Vibrio spp. abundance on microplastics in marine environment

Microplastics are widely found in the marine environment. Recent studies have shown that pathogenic microorganisms can hitchhike on microplastics, which might act as a vector for the spread of pathogens. Vibrio spp. are known to be pathogenic to humans and can cause serious foodborne diseases. In this study, using datasets from an estuary and a mariculture zone in China, five machine learning models were established to predict the relative abundance of Vibrio spp. on microplastics. The results showed that deep neural network (DNN) model and RandomForest algorithm achieved the best predictive performance. Different data sources, data sampling, and processing methods had a little impact on the prediction performance of DNN and RandomForest models. SHapley Additive exPlanations (SHAP) indicated that salinity and temperature are the primary factors affecting the relative abundance of Vibrio spp. The prediction performances of the five machine learning models were further improved by feature selection, providing information to support future experimental research. The results of this study could help establish a long-term and dynamic monitoring system for the relative abundance of Vibrio spp. on microplastics in response to environmental factors as well as provide useful information for assessing the potential health impacts of microplastics on marine ecology and humans.

Supervised and semi-supervised 3D organ localisation in CT images combining reinforcement learning with imitation learning

Computer aided diagnostics often requires analysis of a region of interest (ROI) within a radiology scan, and the ROI may be an organ or a suborgan. Although deep learning algorithms have the ability to outperform other methods, they rely on the availability of a large amount of annotated data. Motivated by the need to address this limitation, an approach to localisation and detection of multiple organs based on supervised and semi-supervised learning is presented here. It draws upon previous work by the authors on localising the thoracic and lumbar spine region in CT images. The method generates six bounding boxes of organs of interest, which are then fused to a single bounding box. The results of experiments on localisation of the Spleen, Left and Right Kidneys in CT Images using supervised and semi supervised learning (SSL) demonstrate the ability to address data limitations with a much smaller data set and fewer annotations, compared to other state-of-the-art methods. The SSL performance was evaluated using three different mixes of labelled and unlabelled data (i.e. 30:70,35:65,40:60) for each of lumbar spine, spleen left and right kidneys respectively. The results indicate that SSL provides a workable alternative especially in medical imaging where it is difficult to obtain annotated data.

Automatic scan range for dose-reduced multiphase CT imaging of the liver utilizing CNNs and Gaussian models

Multiphase CT scanning of the liver is performed for several clinical applications; however, radiation exposure from CT scanning poses a nontrivial cancer risk to the patients. The radiation dose may be reduced by determining the scan range of the subsequent scans by the location of the target of interest in the first scan phase. The purpose of this study is to present and assess an automatic method for determining the scan range for multiphase CT scans. Our strategy is to first apply a CNN-based method for detecting the liver in 2D slices, and to use a liver range search algorithm for detecting the liver range in the scout volume. The target liver scan range for subsequent scans can be obtained by adding safety margins achieved from Gaussian liver motion models to the scan range determined from the scout. Experiments were performed on 657 multiphase CT volumes obtained from multiple hospitals. The experiment shows that the proposed liver detection method can detect the liver in 223 out of a total of 224 3D volumes on average within one second, with mean intersection of union, wall distance and centroid distance of 85.5%, 5.7 mm and 9.7 mm, respectively. In addition, the performance of the proposed liver detection method is comparable to the best of the state-of-the-art 3D liver detectors in the liver detection accuracy while it requires less processing time. Furthermore, we apply the liver scan range generation method on the liver CT images acquired from radiofrequency ablation and Y-90 transarterial radioembolization (selective internal radiation therapy) interventions of 46 patients from two hospitals. The result shows that the automatic scan range generation can significantly reduce the effective radiation dose by an average of 14.5% (2.56 mSv) compared to manual performance by the radiographer from Y-90 transarterial radioembolization, while no statistically significant difference in performance was found with the CT images from intra RFA intervention (p = 0.81). Finally, three radiologists assess both the original and the range-reduced images for evaluating the effect of the range reduction method on their clinical decisions. We conclude that the automatic liver scan range generation method is able to reduce excess radiation compared to the manual performance with a high accuracy and without penalizing the clinical decision.

Deep learning-based body part recognition algorithm for three-dimensional medical images

BACKGROUND

The automatic recognition of human body parts in three-dimensional (3D) medical images is important in many clinical applications. However, methods presented in prior studies have mainly classified each two-dimensional (2D) slice independently rather than recognizing a batch of consecutive slices as a specific body part.

PURPOSE

In this study, we aim to develop a deep-learning-based method designed to automatically divide computed tomography (CT) and magnetic resonance imaging (MRI) scans into five consecutive body parts: head, neck, chest, abdomen, and pelvis.

METHODS

A deep learning framework was developed to recognize body parts in two stages. In the first pre-classification stage, a convolutional neural network (CNN) using the GoogLeNet Inception v3 architecture and a long short-term memory (LSTM) network were combined to classify each 2D slice; the CNN extracted information from a single slice, whereas the LSTM employed rich contextual information among consecutive slices. In the second post-processing stage, the input scan was further partitioned into consecutive body parts by identifying the optimal boundaries between them based on the slice classification results of the first stage. To evaluate the performance of the proposed method, 662 CT and 1434 MRI scans were used.

RESULTS

Our method achieved a very good performance in 2D slice classification compared with state-of-the-art methods, with overall classification accuracies of 97.3% and 98.2% for CT and MRI scans, respectively. Moreover, our method further divided whole scans into consecutive body parts with mean boundary errors of 8.9 mm and 3.5 mm for CT and MRI data, respectively.

CONCLUSIONS

The proposed method significantly improved the slice classification accuracy compared with state-of-the-art methods, and further accurately divided CT and MRI scans into consecutive body parts based on the results of slice classification. The developed method can be employed as an important step in various computer-aided diagnosis and medical image analysis schemes. This article is protected by copyright. All rights reserved.

An Anatomy-based Iteratively Searching Convolutional Neural Network for Organ Localization in CT images

Organ localization is a common and essential preprocessing operation for many medical image analysis tasks. We propose a novel multi-organ localization method based on an end-to-end 3D convolutional neural network. The proposed algorithm employs a regression network to learn the position relationship between any patch and target organs in a medical computed tomography (CT) image. With this framework, it can iteratively localize the target organs in a coarse-to-fine manner. The main idea behind this method is to embed the anatomy of structures in a deep learning-based approach. For implementation, the proposed network outputs an 8-dimensional vector that contains information about the position, scale, and presence of each target organ. A piecewise loss function and a multi-density sampling strategy help to optimize this network to learn anatomy layout characteristics over the entire CT image. Starting from a random position, this network can accurately locate the target organ with a few iterations. Moreover, a dual-resolution strategy is employed to improve the accuracy affected by varying organ scales, further enhancing the localizing performance for all organs. We evaluate our method on a public data set (LiTS) to locate 11 organs in the thoraco-abdomino-pelvic region. The proposed method outperforms state-of-the-art methods with a mean intersection over union (IOU) of 80.84%, mean wall distance of 3.63 mm, and mean centroid distance of 4.93 mm, constituting excellent accuracy. The improvements on relatively small-size and medium-size organs are noteworthy.

Fast and Accurate Craniomaxillofacial Landmark Detection via 3D Faster R-CNN

Automatic craniomaxillofacial (CMF) landmark localization from cone-beam computed tomography (CBCT) images is challenging, considering that 1) the number of landmarks in the images may change due to varying deformities and traumatic defects, and 2) the CBCT images used in clinical practice are typically large. In this paper, we propose a two-stage, coarse-to-fine deep learning method to tackle these challenges with both speed and accuracy in mind. Specifically, we first use a 3D faster R-CNN to roughly locate landmarks in down-sampled CBCT images that have varying numbers of landmarks. By converting the landmark point detection problem to a generic object detection problem, our 3D faster R-CNN is formulated to detect virtual, fixed-size objects in small boxes with centers indicating the approximate locations of the landmarks. Based on the rough landmark locations, we then crop 3D patches from the high-resolution images and send them to a multi-scale UNet for the regression of heatmaps, from which the refined landmark locations are finally derived. We evaluated the proposed approach by detecting up to 18 landmarks on a real clinical dataset of CMF CBCT images with various conditions. Experiments show that our approach achieves state-of-the-art accuracy of 0.89 ± 0.64mm in an average time of 26.2 seconds per volume.

On the use of machine learning methods for mPSD calibration in HDR brachytherapy

We sought to evaluate the feasibility of using machine learning (ML) algorithms for multipoint plastic scintillator detector (mPSD) calibration in high-dose-rate (HDR) brachytherapy. Dose measurements were conducted under HDR brachytherapy conditions. The dosimetry system consisted of an optimized 1-mm-core mPSD and a compact assembly of photomultiplier tubes coupled with dichroic mirrors and filters. An ¹⁹²Ir source was remotely controlled and sent to various positions in a homemade PMMA holder, ensuring 0.1-mm positional accuracy. Dose measurements covering a range of 0.5 to 12 cm of source displacement were carried out according to TG-43 U1 recommendations. Individual scintillator doses were decoupled using a linear regression model, a random forest estimator, and artificial neural network algorithms. The dose predicted by the TG-43U1 formalism was used as the reference for system calibration and ML algorithm training. The performance of the different algorithms was evaluated using different sample sizes and distances to the source for the mPSD system calibration. We found that the calibration conditions influenced the accuracy in predicting the measured dose. The decoupling methods' deviations from the expected TG-43 U1 dose generally remained below 20%. However, the dose prediction with the three algorithms was accurate to within 7% relative to the dose predicted by the TG-43 U1 formalism when measurements were performed in the same range of distances used for calibration. In such cases, the predictions with random forest exhibited minimal deviations (<2%). However, the performance random forest was compromised when the predictions were done beyond the range of distances used for calibration. Because the linear regression algorithm can extrapolate the data, the dose prediction by the linear regression was less influenced by the calibration conditions than random forest. The linear regression algorithm's behavior along the distances to the source was smoother than those for the random forest and neural network algorithms, but the observed deviations were more significant than those for the neural network and random forest algorithms. The number of available measurements for training purposes influenced the random forest and neural network models the most. Their accuracy tended to converge toward deviation values close to 1% from a number of dwell positions greater than 100. In performing HDR brachytherapy dose measurements with an optimized mPSD system, ML algorithms are good alternatives for precise dose reporting and treatment assessment during this kind of cancer treatment.

Cascaded Regression Neural Nets for Kidney Localization and Segmentation-free Volume Estimation

Kidney volume is an essential biomarker for a number of kidney disease diagnoses, for example, chronic kidney disease. Existing total kidney volume estimation methods often rely on an intermediate kidney segmentation step. On the other hand, automatic kidney localization in volumetric medical images is a critical step that often precedes subsequent data processing and analysis. Most current approaches perform kidney localization via an intermediate classification or regression step. This paper proposes an integrated deep learning approach for (i) kidney localization in computed tomography scans and (ii) segmentation-free renal volume estimation. Our localization method uses a selection-convolutional neural network that approximates the kidney inferior-superior span along the axial direction. Cross-sectional (2D) slices from the estimated span are subsequently used in a combined sagittal-axial Mask-RCNN that detects the organ bounding boxes on the axial and sagittal slices, the combination of which produces a final 3D organ bounding box. Furthermore, we use a fully convolutional network to estimate the kidney volume that skips the segmentation procedure. We also present a mathematical expression to approximate the 'volume error' metric from the 'Sørensen-Dice coefficient.' We accessed 100 patients' CT scans from the Vancouver General Hospital records and obtained 210 patients' CT scans from the 2019 Kidney Tumor Segmentation Challenge database to validate our method. Our method produces a kidney boundary wall localization error of ~2.4mm and a mean volume estimation error of ~5%.

A novel segmentation method for cervical vertebrae based on PointNet++ and converge segmentation

BACKGROUND

Cervical spine instability is the key pathogenic factor for cervical spondylosis, which may easily cause cervical spinal cord nerve compression, numbness, weakness, and even paralysis of the limbs. The reconstruction of the internal fixation of the cervical spine is of great therapeutic significance, but is a high-risk and difficult procedure that requires precise planning. The high similarities between vertebrae may interfere with automatic operation planning; therefore, the segmentation of vertebrae is of great significance.

METHODS

Our segmentation algorithm has 3 parts. Firstly, an adaptive threshold filter to segment the cervical vertebra tissue structure form CT images. Secondly, segmentation of single vertebrae based on PointNet++ is introduced to segmentation cervical spine. Finally, converge segmentation which is based on edge information is utilized to clearly distinguish the edges of the two vertebrae to enhance the accuracy segmentation result.

RESULTS

Our approach improved the accuracy of the system up to 96.15%, and achieved the highest reported average score based on this dataset. We compared the results of the CNN and PointNet methods on a separate dataset of 240 CT scans with 18 classes and achieved a significantly higher performance for any given vertebra. Our experiments illustrated the promise and robustness of recent PointNet++-based segmentation of medical images.

CONCLUSION

The proposed method has better classification performance for segmentation cervical spine images, which segment a three-dimensional vertebral body directly and effectively. Furthermore, the precise segmentation of a single vertebral body can be used in automatic biomechanical analysis, computer-aided diagnosis and other aspects, so as to improve the level of automation in the treatment of cervical spondylosis.

A pilot study of radiomics signature based on biparametric MRI for preoperative prediction of extrathyroidal extension in papillary thyroid carcinoma

OBJECTIVE

To investigate efficiency of radiomics signature to preoperatively predict histological features of aggressive extrathyroidal extension (ETE) in papillary thyroid carcinoma (PTC) with biparametric magnetic resonance imaging findings.

MATERIALS AND METHODS

Sixty PTC patients with preoperative MR including T2WI and T2WI-fat-suppression (T2WI-FS) were retrospectively analyzed. Among them, 35 had ETE and 25 did not. Pre-contrast T2WI and T2WI-FS images depicting the largest section of tumor were selected. Tumor regions were manually segmented using ITK-SNAP software and 107 radiomics features were computed from the segmented regions using the open Pyradiomics package. Then, a random forest model was built to do classification in which the datasets were partitioned randomly 10 times to do training and testing with ratio of 1:1. Furthermore, forward greedy feature selection based on feature importance was adopted to reduce model overfitting. Classification accuracy was estimated on the test set using area under ROC curve (AUC).

RESULTS

The model using T2WI-FS image features yields much higher performance than the model using T2WI features (AUC = 0.906 vs. 0.760 using 107 features). Among the top 10 important features of T2WI and T2WI-FS, there are 5 common features. After feature selection, the models trained using top 2 features of T2WI and the top 6 features of T2WI-FS achieve AUC 0.845 and 0.928, respectively. Combining features computed from T2WI and T2WI-FS, model performance decreases slightly (AUC = 0.882 based on all features and AUC = 0.913 based on top features after feature selection). Adjusting hyper parameters of the random forest model have negligible influence on the model performance with mean AUC = 0.907 for T2WI-FS images.

CONCLUSIONS

Radiomics features based on pre-contrast T2WI and T2WI-FS is helpful to predict aggressive ETE in PTC. Particularly, the model trained using the optimally selected T2WI-FS image features yields the best classification performance. The most important features relate to lesion size and the texture heterogeneity of the tumor region.

Deep Q-CapsNet Reinforcement Learning Framework for Intrauterine Cavity Segmentation in TTTS Fetal Surgery Planning

Fetoscopic laser photocoagulation is the most effective treatment for Twin-to-Twin Transfusion Syndrome, a condition affecting twin pregnancies in which there is a deregulation of blood circulation through the placenta, that can be fatal to both babies. For the purposes of surgical planning, we design the first automatic approach to detect and segment the intrauterine cavity from axial, sagittal and coronal MRI stacks. Our methodology relies on the ability of capsule networks to successfully capture the part-whole interdependency of objects in the scene, particularly for unique class instances (i.e., intrauterine cavity). The presented deep Q-CapsNet reinforcement learning framework is built upon a context-adaptive detection policy to generate a bounding box of the womb. A capsule architecture is subsequently designed to segment (or refine) the whole intrauterine cavity. This network is coupled with a strided nnU-Net feature extractor, which encodes discriminative feature maps to construct strong primary capsules. The method is robustly evaluated with and without the localization stage using 13 performance measures, and directly compared with 15 state-of-the-art deep neural networks trained on 71 singleton and monochorionic twin pregnancies. An average Dice score above 0.91 is achieved for all ablations, revealing the potential of our approach to be used in clinical practice.

BRR-Net: A tandem architectural CNN-RNN for automatic body region localization in CT images

PURPOSE

Automatic identification of consistently defined body regions in medical images is vital in many applications. In this paper, we describe a method to automatically demarcate the superior and inferior boundaries for neck, thorax, abdomen, and pelvis body regions in computed tomography (CT) images.

METHODS

For any three-dimensional (3D) CT image I, following precise anatomic definitions, we denote the superior and inferior axial boundary slices of the neck, thorax, abdomen, and pelvis body regions by NS(I), NI(I), TS(I), TI(I), AS(I), AI(I), PS(I), and PI(I), respectively. Of these, by definition, AI(I) = PS(I), and so the problem reduces to demarcating seven body region boundaries. Our method consists of a two-step approach. In the first step, a convolutional neural network (CNN) is trained to classify each axial slice in I into one of nine categories: the seven body region boundaries, plus legs (defined as all axial slices inferior to PI(I)), and the none-of-the-above category. This CNN uses a multichannel approach to exploit the interslice contrast, providing the neural network with additional visual context at the body region boundaries. In the second step, to improve the predictions for body region boundaries that are very subtle and that exhibit low contrast, a recurrent neural network (RNN) is trained on features extracted by CNN, limited to a flexible window about the predictions from the CNN.

RESULTS

The method is evaluated on low-dose CT images from 442 patient scans, divided into training and testing sets with a ratio of 70:30. Using only the CNN, overall absolute localization error for NS(I), NI(I), TS(I), TI(I), AS(I), AI(I), and PI(I) expressed in terms of number of slices (nS) is (mean ± SD): 0.61 ± 0.58, 1.05 ± 1.13, 0.31 ± 0.46, 1.85 ± 1.96, 0.57 ± 2.44, 3.42 ± 3.16, and 0.50 ± 0.50, respectively. Using the RNN to refine the CNN's predictions for select classes improved the accuracy of TI(I) and AI(I) to: 1.35 ± 1.71 and 2.83 ± 2.75, respectively. This model outperforms the results achieved in our previous work by 2.4, 1.7, 3.1, 1.1, and 2 slices, respectively for TS(I), TI(I), AS(I), AI(I) = PS(I), and PI(I) classes with statistical significance. The model trained on low-dose CT images was also tested on diagnostic CT images for NS(I), NI(I), and TS(I) classes; the resulting errors were: 1.48 ± 1.33, 2.56 ± 2.05, and 0.58 ± 0.71, respectively.

CONCLUSIONS

Standardized body region definitions are a prerequisite for effective implementation of quantitative radiology, but the literature is severely lacking in the precise identification of body regions. The method presented in this paper significantly outperforms earlier works by a large margin, and the deviations of our results from ground truth are comparable to variations observed in manual labeling by experts. The solution presented in this work is critical to the adoption and employment of the idea of standardized body regions, and clears the path for development of applications requiring accurate demarcations of body regions. The work is indispensable for automatic anatomy recognition, delineation, and contouring for radiation therapy planning, as it not only automates an essential part of the process, but also removes the dependency on experts for accurately demarcating body regions in a study.

Augmented reality for inner ear procedures: visualization of the cochlear central axis in microscopic videos

PURPOSE

Visualization of the cochlea is impossible due to the delicate and intricate ear anatomy. Augmented reality may be used to perform auditory nerve implantation by transmodiolar approach in patients with profound hearing loss.

METHODS

We present an augmented reality system for the visualization of the cochlear axis in surgical videos. The system starts with an automatic anatomical landmark detection in preoperative computed tomography images based on deep reinforcement learning. These landmarks are used to register the preoperative geometry with the real-time microscopic video captured inside the auditory canal. Three-dimensional pose of the cochlear axis is determined using the registration projection matrices. In addition, the patient microscope movements are tracked using an image feature-based tracking process.

RESULTS

The landmark detection stage yielded an average localization error of [Formula: see text] mm ([Formula: see text]). The target registration error was [Formula: see text] mm for the cochlear apex and [Formula: see text] for the cochlear axis.

CONCLUSION

We developed an augmented reality system to visualize the cochlear axis in intraoperative videos. The system yielded millimetric accuracy and remained stable throughout the experimental study despite camera movements throughout the procedure in experimental conditions.

Semi-supervised labelling of the femur in a whole-body post-mortem CT database using deep learning

A deep learning pipeline was developed and used to localize and classify a variety of implants in the femur contained in whole-body post-mortem computed tomography (PMCT) scans. The results provide a proof-of-principle approach for labelling content not described in medical/autopsy reports. The pipeline, which incorporated residual networks and an autoencoder, was trained and tested using n = 450 full-body PMCT scans. For the localization component, Dice scores of 0.99, 0.96, and 0.98 and mean absolute errors of 3.2, 7.1, and 4.2 mm were obtained in the axial, coronal, and sagittal views, respectively. A regression analysis found the orientation of the implant to the scanner axis and also the relative positioning of extremities to be statistically significant factors. For the classification component, test cases were properly labelled as nail (N⁺), hip replacement (H⁺), knee replacement (K⁺) or without-implant (I^-) with an accuracy >97%. The recall for I^- and H⁺ cases was 1.00, but fell to 0.82 and 0.65 for cases with K⁺ and N⁺. This semi-automatic approach provides a generalized structure for image-based labelling of features, without requiring time-consuming segmentation.

HeadLocNet: Deep convolutional neural networks for accurate classification and multi-landmark localization of head CTs

Cochlear implants (CIs) are used to treat subjects with hearing loss. In a CI surgery, an electrode array is inserted into the cochlea to stimulate auditory nerves. After surgery, CIs need to be programmed. Studies have shown that the cochlea-electrode spatial relationship derived from medical images can guide CI programming and lead to significant improvement in hearing outcomes. We have developed a series of algorithms to segment the inner ear anatomy and localize the electrodes. But, because clinical head CT images are acquired with different protocols, the field of view and orientation of the image volumes vary greatly. As a consequence, visual inspection and manual image registration to an atlas image are needed to document their content and to initialize intensity-based registration algorithms used in our processing pipeline. For large-scale evaluation and deployment of our methods these steps need to be automated. In this article we propose to achieve this with a deep convolutional neural network (CNN) that can be trained end-to-end to classify a head CT image in terms of its content and to localize landmarks. The detected landmarks can then be used to estimate a point-based registration with the atlas image in which the same landmark set's positions are known. We achieve 99.5% classification accuracy and an average localization error of 3.45 mm for 7 landmarks located around each inner ear. This is better than what was achieved with earlier methods we have proposed for the same tasks.

Partial Policy-Based Reinforcement Learning for Anatomical Landmark Localization in 3D Medical Images

Utilizing the idea of long-term cumulative return, reinforcement learning (RL) has shown remarkable performance in various fields. We follow the formulation of landmark localization in 3D medical images as an RL problem. Whereas value-based methods have been widely used to solve RL-based localization problems, we adopt an actor-critic based direct policy search method framed in a temporal difference learning approach. In RL problems with large state and/or action spaces, learning the optimal behavior is challenging and requires many trials. To improve the learning, we introduce a partial policy-based reinforcement learning to enable solving the large problem of localization by learning the optimal policy on smaller partial domains. Independent actors efficiently learn the corresponding partial policies, each utilizing their own independent critic. The proposed policy reconstruction from the partial policies ensures a robust and efficient localization, where the sub-agents uniformly contribute to the state-transitions based on their simple partial policies mapping to binary actions. Experiments with three different localization problems in 3D CT and MR images showed that the proposed reinforcement learning requires a significantly smaller number of trials to learn the optimal behavior compared to the original behavior learning scheme in RL. It also ensures a satisfactory performance when trained on fewer images.

Anatomically consistent CNN-based segmentation of organs-at-risk in cranial radiotherapy

Planning of radiotherapy involves accurate segmentation of a large number of organs at risk (OAR), i.e., organs for which irradiation doses should be minimized to avoid important side effects of the therapy. We propose a deep learning method for segmentation of OAR inside the head, from magnetic resonance images (MRIs). Our system performs segmentation of eight structures: eye, lens, optic nerve, optic chiasm, pituitary gland, hippocampus, brainstem, and brain. We propose an efficient algorithm to train neural networks for an end-to-end segmentation of multiple and nonexclusive classes, addressing problems related to computational costs and missing ground truth segmentations for a subset of classes. We enforce anatomical consistency of the result in a postprocessing step. In particular, we introduce a graph-based algorithm for segmentation of the optic nerves, enforcing the connectivity between the eyes and the optic chiasm. We report cross-validated quantitative results on a database of 44 contrast-enhanced T1-weighted MRIs with provided segmentations of the considered OAR, which were originally used for radiotherapy planning. In addition, the segmentations produced by our model on an independent test set of 50 MRIs were evaluated by an experienced radiotherapist in order to qualitatively assess their accuracy. The mean distances between produced segmentations and the ground truth ranged from 0.1 to 0.7 mm across different organs. A vast majority (96%) of the produced segmentations were found acceptable for radiotherapy planning.

Context-guided fully convolutional networks for joint craniomaxillofacial bone segmentation and landmark digitization

Cone-beam computed tomography (CBCT) scans are commonly used in diagnosing and planning surgical or orthodontic treatment to correct craniomaxillofacial (CMF) deformities. Based on CBCT images, it is clinically essential to generate an accurate 3D model of CMF structures (e.g., midface, and mandible) and digitize anatomical landmarks. This process often involves two tasks, i.e., bone segmentation and anatomical landmark digitization. Because landmarks usually lie on the boundaries of segmented bone regions, the tasks of bone segmentation and landmark digitization could be highly associated. Also, the spatial context information (e.g., displacements from voxels to landmarks) in CBCT images is intuitively important for accurately indicating the spatial association between voxels and landmarks. However, most of the existing studies simply treat bone segmentation and landmark digitization as two standalone tasks without considering their inherent relationship, and rarely take advantage of the spatial context information contained in CBCT images. To address these issues, we propose a Joint bone Segmentation and landmark Digitization (JSD) framework via context-guided fully convolutional networks (FCNs). Specifically, we first utilize displacement maps to model the spatial context information in CBCT images, where each element in the displacement map denotes the displacement from a voxel to a particular landmark. An FCN is learned to construct the mapping from the input image to its corresponding displacement maps. Using the learned displacement maps as guidance, we further develop a multi-task FCN model to perform bone segmentation and landmark digitization jointly. We validate the proposed JSD method on 107 subjects, and the experimental results demonstrate that our method is superior to the state-of-the-art approaches in both tasks of bone segmentation and landmark digitization.

Automated age estimation from MRI volumes of the hand

Highly relevant for both clinical and legal medicine applications, the established radiological methods for estimating unknown age in children and adolescents are based on visual examination of bone ossification in X-ray images of the hand. Our group has initiated the development of fully automatic age estimation methods from 3D MRI scans of the hand, in order to simultaneously overcome the problems of the radiological methods including (1) exposure to ionizing radiation, (2) necessity to define new, MRI specific staging systems, and (3) subjective influence of the examiner. The present work provides a theoretical background for understanding the nonlinear regression problem of biological age estimation and chronological age approximation. Based on this theoretical background, we comprehensively evaluate machine learning methods (random forests, deep convolutional neural networks) with different simplifications of the image information used as an input for learning. Trained on a large dataset of 328 MR images, we compare the performance of the different input strategies and demonstrate unprecedented results. For estimating biological age, we obtain a mean absolute error of 0.37 ± 0.51 years for the age range of the subjects  ≤  18 years, i.e. where bone ossification has not yet saturated. Finally, we validate our findings by adapting our best performing method to 2D images and applying it to a publicly available dataset of X-ray images, showing that we are in line with the state-of-the-art automatic methods for this task.

Liver Segmentation in Abdominal CT Images Using Probabilistic Atlas and Adaptive 3D Region Growing

Automatic liver segmentation plays a vital role in computer-aided diagnosis or treatment. Manual segmentation of organs is a tedious and challenging task and is prone to human errors. In this paper, we propose innovative pre-processing and adaptive 3D region growing methods with subject-specific conditions. To obtain strong edges and high contrast, we propose effective contrast enhancement algorithm then we use the atlas intensity distribution of most probable voxels in probability maps along with location before designing conditions for our 3D region growing method. We also incorporate the organ boundary to restrict the region growing. We compare our method with the label fusion of 13 organs on state-of-the-art Deeds registration method and achieved Dice score of 92.56%.

Learning-Based Quality Control for Cardiac MR Images

The effectiveness of a cardiovascular magnetic resonance (CMR) scan depends on the ability of the operator to correctly tune the acquisition parameters to the subject being scanned and on the potential occurrence of imaging artifacts, such as cardiac and respiratory motion. In the clinical practice, a quality control step is performed by visual assessment of the acquired images; however, this procedure is strongly operator-dependent, cumbersome, and sometimes incompatible with the time constraints in clinical settings and large-scale studies. We propose a fast, fully automated, and learning-based quality control pipeline for CMR images, specifically for short-axis image stacks. Our pipeline performs three important quality checks: 1) heart coverage estimation; 2) inter-slice motion detection; 3) image contrast estimation in the cardiac region. The pipeline uses a hybrid decision forest method-integrating both regression and structured classification models-to extract landmarks and probabilistic segmentation maps from both long- and short-axis images as a basis to perform the quality checks. The technique was tested on up to 3000 cases from the UK Biobank and on 100 cases from the UK Digital Heart Project and validated against manual annotations and visual inspections performed by expert interpreters. The results show the capability of the proposed pipeline to correctly detect incomplete or corrupted scans (e.g., on UK Biobank, sensitivity and specificity, respectively, 88% and 99% for heart coverage estimation and 85% and 95% for motion detection), allowing their exclusion from the analyzed dataset or the triggering of a new acquisition.

Evaluating reinforcement learning agents for anatomical landmark detection

Automatic detection of anatomical landmarks is an important step for a wide range of applications in medical image analysis. Manual annotation of landmarks is a tedious task and prone to observer errors. In this paper, we evaluate novel deep reinforcement learning (RL) strategies to train agents that can precisely and robustly localize target landmarks in medical scans. An artificial RL agent learns to identify the optimal path to the landmark by interacting with an environment, in our case 3D images. Furthermore, we investigate the use of fixed- and multi-scale search strategies with novel hierarchical action steps in a coarse-to-fine manner. Several deep Q-network (DQN) architectures are evaluated for detecting multiple landmarks using three different medical imaging datasets: fetal head ultrasound (US), adult brain and cardiac magnetic resonance imaging (MRI). The performance of our agents surpasses state-of-the-art supervised and RL methods. Our experiments also show that multi-scale search strategies perform significantly better than fixed-scale agents in images with large field of view and noisy background such as in cardiac MRI. Moreover, the novel hierarchical steps can significantly speed up the searching process by a factor of 4-5 times.

Automatic localization of the subthalamic nucleus on patient-specific clinical MRI by incorporating 7 T MRI and machine learning: Application in deep brain stimulation

Deep brain stimulation (DBS) of the subthalamic nucleus (STN) has shown clinical potential for relieving the motor symptoms of advanced Parkinson's disease. While accurate localization of the STN is critical for consistent across-patients effective DBS, clear visualization of the STN under standard clinical MR protocols is still challenging. Therefore, intraoperative microelectrode recordings (MER) are incorporated to accurately localize the STN. However, MER require significant neurosurgical expertise and lengthen the surgery time. Recent advances in 7 T MR technology facilitate the ability to clearly visualize the STN. The vast majority of centers, however, still do not have 7 T MRI systems, and fewer have the ability to collect and analyze the data. This work introduces an automatic STN localization framework based on standard clinical MRIs without additional cost in the current DBS planning protocol. Our approach benefits from a large database of 7 T MRI and its clinical MRI pairs. We first model in the 7 T database, using efficient machine learning algorithms, the spatial and geometric dependency between the STN and its adjacent structures (predictors). Given a standard clinical MRI, our method automatically computes the predictors and uses the learned information to predict the patient-specific STN. We validate our proposed method on clinical T2 W MRI of 80 subjects, comparing with experts-segmented STNs from the corresponding 7 T MRI pairs. The experimental results show that our framework provides more accurate and robust patient-specific STN localization than using state-of-the-art atlases. We also demonstrate the clinical feasibility of the proposed technique assessing the post-operative electrode active contact locations.

Body region localization in whole-body low-dose CT images of PET/CT scans using virtual landmarks

PURPOSE

Radiological imaging and image interpretation for clinical decision making are mostly specific to each body region such as head and neck, thorax, abdomen, pelvis, and extremities. In this study, we present a new solution to trim automatically the given axial image stack into image volumes satisfying the given body region definition.

METHODS

The proposed approach consists of the following steps. First, a set of reference objects is selected and roughly segmented. Virtual landmarks (VLs) for the objects are then identified by using principal component analysis and recursive subdivision of the object via the principal axes system. The VLs can be defined based on just the binary objects or objects with gray values also considered. The VLs may lie anywhere with respect to the object, inside or outside, and rarely on the object surface, and are tethered to the object. Second, a classic neural network regressor is configured to learn the geometric mapping relationship between the VLs and the boundary locations of each body region. The trained network is then used to predict the locations of the body region boundaries. In this study, we focus on three body regions - thorax, abdomen, and pelvis, and predict their superior and inferior axial locations denoted by TS(I), TI(I), AS(I), AI(I), PS(I), and PI(I), respectively, for any given volume image I. Two kinds of reference objects - the skeleton and the lungs and airways, are employed to test the localization performance of the proposed approach.

RESULTS

Our method is tested by using low-dose unenhanced computed tomography (CT) images of 180 near whole-body ¹⁸ F-fluorodeoxyglucose-positron emission tomography/computed tomography (FDG-PET/CT) scans (including 34 whole-body scans) which are randomly divided into training and testing sets with a ratio of 85%:15%. The procedure is repeated six times and three times for the case of lungs and skeleton, respectively, with different divisions of the entire data set at this proportion. For the case of using skeleton as a reference object, the overall mean localization error for the six locations expressed as number of slices (nS) and distance (dS) in mm, is found to be nS: 3.4, 4.7, 4.1, 5.2, 5.2, and 3.9; dS: 13.4, 18.9, 16.5, 20.8, 20.8, and 15.5 mm for binary objects; nS: 4.1, 5.7, 4.3, 5.9, 5.9, and 4.0; dS: 16.2, 22.7, 17.2, 23.7, 23.7, and 16.1 mm for gray objects, respectively. For the case of using lungs and airways as a reference object, the corresponding results are, nS: 4.0, 5.3, 4.1, 6.9, 6.9, and 7.4; dS: 15.0, 19.7, 15.3, 26.2, 26.2, and 27.9 mm for binary objects; nS: 3.9, 5.4, 3.6, 7.2, 7.2, and 7.6; dS: 14.6, 20.1, 13.7, 27.3, 27.3, and 28.6 mm for gray objects, respectively.

CONCLUSIONS

Precise body region identification automatically in whole-body or body region tomographic images is vital for numerous medical image analysis and analytics applications. Despite its importance, this issue has received very little attention in the literature. We present a solution to this problem in this study using the concept of virtual landmarks. The method achieves localization accuracy within 2-3 slices, which is roughly comparable to the variation found in localization by experts. As long as the reference objects can be roughly segmented, the method with its learned VLs-to-boundary location relationship and predictive ability is transferable from one image modality to another.

Efficient Multiple Organ Localization in CT Image using 3D Region Proposal Network

Organ localization is an essential preprocessing step for many medical image analysis tasks such as image registration, organ segmentation and lesion detection. In this work, we propose an efficient method for multiple organ localization in CT image using 3D region proposal network. Compared with other convolutional neural network based methods that successively detect the target organs in all slices to assemble the final 3D bounding box, our method is fully implemented in 3D manner, thus can take full advantages of the spatial context information in CT image to perform efficient organ localization with only one prediction. We also propose a novel backbone network architecture that generates high-resolution feature maps to further improve the localization performance on small organs. We evaluate our method on two clinical datasets, where 11 body organs and 12 head organs (or anatomical structures) are included. As our results shown, the proposed method achieves higher detection precision and localization accuracy than the current state-of-theart methods with approximate 4 to 18 times faster processing speed. Additionally, we have established a public dataset dedicated for organ localization on http://dx. doi.org/10.21227/df8g-pq27. The full implementation of the proposed method have also been made publicly available on https://github.com/superxuang/caffe_3d_faster_rcnn.

Automatic Vertebrae Localization and Identification by Combining Deep SSAE Contextual Features and Structured Regression Forest

Automatic vertebrae localization and identification in medical computed tomography (CT) scans is of great value for computer-aided spine diseases diagnosis. In order to overcome the disadvantages of the approaches employing hand-crafted, low-level features and based on field-of-view priori assumption of spine structure, an automatic method is proposed to localize and identify vertebrae by combining deep stacked sparse autoencoder (SSAE) contextual features and structured regression forest (SRF). The method employs SSAE to learn image deep contextual features instead of hand-crafted ones by building larger-range input samples to improve their contextual discrimination ability. In the localization and identification stage, it incorporates the SRF model to achieve whole spine localization, then screens those vertebrae within the image, thus relieves the assumption that the part of spine in the field of image is visible. In the end, the output distribution of SRF and spine CT scans properties are assembled to develop a two-stage progressive refining strategy, where the mean-shift kernel density estimation and Otsu method instead of Markov random field (MRF) are adopted to reduce model complexity and refine vertebrae localization results. Extensive evaluation was performed on a challenging data set of 98 spine CT scans. Compared with the hidden Markov model and the method based on convolutional neural network (CNN), the proposed approach could effectively and automatically locate and identify spinal targets in CT scans, and achieve higher localization accuracy, low model complexity, and no need for any assumptions about visual field in CT scans.

Segmentation of the proximal femur in radial MR scans using a random forest classifier and deformable model registration

BACKGROUND

Radial 2D MRI scans of the hip are routinely used for the diagnosis of the cam type of femoroacetabular impingement (FAI) and of avascular necrosis (AVN) of the femoral head, both considered causes of hip joint osteoarthritis in young and active patients. A method for automated and accurate segmentation of the proximal femur from radial MRI scans could be very useful in both clinical routine and biomechanical studies. However, to our knowledge, no such method has been published before.

PURPOSE

The aims of this study are the development of a system for the segmentation of the proximal femur from radial MRI scans and the reconstruction of its 3D model that can be used for diagnosis and planning of hip-preserving surgery.

METHODS

The proposed system relies on: (a) a random forest classifier and (b) the registration of a 3D template mesh of the femur to the radial slices based on a physically based deformable model. The input to the system are the radial slices and the manually specified positions of three landmarks. Our dataset consists of the radial MRI scans of 25 patients symptomatic of FAI or AVN and accompanying manual segmentation of the femur, treated as the ground truth.

RESULTS

The achieved segmentation of the proximal femur has an average Dice similarity coefficient (DSC) of 96.37 ± 1.55%, an average symmetric mean absolute distance (SMAD) of 0.94 ± 0.39 mm and an average Hausdorff distance of 2.37 ± 1.14 mm. In the femoral head subregion, the average SMAD is 0.64 ± 0.18 mm and the average Hausdorff distance is 1.41 ± 0.56 mm.

CONCLUSIONS

We validated a semiautomated method for the segmentation of the proximal femur from radial MR scans. A 3D model of the proximal femur is also reconstructed, which can be used for the planning of hip-preserving surgery.

Dual-modality multi-atlas segmentation of torso organs from [18F]FDG-PET/CT images

PURPOSE

Automated segmentation of torso organs from positron emission tomography/computed tomography (PET/CT) images is a prerequisite step for nuclear medicine image analysis. However, accurate organ segmentation from clinical PET/CT is challenging due to the poor soft tissue contrast in the low-dose CT image and the low spatial resolution of the PET image. To overcome these challenges, we developed a multi-atlas segmentation (MAS) framework for torso organ segmentation from 2-deoxy-2-[¹⁸F]fluoro-D-glucose PET/CT images.

METHOD

Our key idea is to use PET information to compensate for the imperfect CT contrast and use surface-based atlas fusion to overcome the low PET resolution. First, all the organs are segmented from CT using a conventional MAS method, and then the abdomen region of the PET image is automatically cropped. Focusing on the cropped PET image, a refined MAS segmentation of the abdominal organs is performed, using a surface-based atlas fusion approach to reach subvoxel accuracy.

RESULTS

This method was validated based on 69 PET/CT images. The Dice coefficients of the target organs were between 0.80 and 0.96, and the average surface distances were between 1.58 and 2.44 mm. Compared to the CT-based segmentation, the PET-based segmentation gained a Dice increase of 0.06 and an ASD decrease of 0.38 mm. The surface-based atlas fusion leads to significant accuracy improvement for the liver and kidneys and saved ~ 10 min computation time compared to volumetric atlas fusion.

CONCLUSIONS

The presented method achieves better segmentation accuracy than conventional MAS method within acceptable computation time for clinical applications.

Simultaneous Volumetric Segmentation of Vertebral Bodies and Intervertebral Discs on Fat-Water MR Images

Fat-water magnetic resonance (MR) images allow automated noninvasive analysis of morphological properties and fat fractions of vertebral bodies (VBs) and intervertebral discs (IVDs) that constitute an important part of human biomechanical systems. In this paper, we propose a fully automated approach for simultaneously segmenting multiple VBs and IVDs on fat-water MR images without prior localization or geometry estimation. This method involved a hierarchical random forest (HRF) classifier and a hierarchical conditional random field (HCRF) that encoded a multi-resolution image pyramid based on a set of multiscale local and contextual features. The HRF classifier employed penalized multivariate linear discriminants and SMOTEBagging to handle limited and imbalanced training data with large feature dimension. The HCRF estimated optimum labels according to their spatial and hierarchical consistencies by using the layer-wise significant features determined over the trained HRF classifier. To handle variable sample numbers at different resolutions, resolution-specific hyperparameters were used. This method was trained and evaluated for segmenting 15 thoracic and lumbar VBs and their IVDs on fat-water MR images of a subset of a large cohort data set. It was further evaluated for segmenting seven IVDs of the lower spine on fat-water images of a public grand challenge. These evaluations revealed the comparable accuracy of this method with the state-of-the-art while requiring less computational burden due to a simultaneous localization and segmentation.

Multi-class segmentation of neuronal structures in electron microscopy images

BACKGROUND

Serial block face scanning electron microscopy (SBFEM) is becoming a popular technology in neuroscience. We have seen in the last years an increasing number of works addressing the problem of segmenting cellular structures in SBFEM images of brain tissue. The vast majority of them is designed to segment one specific structure, typically membranes, synapses and mitochondria. Our hypothesis is that the performance of these algorithms can be improved by concurrently segmenting more than one structure using image descriptions obtained at different scales.

RESULTS

We consider the simultaneous segmentation of two structures, namely, synapses with mitochondria, and mitochondra with membranes. To this end we select three image stacks encompassing different SBFEM acquisition technologies and image resolutions. We introduce both a new Boosting algorithm to perform feature scale selection and the Jaccard Curve as a tool compare several segmentation results. We then experimentally study the gains in performance obtained when simultaneously segmenting two structures with properly selected image descriptor scales. The results show that by doing so we achieve significant gains in segmentation accuracy when compared to the best results in the literature.

CONCLUSIONS

Simultaneously segmenting several neuronal structures described at different scales provides voxel classification algorithms with highly discriminating features that significantly improve segmentation accuracy.

Automatic aortic valve landmark localization in coronary CT angiography using colonial walk

The minimally invasive transcatheter aortic valve implantation (TAVI) is the most prevalent method to treat aortic valve stenosis. For pre-operative surgical planning, contrast-enhanced coronary CT angiography (CCTA) is used as the imaging technique to acquire 3-D measurements of the valve. Accurate localization of the eight aortic valve landmarks in CT images plays a vital role in the TAVI workflow because a small error risks blocking the coronary circulation. In order to examine the valve and mark the landmarks, physicians prefer a view parallel to the hinge plane, instead of using the conventional axial, coronal or sagittal view. However, customizing the view is a difficult and time-consuming task because of unclear aorta pose and different artifacts of CCTA. Therefore, automatic localization of landmarks can serve as a useful guide to the physicians customizing the viewpoint. In this paper, we present an automatic method to localize the aortic valve landmarks using colonial walk, a regression tree-based machine-learning algorithm. For efficient learning from the training set, we propose a two-phase optimized search space learning model in which a representative point inside the valvular area is first learned from the whole CT volume. All eight landmarks are then learned from a smaller area around that point. Experiment with preprocedural CCTA images of TAVI undergoing patients showed that our method is robust under high stenotic variation and notably efficient, as it requires only 12 milliseconds to localize all eight landmarks, as tested on a 3.60 GHz single-core CPU.

An anatomic transcriptional atlas of human glioblastoma

Glioblastoma is an aggressive brain tumor that carries a poor prognosis. The tumor's molecular and cellular landscapes are complex, and their relationships to histologic features routinely used for diagnosis are unclear. We present the Ivy Glioblastoma Atlas, an anatomically based transcriptional atlas of human glioblastoma that aligns individual histologic features with genomic alterations and gene expression patterns, thus assigning molecular information to the most important morphologic hallmarks of the tumor. The atlas and its clinical and genomic database are freely accessible online data resources that will serve as a valuable platform for future investigations of glioblastoma pathogenesis, diagnosis, and treatment.

Fully Automatic Myocardial Segmentation of Contrast Echocardiography Sequence Using Random Forests Guided by Shape Model

Myocardial contrast echocardiography (MCE) is an imaging technique that assesses left ventricle function and myocardial perfusion for the detection of coronary artery diseases. Automatic MCE perfusion quantification is challenging and requires accurate segmentation of the myocardium from noisy and time-varying images. Random forests (RF) have been successfully applied to many medical image segmentation tasks. However, the pixel-wise RF classifier ignores contextual relationships between label outputs of individual pixels. RF which only utilizes local appearance features is also susceptible to data suffering from large intensity variations. In this paper, we demonstrate how to overcome the above limitations of classic RF by presenting a fully automatic segmentation pipeline for myocardial segmentation in full-cycle 2-D MCE data. Specifically, a statistical shape model is used to provide shape prior information that guide the RF segmentation in two ways. First, a novel shape model (SM) feature is incorporated into the RF framework to generate a more accurate RF probability map. Second, the shape model is fitted to the RF probability map to refine and constrain the final segmentation to plausible myocardial shapes. We further improve the performance by introducing a bounding box detection algorithm as a preprocessing step in the segmentation pipeline. Our approach on 2-D image is further extended to 2-D+t sequences which ensures temporal consistency in the final sequence segmentations. When evaluated on clinical MCE data sets, our proposed method achieves notable improvement in segmentation accuracy and outperforms other state-of-the-art methods, including the classic RF and its variants, active shape model and image registration.

Efficient organ localization using multi-label convolutional neural networks in thorax-abdomen CT scans

Automatic localization of organs and other structures in medical images is an important preprocessing step that can improve and speed up other algorithms such as organ segmentation, lesion detection, and registration. This work presents an efficient method for simultaneous localization of multiple structures in 3D thorax-abdomen CT scans. Our approach predicts the location of multiple structures using a single multi-label convolutional neural network for each orthogonal view. Each network takes extra slices around the current slice as input to provide extra context. A sigmoid layer is used to perform multi-label classification. The output of the three networks is subsequently combined to compute a 3D bounding box for each structure. We used our approach to locate 11 structures of interest. The neural network was trained and evaluated on a large set of 1884 thorax-abdomen CT scans from patients undergoing oncological workup. Reference bounding boxes were annotated by human observers. The performance of our method was evaluated by computing the wall distance to the reference bounding boxes. The bounding boxes annotated by the first human observer were used as the reference standard for the test set. Using the best configuration, we obtained an average wall distance of [Formula: see text] mm in the test set. The second human observer achieved [Formula: see text] mm. For all structures, the results were better than those reported in previously published studies. In conclusion, we proposed an efficient method for the accurate localization of multiple organs. Our method uses multiple slices as input to provide more context around the slice under analysis, and we have shown that this improves performance. This method can easily be adapted to handle more organs.

Localizing landmark sets in head CTs using random forests and a heuristic search algorithm for registration initialization

Cochlear implants (CIs) use electrode arrays that are surgically inserted into the cochlea to stimulate frequency-mapped nerve endings to treat patients with hearing loss. CIs are programmed postoperatively by audiologists using behavioral tests without information on electrode-cochlea spatial relationship. We have recently developed techniques to segment the intracochlear anatomy and to localize individual contacts in clinically acquired computed tomography (CT) images. Using this information, we have proposed a programming strategy that we call image-guided CI programming (IGCIP), and we have shown that it significantly improves outcomes for both adult and pediatric recipients. One obstacle to large-scale deployment of this technique is the need for manual intervention in some processing steps. One of these is the rough registration of images prior to the use of automated intensity-based algorithms. Although seemingly simple, the heterogeneity of our image set makes this task challenging. We propose a solution that relies on the automated random forest-based localization of multiple landmarks used to estimate an initial transformation with a point-based registration method. Results show that it produces results that are equivalent to a manual initialization. This work is an important step toward the full automation of IGCIP.