Application of neural networks in prenatal diagnosis of atrioventricular septal defect

Background

There is no relevant study on landmarks detection, one of the Convolutional Neural Network algorithms, in the field of fetal echocardiography (FE). This study aimed to explore whether automatic landmarks detection could be used in FE correctly and whether the atrial length (AL) to ventricular length (VL) ratio (AVLR) could be used to diagnose atrioventricular septal defect (AVSD) prenatally.

Methods

This was an observational study. Two hundred and seventy-eight four-chamber views in end diastole, divided into the normal, AVSD, and differential diagnosis groups, were retrospectively included in this study. Seven landmarks were labeled sequentially by the experts on these images, and all images were divided into the training and test sets for normal, AVSD, and differential diagnosis groups. U-net, MA-net, and Link-net were used as landmark prediction neural networks. The accuracy of the landmark detection, AL, and VL measurements, as well as the prenatal diagnostic effectiveness of AVLR for AVSD, was compared with the expert labeled.

Results

U-net, MA-net, and Link-net could detect the landmarks precisely (within the localization error of 0.09 and 0.13 on X and Y axis) and measure AL and VL accurately (the measured pixel distance error of AL and VL were 0.12 and 0.01 separately). AVLR in AVSD was greater than in other groups (P<0.0001), but the statistical difference was not obvious in the complete, partial, and transitional subgroups (P>0.05). The diagnostic effectiveness of AVLR calculated by three models, area under receiver operating characteristic curve could reach 0.992 (0.968-1.000), was consistent with the expert labeled.

Conclusions

U-net, Link-net, and MA-net could detect landmarks and make the measurements accurately. AVLR calculated by three neural networks could be used to make the prenatal diagnosis of AVSD.

A method for Kashin-Beck disease auxiliary diagnosis based on the features in regions of the potential lesion

BACKGROUND

Kashin-Beck disease (KBD) is a severe arthropathy that causes deformity. Patients with advanced stages of KBD often show symptoms, such as short stature. Early-stage diagnosis and treatment can effectively prevent the disease from worsening. Diagnosis of early-stage patients is usually made by X-ray examination. However, the time-consuming image recognition and the lack of professional doctors may delay the patient's condition. Therefore, a method that can efficiently complete the auxiliary diagnosis is necessary.

PURPOSE

This study presents a KBD auxiliary diagnosis method based on radiographs, which uses deep learning to locate potential lesion regions and extract features for accurate diagnosis.

METHODS

This work presents a method that relies on hand radiographs to locate eight regions of the potential lesion (RoPL) and finally make the KBD auxiliary diagnosis. The localization of RoPL is achieved through a two-step model, with the first step predicting a rough location and a deep convolutional neural network (DCNN) with attention mechanism used to generate precise center coordinates based on the previous step's results. Based on the localization result, regional features are extracted, which provides information about the joints and textures of RoPL from a finer granularity. Another DCNN is utilized to obtain general features from hand radiographs, which provide morphological and structural information about the entire hand bone These features offer a concatenated feature for categorization to raise accuracy. A doctor-like approach is adopted to diagnose based on regional features to enhance performance, and a final decision is made using a vote that considers diagnostic outcomes from all aspects. The dataset used in our study was collected by our research team in KBD-endemic areas of Tibet since 2017, containing 373 diseased and 764 normal images.

RESULT

Our model guarantees that over 95% of the predicted coordinates are within five pixels of the real coordinates according to Euclidean distance. The accuracy of the diagnostic network achieved 91.3%, with precision and recall achieving 83% and 87%, respectively. Compared to the approach without exact localization of the illness region on the same test set, our method achieved a roughly 6% increase in accuracy and nearly 30% increase in recall rate.

CONCLUSIONS

Based on hand radiographs, this study suggests a novel method for KBD diagnosis. The high-precision localization network guarantees precise extraction of lesion-prone regional features, and the multi-scale features and innovative classification method further enhance model performance compared to related approaches.

Joint Cranial Bone Labeling and Landmark Detection in Pediatric CT Images Using Context Encoding

Image segmentation, labeling, and landmark detection are essential tasks for pediatric craniofacial evaluation. Although deep neural networks have been recently adopted to segment cranial bones and locate cranial landmarks from computed tomography (CT) or magnetic resonance (MR) images, they may be hard to train and provide suboptimal results in some applications. First, they seldom leverage global contextual information that can improve object detection performance. Second, most methods rely on multi-stage algorithm designs that are inefficient and prone to error accumulation. Third, existing methods often target simple segmentation tasks and have shown low reliability in more challenging scenarios such as multiple cranial bone labeling in highly variable pediatric datasets. In this paper, we present a novel end-to-end neural network architecture based on DenseNet that incorporates context regularization to jointly label cranial bone plates and detect cranial base landmarks from CT images. Specifically, we designed a context-encoding module that encodes global context information as landmark displacement vector maps and uses it to guide feature learning for both bone labeling and landmark identification. We evaluated our model on a highly diverse pediatric CT image dataset of 274 normative subjects and 239 patients with craniosynostosis (age 0.63 ± 0.54 years, range 0-2 years). Our experiments demonstrate improved performance compared to state-of-the-art approaches.

Learning-based landmark detection in pelvis x-rays with attention mechanism: data from the osteoarthritis initiative

Patients with developmental dysplasia of the hip can have this problem throughout their lifetime. The problem is difficult to detect by radiologists throughout x-ray because of an abrasion of anatomical structures. Thus, the landmarks should be automatically and precisely located. In this paper, we propose an attention mechanism of combining multi-dimension information on the basis of separating spatial dimension. The proposed attention mechanism decouples spatial dimension and forms width-channel dimension and height-channel dimension by 1D pooling operations in the height and width of spatial dimension. Then non-local means operations are performed to capture the correlation between long-range pixels in width-channel dimension, as well as that in height-channel dimension at different resolutions. The proposed attention mechanism modules are inserted into the skipped connections of U-Net to form a novel landmark detection structure. This landmark detection method was trained and evaluated through five-fold cross-validation on an open-source dataset, including 524 pelvis x-ray, each containing eight landmarks in pelvis, and achieved excellent performance compared to other landmark detection models. The average point-to-point errors of U-Net, HR-Net, CE-Net, and the proposed network were 3.5651 mm, 3.6118 mm, 3.3914 mm and 3.1350 mm, respectively. The results indicate that the proposed method has the highest detection accuracy. Furthermore, an open-source pelvis dataset is annotated and released for open research.

Part Affinity Fields and CoordConv for Detecting Landmarks of Lumbar Vertebrae and Sacrum in X-ray Images

With the prevalence of degenerative diseases due to the increase in the aging population, we have encountered many spine-related disorders. Since the spine is a crucial part of the body, fast and accurate diagnosis is critically important. Generally, clinicians use X-ray images to diagnose the spine, but X-ray images are commonly occluded by the shadows of some bones, making it hard to identify the whole spine. Therefore, recently, various deep-learning-based spinal X-ray image analysis approaches have been proposed to help diagnose the spine. However, these approaches did not consider the characteristics of frequent occlusion in the X-ray image and the properties of the vertebra shape. Therefore, based on the X-ray image properties and vertebra shape, we present a novel landmark detection network specialized in lumbar X-ray images. The proposed network consists of two stages: The first step detects the centers of the lumbar vertebrae and the upper end plate of the first sacral vertebra (S1), and the second step detects the four corner points of each lumbar vertebra and two corner points of S1 from the image obtained in the first step. We used random spine cutout augmentation in the first step to robustify the network against the commonly obscured X-ray images. Furthermore, in the second step, we used CoordConv to make the network recognize the location distribution of landmarks and part affinity fields to understand the morphological features of the vertebrae, resulting in more accurate landmark detection. The proposed network was evaluated using 304 X-ray images, and it achieved 98.02% accuracy in center detection and 8.34% relative distance error in corner detection. This indicates that our network can detect spinal landmarks reliably enough to support radiologists in analyzing the lumbar X-ray images.

Multi-landmark environment analysis with reinforcement learning for pelvic abnormality detection and quantification

Morphological abnormalities of the femoroacetabular (hip) joint are among the most common human musculoskeletal disorders and often develop asymptomatically at early easily treatable stages. In this paper, we propose an automated framework for landmark-based detection and quantification of hip abnormalities from magnetic resonance (MR) images. The framework relies on a novel idea of multi-landmark environment analysis with reinforcement learning. In particular, we merge the concepts of the graphical lasso and Morris sensitivity analysis with deep neural networks to quantitatively estimate the contribution of individual landmark and landmark subgroup locations to the other landmark locations. Convolutional neural networks for image segmentation are utilized to propose the initial landmark locations, and landmark detection is then formulated as a reinforcement learning (RL) problem, where each landmark-agent can adjust its position by observing the local MR image neighborhood and the locations of the most-contributive landmarks. The framework was validated on T1-, T2- and proton density-weighted MR images of 260 patients with the aim to measure the lateral center-edge angle (LCEA), femoral neck-shaft angle (NSA), and the anterior and posterior acetabular sector angles (AASA and PASA) of the hip, and derive the quantitative abnormality metrics from these angles. The framework was successfully tested using the UNet and feature pyramid network (FPN) segmentation architectures for landmark proposal generation, and the deep Q-network (DeepQN), deep deterministic policy gradient (DDPG), twin delayed deep deterministic policy gradient (TD3), and actor-critic policy gradient (A2C) RL networks for landmark position optimization. The resulting overall landmark detection error of 1.5 mm and angle measurement error of 1.4° indicates a superior performance in comparison to existing methods. Moreover, the automatically estimated abnormality labels were in 95% agreement with those generated by an expert radiologist.

Fast and Accurate Craniomaxillofacial Landmark Detection via 3D Faster R-CNN

Automatic craniomaxillofacial (CMF) landmark localization from cone-beam computed tomography (CBCT) images is challenging, considering that 1) the number of landmarks in the images may change due to varying deformities and traumatic defects, and 2) the CBCT images used in clinical practice are typically large. In this paper, we propose a two-stage, coarse-to-fine deep learning method to tackle these challenges with both speed and accuracy in mind. Specifically, we first use a 3D faster R-CNN to roughly locate landmarks in down-sampled CBCT images that have varying numbers of landmarks. By converting the landmark point detection problem to a generic object detection problem, our 3D faster R-CNN is formulated to detect virtual, fixed-size objects in small boxes with centers indicating the approximate locations of the landmarks. Based on the rough landmark locations, we then crop 3D patches from the high-resolution images and send them to a multi-scale UNet for the regression of heatmaps, from which the refined landmark locations are finally derived. We evaluated the proposed approach by detecting up to 18 landmarks on a real clinical dataset of CMF CBCT images with various conditions. Experiments show that our approach achieves state-of-the-art accuracy of 0.89 ± 0.64mm in an average time of 26.2 seconds per volume.

Automatic generation of three-dimensional dose reconstruction data for two-dimensional radiotherapy plans for historically treated patients

Performing large-scale three-dimensional radiation dose reconstruction for patients requires a large amount of manual work. We present an image processing-based pipeline to automatically reconstruct radiation dose. The pipeline was designed for childhood cancer survivors that received abdominal radiotherapy with anterior-to-posterior and posterior-to-anterior field set-up. First, anatomical landmarks are automatically identified on two-dimensional radiographs. Second, these landmarks are used to derive parameters to emulate the geometry of the plan on a surrogate computed tomography. Finally, the plan is emulated and used as input for dose calculation. For qualitative evaluation, 100 cases of automatic and manual plan emulations were assessed by two experienced radiation dosimetrists in a blinded comparison. The two radiation dosimetrists approved 100%/100% and 92%/91% of the automatic/manual plan emulations, respectively. Similar approval rates of 100% and 94% hold when the automatic pipeline is applied on another 50 cases. Further, quantitative comparisons resulted in on average < 5    mm difference in plan isocenter/borders, and < 0.9    Gy in organ mean dose (prescribed dose: 14.4 Gy) calculated from the automatic and manual plan emulations. No statistically significant difference in terms of dose reconstruction accuracy was found for most organs at risk. Ultimately, our automatic pipeline results are of sufficient quality to enable effortless scaling of dose reconstruction data generation.

The Role of Landscapes and Landmarks in Bee Navigation: A Review

The ability of animals to explore landmarks in their environment is essential to their fitness. Landmarks are widely recognized to play a key role in navigation by providing information in multiple sensory modalities. However, what is a landmark? We propose that animals use a hierarchy of information based upon its utility and salience when an animal is in a given motivational state. Focusing on honeybees, we suggest that foragers choose landmarks based upon their relative uniqueness, conspicuousness, stability, and context. We also propose that it is useful to distinguish between landmarks that provide sensory input that changes (“near”) or does not change (“far”) as the receiver uses these landmarks to navigate. However, we recognize that this distinction occurs on a continuum and is not a clear-cut dichotomy. We review the rich literature on landmarks, focusing on recent studies that have illuminated our understanding of the kinds of information that bees use, how they use it, potential mechanisms, and future research directions.

Detecting Anatomical Landmarks for Fast Alzheimer's Disease Diagnosis

Structural magnetic resonance imaging (MRI) is a very popular and effective technique used to diagnose Alzheimer's disease (AD). The success of computer-aided diagnosis methods using structural MRI data is largely dependent on the two time-consuming steps: 1) nonlinear registration across subjects, and 2) brain tissue segmentation. To overcome this limitation, we propose a landmark-based feature extraction method that does not require nonlinear registration and tissue segmentation. In the training stage, in order to distinguish AD subjects from healthy controls (HCs), group comparisons, based on local morphological features, are first performed to identify brain regions that have significant group differences. In general, the centers of the identified regions become landmark locations (or AD landmarks for short) capable of differentiating AD subjects from HCs. In the testing stage, using the learned AD landmarks, the corresponding landmarks are detected in a testing image using an efficient technique based on a shape-constrained regression-forest algorithm. To improve detection accuracy, an additional set of salient and consistent landmarks are also identified to guide the AD landmark detection. Based on the identified AD landmarks, morphological features are extracted to train a support vector machine (SVM) classifier that is capable of predicting the AD condition. In the experiments, our method is evaluated on landmark detection and AD classification sequentially. Specifically, the landmark detection error (manually annotated versus automatically detected) of the proposed landmark detector is 2.41 mm , and our landmark-based AD classification accuracy is 83.7%. Lastly, the AD classification performance of our method is comparable to, or even better than, that achieved by existing region-based and voxel-based methods, while the proposed method is approximately 50 times faster.

Lung registration using automatically detected landmarks

OBJECTIVES

Accurate registration of lung CT images is inevitable for numerous clinical applications. Usually, nonlinear intensity-based methods are used. Their accuracy is typically evaluated using corresponding anatomical points (landmarks; e.g. bifurcations of bronchial and vessel trees) annotated by medical experts in the images to register. As image registration can be interpreted as correspondence finding problem, these corresponding landmarks can also be used in feature-based registration techniques. Recently, approaches for automated identification of such landmark correspondences in lung CT images have been presented. In this work, a novel combination of variational nonlinear intensity-based registration with an approach for automated landmark correspondence detection in lung CT pairs is presented and evaluated.

METHODS

The main blocks of the proposed hybrid intensity- and feature-based registration scheme are a two-step landmark correspondence detection and the so-called CoLD (Combining Landmarks and Distance Measures) framework. The landmark correspondence identification starts with feature detection in one image followed by a blockmatching-based transfer of the features to the other image. The established correspondences are used to compute a thin-plate spline (TPS) transformation. Within CoLD, the TPS transformation is improved by minimization of an objective function consisting of a Normalized Gradient Field distance measure and a curvature regularizer; the landmark correspondences are guaranteed to be preserved by optimization on the kernel of the discretized landmark constraints.

RESULTS

Based on ten publicly available end-inspiration/expiration CT scan pairs with anatomical landmark sets annotated by medical experts from the DIR-Lab database, it is shown that the hybrid registration approach is superior in terms of accuracy: The mean distance of expert landmarks is decreased from 8.46 mm before to 1.15 mm after registration, outperforming both the TPS transformation (1.68 mm) and a nonlinear registration without usage of automatically detected landmarks (2.44 mm). The improvement is statistically significant in eight of ten datasets in comparison to TPS and in nine of ten datasets in comparison to the intensity-based registration. Furthermore, CoLD globally estimates the breathing-induced lung volume change well and results in smooth and physiologically plausible motion fields of the lungs.

CONCLUSIONS

We demonstrated that our novel landmark-based registration pipeline outperforms both TPS and the underlying nonlinear intensity-based registration without landmark usage. This highlights the potential of automatic landmark correspondence detection for improvement of lung CT registration accuracy.

Automatic model-based semantic registration of multimodal MRI knee data

PURPOSE

To propose a robust and automated model-based semantic registration for the multimodal alignment of the knee bone and cartilage from three-dimensional (3D) MR image data.

MATERIALS AND METHODS

The movement of the knee joint can be semantically interpreted as a combination of movements of each bone. A semantic registration of the knee joint was implemented by separately reconstructing the rigid movements of the three bones. The proposed method was validated by registering 3D morphological MR datasets of 25 subjects into the corresponding T2 map datasets, and was compared with rigid and elastic methods using two criteria: the spatial overlap of the manually segmented cartilage and the distance between the same landmarks in the reference and target datasets.

RESULTS

The mean Dice Similarity Coefficient (DSC) of the overlapped cartilage segmentation was increased to 0.68 ± 0.1 (mean ± SD) and the landmark distance was reduced to 1.3 ± 0.3 mm after the proposed registration method. Both metrics were statistically superior to using rigid (DSC: 0.59 ± 0.12; landmark distance: 2.1 ± 0.4 mm) and elastic (DSC: 0.64 ± 0.11; landmark distance: 1.5 ± 0.5 mm) registrations.

CONCLUSION

The proposed method is an efficient and robust approach for the automated registration between morphological knee datasets and T2 MRI relaxation maps.

Salient and Non-Salient Fiducial Detection using a Probabilistic Graphical Model

Deformable shape detection is an important problem in computer vision and pattern recognition. However, standard detectors are typically limited to locating only a few salient landmarks such as landmarks near edges or areas of high contrast, often conveying insufficient shape information. This paper presents a novel statistical pattern recognition approach to locate a dense set of salient and non-salient landmarks in images of a deformable object. We explore the fact that several object classes exhibit a homogeneous structure such that each landmark position provides some information about the position of the other landmarks. In our model, the relationship between all pairs of landmarks is naturally encoded as a probabilistic graph. Dense landmark detections are then obtained with a new sampling algorithm that, given a set of candidate detections, selects the most likely positions as to maximize the probability of the graph. Our experimental results demonstrate accurate, dense landmark detections within and across different databases.

Automated Identification of Fiducial Points on 3D Torso Images

Breast reconstruction is an important part of the breast cancer treatment process for many women. Recently, 2D and 3D images have been used by plastic surgeons for evaluating surgical outcomes. Distances between different fiducial points are frequently used as quantitative measures for characterizing breast morphology. Fiducial points can be directly marked on subjects for direct anthropometry, or can be manually marked on images. This paper introduces novel algorithms to automate the identification of fiducial points in 3D images. Automating the process will make measurements of breast morphology more reliable, reducing the inter- and intra-observer bias. Algorithms to identify three fiducial points, the nipples, sternal notch, and umbilicus, are described. The algorithms used for localization of these fiducial points are formulated using a combination of surface curvature and 2D color information. Comparison of the 3D co-ordinates of automatically detected fiducial points and those identified manually, and geodesic distances between the fiducial points are used to validate algorithm performance. The algorithms reliably identified the location of all three of the fiducial points. We dedicate this article to our late colleague and friend, Dr. Elisabeth K. Beahm. Elisabeth was both a talented plastic surgeon and physician-scientist; we deeply miss her insight and her fellowship.