Hierarchical Fully Convolutional Network for Joint Atrophy Localization and Alzheimer's Disease Diagnosis Using Structural MRI

FedSPL: federated self-paced learning for privacy-preserving disease diagnosis

The growing expansion of data availability in medical fields could help improve the performance of machine learning methods. However, with healthcare data, using multi-institutional datasets is challenging due to privacy and security concerns. Therefore, privacy-preserving machine learning methods are required. Thus, we use a federated learning model to train a shared global model, which is a central server that does not contain private data, and all clients maintain the sensitive data in their own institutions. The scattered training data are connected to improve model performance, while preserving data privacy. However, in the federated training procedure, data errors or noise can reduce learning performance. Therefore, we introduce the self-paced learning, which can effectively select high-confidence samples and drop high noisy samples to improve the performances of the training model and reduce the risk of data privacy leakage. We propose the federated self-paced learning (FedSPL), which combines the advantage of federated learning and self-paced learning. The proposed FedSPL model was evaluated on gene expression data distributed across different institutions where the privacy concerns must be considered. The results demonstrate that the proposed FedSPL model is secure, i.e. it does not expose the original record to other parties, and the computational overhead during training is acceptable. Compared with learning methods based on the local data of all parties, the proposed model can significantly improve the predicted F1-score by approximately 4.3%. We believe that the proposed method has the potential to benefit clinicians in gene selections and disease prognosis.

Classification of Initial Stages of Alzheimer’s Disease through Pet Neuroimaging Modality and Deep Learning: Quantifying the Impact of Image Filtering Approaches

Alzheimer’s disease (AD) is a leading health concern affecting the elderly population worldwide. It is defined by amyloid plaques, neurofibrillary tangles, and neuronal loss. Neuroimaging modalities such as positron emission tomography (PET) and magnetic resonance imaging are routinely used in clinical settings to monitor the alterations in the brain during the course of progression of AD. Deep learning techniques such as convolutional neural networks (CNNs) have found numerous applications in healthcare and other technologies. Together with neuroimaging modalities, they can be deployed in clinical settings to learn effective representations of data for different tasks such as classification, segmentation, detection, etc. Image filtering methods are instrumental in making images viable for image processing operations and have found numerous applications in image-processing-related tasks. In this work, we deployed 3D-CNNs to learn effective representations of PET modality data to quantify the impact of different image filtering approaches. We used box filtering, median filtering, Gaussian filtering, and modified Gaussian filtering approaches to preprocess the images and use them for classification using 3D-CNN architecture. Our findings suggest that these approaches are nearly equivalent and have no distinct advantage over one another. For the multiclass classification task between normal control (NC), mild cognitive impairment (MCI), and AD classes, the 3D-CNN architecture trained using Gaussian-filtered data performed the best. For binary classification between NC and MCI classes, the 3D-CNN architecture trained using median-filtered data performed the best, while, for binary classification between AD and MCI classes, the 3D-CNN architecture trained using modified Gaussian-filtered data performed the best. Finally, for binary classification between AD and NC classes, the 3D-CNN architecture trained using box-filtered data performed the best.

Improving 3D convolutional neural network comprehensibility via interactive visualization of relevance maps: evaluation in Alzheimer's disease

Background

Although convolutional neural networks (CNNs) achieve high diagnostic accuracy for detecting Alzheimer’s disease (AD) dementia based on magnetic resonance imaging (MRI) scans, they are not yet applied in clinical routine. One important reason for this is a lack of model comprehensibility. Recently developed visualization methods for deriving CNN relevance maps may help to fill this gap as they allow the visualization of key input image features that drive the decision of the model. We investigated whether models with higher accuracy also rely more on discriminative brain regions predefined by prior knowledge.

Methods

We trained a CNN for the detection of AD in N = 663 T1-weighted MRI scans of patients with dementia and amnestic mild cognitive impairment (MCI) and verified the accuracy of the models via cross-validation and in three independent samples including in total N = 1655 cases. We evaluated the association of relevance scores and hippocampus volume to validate the clinical utility of this approach. To improve model comprehensibility, we implemented an interactive visualization of 3D CNN relevance maps, thereby allowing intuitive model inspection.

Results

Across the three independent datasets, group separation showed high accuracy for AD dementia versus controls (AUC ≥ 0.91) and moderate accuracy for amnestic MCI versus controls (AUC ≈ 0.74). Relevance maps indicated that hippocampal atrophy was considered the most informative factor for AD detection, with additional contributions from atrophy in other cortical and subcortical regions. Relevance scores within the hippocampus were highly correlated with hippocampal volumes (Pearson’s r ≈ −0.86, p < 0.001).

Conclusion

The relevance maps highlighted atrophy in regions that we had hypothesized a priori. This strengthens the comprehensibility of the CNN models, which were trained in a purely data-driven manner based on the scans and diagnosis labels. The high hippocampus relevance scores as well as the high performance achieved in independent samples support the validity of the CNN models in the detection of AD-related MRI abnormalities. The presented data-driven and hypothesis-free CNN modeling approach might provide a useful tool to automatically derive discriminative features for complex diagnostic tasks where clear clinical criteria are still missing, for instance for the differential diagnosis between various types of dementia.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13195-021-00924-2.

Deep learning based pipelines for Alzheimer's disease diagnosis: A comparative study and a novel deep-ensemble method

BACKGROUND

Alzheimer's disease is a chronic neurodegenerative disease that destroys brain cells, causing irreversible degeneration of cognitive functions and dementia. Its causes are not yet fully understood, and there is no curative treatment. However, neuroimaging tools currently offer help in clinical diagnosis, and, recently, deep learning methods have rapidly become a key methodology applied to these tools. The reason is that they require little or no image preprocessing and can automatically infer an optimal representation of the data from raw images without requiring prior feature selection, resulting in a more objective and less biased process. However, training a reliable model is challenging due to the significant differences in brain image types.

METHODS

We aim to contribute to the research and study of Alzheimer's disease through computer-aided diagnosis (CAD) by comparing different deep learning models. In this work, there are three main objectives: i) to present a fully automated deep-ensemble approach for dementia-level classification from brain images, ii) to compare different deep learning architectures to obtain the most suitable one for the task, and (iii) evaluate the robustness of the proposed strategy in a deep learning framework to detect Alzheimer's disease and recognise different levels of dementia. The proposed approach is specifically designed to be potential support for clinical care based on patients' brain images.

RESULTS

Our strategy was developed and tested on three MRI and one fMRI public datasets with heterogeneous characteristics. By performing a comprehensive analysis of binary classification (Alzheimer's disease status or not) and multiclass classification (recognising different levels of dementia), the proposed approach can exceed state of the art in both tasks, reaching an accuracy of 98.51% in the binary case, and 98.67% in the multiclass case averaged over the four different data sets.

CONCLUSION

We strongly believe that integrating the proposed deep-ensemble approach will result in robust and reliable CAD systems, considering the numerous cross-dataset experiments performed. Being tested on MRIs and fMRIs, our strategy can be easily extended to other imaging techniques. In conclusion, we found that our deep-ensemble strategy could be efficiently applied for this task with a considerable potential benefit for patient management.

Diagnosis of Alzheimer's Disease with Ensemble Learning Classifier and 3D Convolutional Neural Network

Alzheimer's disease (AD), the most common type of dementia, is a progressive disease beginning with mild memory loss, possibly leading to loss of the ability to carry on a conversation and respond to environments. It can seriously affect a person's ability to carry out daily activities. Therefore, early diagnosis of AD is conducive to better treatment and avoiding further deterioration of the disease. Magnetic resonance imaging (MRI) has become the main tool for humans to study brain tissues. It can clearly reflect the internal structure of a brain and plays an important role in the diagnosis of Alzheimer's disease. MRI data is widely used for disease diagnosis. In this paper, based on MRI data, a method combining a 3D convolutional neural network and ensemble learning is proposed to improve the diagnosis accuracy. Then, a data denoising module is proposed to reduce boundary noise. The experimental results on ADNI dataset demonstrate that the model proposed in this paper improves the training speed of the neural network and achieves 95.2% accuracy in AD vs. NC (normal control) task and 77.8% accuracy in sMCI (stable mild cognitive impairment) vs. pMCI (progressive mild cognitive impairment) task in the diagnosis of Alzheimer's disease.

A Fusion of Multi-view 2D and 3D Convolution Neural Network based MRI for Alzheimer's Disease Diagnosis

Alzheimer's disease (AD) is a neurodegenerative disease leading to irreversible and progressive brain damage. Close monitoring is essential for slowing down the progression of AD. Magnetic Resonance Imaging (MRI) has been widely used for AD diagnosis and disease monitoring. Previous studies usually focused on extracting features from whole image or specific slices separately, but ignore the characteristics of each slice from multiple perspectives and the complementarity between features at different scales. In this study, we proposed a novel classification method based on the fusion of multi-view 2D and 3D convolutions for MRI-based AD diagnosis. Specifically, we first use multiple sub-networks to extract the local slice-level feature of each slice in different dimensions. Then a 3D convolution network was used to extract the global subject-level information of MRI. Finally, local and global information were fused to acquire more discriminative features. Experiments conducted on the ADNI-1 and ADNI-2 dataset demonstrated the superiority of this proposed model over other state-of-the-art methods for their ability to discriminate AD and Normal Controls (NC). Our model achieves 90.2% and 85.2% of accuracy on ADNI-2 and ADNI-1 respectively, thus it can be effective in AD diagnosis. The source code of our model is freely available at https://github.com/fengduqianhe/ADMultiView.

Autistic Spectrum Disorder Detection and Structural Biomarker Identification Using Self-Attention Model and Individual-Level Morphological Covariance Brain Networks

Autism spectrum disorder (ASD) is a range of neurodevelopmental disorders, which brings enormous burdens to the families of patients and society. However, due to the lack of representation of variance for diseases and the absence of biomarkers for diagnosis, the early detection and intervention of ASD are remarkably challenging. In this study, we proposed a self-attention deep learning framework based on the transformer model on structural MR images from the ABIDE consortium to classify ASD patients from normal controls and simultaneously identify the structural biomarkers. In our work, the individual structural covariance networks are used to perform ASD/NC classification via a self-attention deep learning framework, instead of the original structural MR data, to take full advantage of the coordination patterns of morphological features between brain regions. The self-attention deep learning framework based on the transformer model can extract both local and global information from the input data, making it more suitable for the brain network data than the CNN- structural model. Meanwhile, the potential diagnosis structural biomarkers are identified by the self-attention coefficients map. The experimental results showed that our proposed method outperforms most of the current methods for classifying ASD patients with the ABIDE data and achieves a classification accuracy of 72.5% across different sites. Furthermore, the potential diagnosis biomarkers were found mainly located in the prefrontal cortex, temporal cortex, and cerebellum, which may be treated as the early biomarkers for the ASD diagnosis. Our study demonstrated that the self-attention deep learning framework is an effective way to diagnose ASD and establish the potential biomarkers for ASD.

A parallel attention-augmented bilinear network for early magnetic resonance imaging-based diagnosis of Alzheimer's disease

Structural magnetic resonance imaging (sMRI) can capture the spatial patterns of brain atrophy in Alzheimer's disease (AD) and incipient dementia. Recently, many sMRI‐based deep learning methods have been developed for AD diagnosis. Some of these methods utilize neural networks to extract high‐level representations on the basis of handcrafted features, while others attempt to learn useful features from brain regions proposed by a separate module. However, these methods require considerable manual engineering. Their stepwise training procedures would introduce cascading errors. Here, we propose the parallel attention‐augmented bilinear network, a novel deep learning framework for AD diagnosis. Based on a 3D convolutional neural network, the framework directly learns both global and local features from sMRI scans without any prior knowledge. The framework is lightweight and suitable for end‐to‐end training. We evaluate the framework on two public datasets (ADNI‐1 and ADNI‐2) containing 1,340 subjects. On both the AD classification and mild cognitive impairment conversion prediction tasks, our framework achieves competitive results. Furthermore, we generate heat maps that highlight discriminative areas for visual interpretation. Experiments demonstrate the effectiveness of the proposed framework when medical priors are unavailable or the computing resources are limited. The proposed framework is general for 3D medical image analysis with both efficiency and interpretability.

Hippocampus segmentation and classification for dementia analysis using pre-trained neural network models

The diagnostic and clinical overlap of early mild cognitive impairment (EMCI), mild cognitive impairment (MCI), late mild cognitive impairment (LMCI) and Alzheimer disease (AD) is a vital oncological issue in dementia disorder. This study is designed to examine Whole brain (WB), grey matter (GM) and Hippocampus (HC) morphological variation and identify the prominent biomarkers in MR brain images of demented subjects to understand the severity progression. Curve evolution based on shape constraint is carried out to segment the complex brain structure such as HC and GM. Pre-trained models are used to observe the severity variation in these regions. This work is evaluated on ADNI database. The outcome of the proposed work shows that curve evolution method could segment HC and GM regions with better correlation. Pre-trained models are able to show significant severity difference among WB, GM and HC regions for the considered classes. Further, prominent variation is observed between AD vs. EMCI, AD vs. MCI and AD vs. LMCI in the whole brain, GM and HC. It is concluded that AlexNet model for HC region result in better classification for AD vs. EMCI, AD vs. MCI and AD vs. LMCI with an accuracy of 93, 78.3 and 91% respectively.

Regression and Classification of Alzheimers Disease Diagnosis using NMF-TDNet Features from 3D Brain MR Image

With the development of deep learning and medical imaging technology, many researchers use convolutional neural network(CNN) to obtain deep-level features of medical image in order to better classify Alzheimer's disease (AD) and predict clinical scores. The principal component analysis network (PCANet) is a lightweight deep-learning network that mainly uses principal component analysis (PCA) to generate multilevel filter banks for the centralized learning of samples and then performs binarization and generates blockwise histograms to obtain image features. However, the dimensions of the extracted PCANet features reaching tens of thousands or even hundreds of thousands, and the formation of the multilevel filter banks is sample data dependent, reducing the flexibility of PCANet. In this paper, based on the idea of PCANet, we propose a data-independent network called the nonnegative matrix factorization tensor decomposition network (NMF-TDNet), which improves the computational efficiency and solves the data dependence problem of PCANet. In this network, we use nonnegative matrix factorization (NMF) instead of PCA to create multilevel filter banks for sample learning, then uses the learning results to build a higher-order tensor and perform tensor decomposition (TD) to achieve data dimensionality reduction, producing the final image features. Finally, our method use these features as the input of the support vector machine (SVM) for AD classification diagnosis and clinical score prediction. The performance of our method is extensively evaluated on the ADNI-1, ADNI-2 and OASIS datasets. The experimental results show that NMF-TDNet can achieve data dimensionality reduction (the dimensionality of the extracted features numbers only a few hundred dimensions, far less than the hundreds of thousands required by PCANet) and the NMF-TDNet features as input achieved superior performance than using PCANet features as input.

Dual Attention Multi-Instance Deep Learning for Alzheimer's Disease Diagnosis With Structural MRI

Structural magnetic resonance imaging (sMRI) is widely used for the brain neurological disease diagnosis, which could reflect the variations of brain. However, due to the local brain atrophy, only a few regions in sMRI scans have obvious structural changes, which are highly correlative with pathological features. Hence, the key challenge of sMRI-based brain disease diagnosis is to enhance the identification of discriminative features. To address this issue, we propose a dual attention multi-instance deep learning network (DA-MIDL) for the early diagnosis of Alzheimer's disease (AD) and its prodromal stage mild cognitive impairment (MCI). Specifically, DA-MIDL consists of three primary components: 1) the Patch-Nets with spatial attention blocks for extracting discriminative features within each sMRI patch whilst enhancing the features of abnormally changed micro-structures in the cerebrum, 2) an attention multi-instance learning (MIL) pooling operation for balancing the relative contribution of each patch and yield a global different weighted representation for the whole brain structure, and 3) an attention-aware global classifier for further learning the integral features and making the AD-related classification decisions. Our proposed DA-MIDL model is evaluated on the baseline sMRI scans of 1689 subjects from two independent datasets (i.e., ADNI and AIBL). The experimental results show that our DA-MIDL model can identify discriminative pathological locations and achieve better classification performance in terms of accuracy and generalizability, compared with several state-of-the-art methods.

Early Alzheimer's disease diagnosis with the contrastive loss using paired structural MRIs

BACKGROUND AND OBJECTIVE

Alzheimer's Disease (AD) is a chronic and fatal neurodegenerative disease with progressive impairment of memory. Brain structural magnetic resonance imaging (sMRI) has been widely applied as important biomarkers of AD. Various machine learning approaches, especially deep learning-based models, have been proposed for the early diagnosis of AD and monitoring the disease progression on sMRI data. However, the requirement for a large number of training images still hinders the extensive usage of AD diagnosis. In addition, due to the similarities in human whole-brain structure, finding the subtle brain changes is essential to extract discriminative features from limited sMRI data effectively.

METHODS

In this work, we proposed two types of contrastive losses with paired sMRIs to promote the diagnostic performance using group categories (G-CAT) and varying subject mini-mental state examination (S-MMSE) information, respectively. Specifically, G-CAT contrastive loss layer was used to learn the closer feature representation from sMRIs with the same categories, while ranking information from S-MMSE assists the model to explore subtle changes between individuals.

RESULTS

The model was trained on ADNI-1. Comparison with baseline methods was performed on MIRIAD and ADNI-2. For the classification task on MIRIAD, S-MMSE achieves 93.5% of accuracy, 96.6% of sensitivity, and 94.9% of specificity, respectively. G-CAT and S-MMSE both reach remarkable performance in terms of classification sensitivity and specificity respectively. Comparing with state-of-the-art methods, we found this proposed method could achieve comparable results with other approaches.

CONCLUSION

The proposed model could extract discriminative features under whole-brain similarity. Extensive experiments also support the accuracy of this model, i.e., it provides better ability to identify uncertain samples, especially for the classification task of subjects with MMSE in 22-27. Source code is freely available at https://github.com/fengduqianhe/ADComparative.

Auto-Metric Graph Neural Network Based on a Meta-Learning Strategy for the Diagnosis of Alzheimer's Disease

Alzheimer's disease (AD) is the most common cognitive disorder. In recent years, many computer-aided diagnosis techniques have been proposed for AD diagnosis and progression predictions. Among them, graph neural networks (GNNs) have received extensive attention owing to their ability to effectively fuse multimodal features and model the correlation between samples. However, many GNNs for node classification use an entire dataset to construct a large fixed-graph structure, which cannot be used for independent testing. To overcome this limitation while maintaining the advantages of the GNN, we propose an auto-metric GNN (AMGNN) model for AD diagnosis. First, a metric-based meta-learning strategy is introduced to realize inductive learning for independent testing through multiple node classification tasks. In the meta-tasks, the small graphs help make the model insensitive to the sample size, thus improving the performance under small sample size conditions. Furthermore, an AMGNN layer with a probability constraint is designed to realize node similarity metric learning and effectively fuse multimodal data. We verified the model on two tasks based on the TADPOLE dataset: early AD diagnosis and mild cognitive impairment (MCI) conversion prediction. Our model provides excellent performance on both tasks with accuracies of 94.44% and 87.50% and median accuracies of 94.19% and 86.25%, respectively. These results show that our model improves flexibility while ensuring a good classification performance, thus promoting the development of graph-based deep learning algorithms for disease diagnosis.

Data-driven approaches to neuroimaging biomarkers for neurological and psychiatric disorders: emerging approaches and examples

PURPOSE OF REVIEW

The 'holy grail' of clinical applications of neuroimaging to neurological and psychiatric disorders via personalized biomarkers has remained mostly elusive, despite considerable effort. However, there are many reasons to continue to be hopeful, as the field has made remarkable advances over the past few years, fueled by a variety of converging technical and data developments.

RECENT FINDINGS

We discuss a number of advances that are accelerating the push for neuroimaging biomarkers including the advent of the 'neuroscience big data' era, biomarker data competitions, the development of more sophisticated algorithms including 'guided' data-driven approaches that facilitate automation of network-based analyses, dynamic connectivity, and deep learning. Another key advance includes multimodal data fusion approaches which can provide convergent and complementary evidence pointing to possible mechanisms as well as increase predictive accuracy.

SUMMARY

The search for clinically relevant neuroimaging biomarkers for neurological and psychiatric disorders is rapidly accelerating. Here, we highlight some of these aspects, provide recent examples from studies in our group, and link to other ongoing work in the field. It is critical that access and use of these advanced approaches becomes mainstream, this will help propel the community forward and facilitate the production of robust and replicable neuroimaging biomarkers.

Multi-model and multi-slice ensemble learning architecture based on 2D convolutional neural networks for Alzheimer's disease diagnosis

Alzheimer's Disease (AD) is a chronic neurodegenerative disease without effective medications or supplemental treatments. Thus, predicting AD progression is crucial for clinical practice and medical research. Due to limited neuroimaging data, two-dimensional convolutional neural networks (2D CNNs) have been commonly adopted to differentiate among cognitively normal subjects (CN), people with mild cognitive impairment (MCI), and AD patients. Therefore, this paper proposes an ensemble learning (EL) architecture based on 2D CNNs, using a multi-model and multi-slice ensemble. First, the top 11 coronal slices of grey matter density maps for AD versus CN classifications were selected. Second, the discriminator of a generative adversarial network, VGG16, and ResNet50 were trained with the selected slices, and the majority voting scheme was used to merge the multi-slice decisions of each model. Afterwards, those three classifiers were used to construct an ensemble model. Multi-slice ensemble learning was designed to obtain spatial features, while multi-model integration reduced the prediction error rate. Finally, transfer learning was used in domain adaptation to refine those CNNs, moving them from working solely with AD versus CN classifications to being applicable to other tasks. This ensemble approach achieved accuracy values of 90.36%, 77.19%, and 72.36% when classifying AD versus CN, AD versus MCI, and MCI versus CN, respectively. Compared with other state-of-the-art 2D studies, the proposed approach provides an effective, accurate, automatic diagnosis along the AD continuum. This technique may enhance AD diagnostics when the sample size is limited.

Deep cross-view co-regularized representation learning for glioma subtype identification

The new subtypes of diffuse gliomas are recognized by the World Health Organization (WHO) on the basis of genotypes, e.g., isocitrate dehydrogenase and chromosome arms 1p/19q, in addition to the histologic phenotype. Glioma subtype identification can provide valid guidances for both risk-benefit assessment and clinical decision. The feature representations of gliomas in magnetic resonance imaging (MRI) have been prevalent for revealing underlying subtype status. However, since gliomas are highly heterogeneous tumors with quite variable imaging phenotypes, learning discriminative feature representations in MRI for gliomas remains challenging. In this paper, we propose a deep cross-view co-regularized representation learning framework for glioma subtype identification, in which view representation learning and multiple constraints are integrated into a unified paradigm. Specifically, we first learn latent view-specific representations based on cross-view images generated from MRI via a bi-directional mapping connecting original imaging space and latent space, and view-correlated regularizer and output-consistent regularizer in the latent space are employed to explore view correlation and derive view consistency, respectively. We further learn view-sharable representations which can explore complementary information of multiple views by projecting the view-specific representations into a holistically shared space and enhancing via adversary learning strategy. Finally, the view-specific and view-sharable representations are incorporated for identifying glioma subtype. Experimental results on multi-site datasets demonstrate the proposed method outperforms several state-of-the-art methods in detection of glioma subtype status.

THAN: task-driven hierarchical attention network for the diagnosis of mild cognitive impairment and Alzheimer's disease

BACKGROUND

To assist doctors to diagnose mild cognitive impairment (MCI) and Alzheimer's disease (AD) early and accurately, convolutional neural networks based on structural magnetic resonance imaging (sMRI) images have been developed and shown excellent performance. However, they are still limited in their capacity in extracting discriminative features because of large sMRI image volumes yet small lesion regions and the small number of sMRI images.

METHODS

We proposed a task-driven hierarchical attention network (THAN) taking advantage of the merits of patch-based and attention-based convolutional neural networks for MCI and AD diagnosis. THAN consists of an information sub-network and a hierarchical attention sub-network. In the information sub-network, an information map extractor, a patch-assistant module, and a mutual-boosting loss function are designed to generate a task-driven information map, which automatically highlights disease-related regions and their importance for final classification. In the hierarchical attention sub-network, a visual attention module and a semantic attention module are devised based on the information map to extract discriminative features for disease diagnosis.

RESULTS

Extensive experiments were conducted for four classification tasks: MCI versus (vs.) normal controls (NC), AD vs. NC, AD vs. MCI, and AD vs. MCI vs. NC. Results demonstrated that THAN attained the accuracy of 81.6% for MCI vs. NC, 93.5% for AD vs. NC, 80.8% for AD vs. MCI, and 62.9% for AD vs. MCI vs. NC. It outperformed advanced attention-based and patch-based methods. Moreover, information maps generated by the information sub-network could highlight the potential biomarkers of MCI and AD, such as the hippocampus and ventricles. Furthermore, when the visual and semantic attention modules were combined, the performance of the four tasks was highly improved.

CONCLUSIONS

The information sub-network can automatically highlight the disease-related regions. The hierarchical attention sub-network can extract discriminative visual and semantic features. Through the two sub-networks, THAN fully exploits the visual and semantic features of disease-related regions and meanwhile considers global features of sMRI images, which finally facilitate the diagnosis of MCI and AD.

Multi-site MRI harmonization via attention-guided deep domain adaptation for brain disorder identification

Structural magnetic resonance imaging (MRI) has shown great clinical and practical values in computer-aided brain disorder identification. Multi-site MRI data increase sample size and statistical power, but are susceptible to inter-site heterogeneity caused by different scanners, scanning protocols, and subject cohorts. Multi-site MRI harmonization (MMH) helps alleviate the inter-site difference for subsequent analysis. Some MMH methods performed at imaging level or feature extraction level are concise but lack robustness and flexibility to some extent. Even though several machine/deep learning-based methods have been proposed for MMH, some of them require a portion of labeled data in the to-be-analyzed target domain or ignore the potential contributions of different brain regions to the identification of brain disorders. In this work, we propose an attention-guided deep domain adaptation (AD2A) framework for MMH and apply it to automated brain disorder identification with multi-site MRIs. The proposed framework does not need any category label information of target data, and can also automatically identify discriminative regions in whole-brain MR images. Specifically, the proposed AD2A is composed of three key modules: (1) an MRI feature encoding module to extract representations of input MRIs, (2) an attention discovery module to automatically locate discriminative dementia-related regions in each whole-brain MRI scan, and (3) a domain transfer module trained with adversarial learning for knowledge transfer between the source and target domains. Experiments have been performed on 2572 subjects from four benchmark datasets with T1-weighted structural MRIs, with results demonstrating the effectiveness of the proposed method in both tasks of brain disorder identification and disease progression prediction.

Spatial architecture of the immune microenvironment orchestrates tumor immunity and therapeutic response

Tumors are not only aggregates of malignant cells but also well-organized complex ecosystems. The immunological components within tumors, termed the tumor immune microenvironment (TIME), have long been shown to be strongly related to tumor development, recurrence and metastasis. However, conventional studies that underestimate the potential value of the spatial architecture of the TIME are unable to completely elucidate its complexity. As innovative high-flux and high-dimensional technologies emerge, researchers can more feasibly and accurately detect and depict the spatial architecture of the TIME. These findings have improved our understanding of the complexity and role of the TIME in tumor biology. In this review, we first epitomized some representative emerging technologies in the study of the spatial architecture of the TIME and categorized the description methods used to characterize these structures. Then, we determined the functions of the spatial architecture of the TIME in tumor biology and the effects of the gradient of extracellular nonspecific chemicals (ENSCs) on the TIME. We also discussed the potential clinical value of our understanding of the spatial architectures of the TIME, as well as current limitations and future prospects in this novel field. This review will bring spatial architectures of the TIME, an emerging dimension of tumor ecosystem research, to the attention of more researchers and promote its application in tumor research and clinical practice.

Deep Learning for Differentiating Benign From Malignant Parotid Lesions on MR Images

Purpose/Objectives(s)

Salivary gland tumors are a rare, histologically heterogeneous group of tumors. The distinction between malignant and benign tumors of the parotid gland is clinically important. This study aims to develop and evaluate a deep-learning network for diagnosing parotid gland tumors via the deep learning of MR images.

Materials/Methods

Two hundred thirty-three patients with parotid gland tumors were enrolled in this study. Histology results were available for all tumors. All patients underwent MRI scans, including T1-weighted, CE-T1-weighted and T2-weighted imaging series. The parotid glands and tumors were segmented on all three MR image series by a radiologist with 10 years of clinical experience. A total of 3791 parotid gland region images were cropped from the MR images. A label (pleomorphic adenoma and Warthin tumor, malignant tumor or free of tumor), which was based on histology results, was assigned to each image. To train the deep-learning model, these data were randomly divided into a training dataset (90%, comprising 3035 MR images from 212 patients: 714 pleomorphic adenoma images, 558 Warthin tumor images, 861 malignant tumor images, and 902 images free of tumor) and a validation dataset (10%, comprising 275 images from 21 patients: 57 pleomorphic adenoma images, 36 Warthin tumor images, 93 malignant tumor images, and 89 images free of tumor). A modified ResNet model was developed to classify these images. The input images were resized to 224x224 pixels, including four channels (T1-weighted tumor images only, T2-weighted tumor images only, CE-T1-weighted tumor images only and parotid gland images). Random image flipping and contrast adjustment were used for data enhancement. The model was trained for 1200 epochs with a learning rate of 1e-6, and the Adam optimizer was implemented. It took approximately 2 hours to complete the whole training procedure. The whole program was developed with PyTorch (version 1.2).

Results

The model accuracy with the training dataset was 92.94% (95% CI [0.91, 0.93]). The micro-AUC was 0.98. The experimental results showed that the accuracy of the final algorithm in the diagnosis and staging of parotid cancer was 82.18% (95% CI [0.77, 0.86]). The micro-AUC was 0.93.

Conclusion

The proposed model may be used to assist clinicians in the diagnosis of parotid tumors. However, future larger-scale multicenter studies are required for full validation.

Analysis of Features of Alzheimer's Disease: Detection of Early Stage from Functional Brain Changes in Magnetic Resonance Images Using a Finetuned ResNet18 Network

One of the first signs of Alzheimer's disease (AD) is mild cognitive impairment (MCI), in which there are small variants of brain changes among the intermediate stages. Although there has been an increase in research into the diagnosis of AD in its early levels of developments lately, brain changes, and their complexity for functional magnetic resonance imaging (fMRI), makes early detection of AD difficult. This paper proposes a deep learning-based method that can predict MCI, early MCI (EMCI), late MCI (LMCI), and AD. The Alzheimer's Disease Neuroimaging Initiative (ADNI) fMRI dataset consisting of 138 subjects was used for evaluation. The finetuned ResNet18 network achieved a classification accuracy of 99.99%, 99.95%, and 99.95% on EMCI vs. AD, LMCI vs. AD, and MCI vs. EMCI classification scenarios, respectively. The proposed model performed better than other known models in terms of accuracy, sensitivity, and specificity.

Relation-Induced Multi-Modal Shared Representation Learning for Alzheimer's Disease Diagnosis

The fusion of multi-modal data (e.g., magnetic resonance imaging (MRI) and positron emission tomography (PET)) has been prevalent for accurate identification of Alzheimer's disease (AD) by providing complementary structural and functional information. However, most of the existing methods simply concatenate multi-modal features in the original space and ignore their underlying associations which may provide more discriminative characteristics for AD identification. Meanwhile, how to overcome the overfitting issue caused by high-dimensional multi-modal data remains appealing. To this end, we propose a relation-induced multi-modal shared representation learning method for AD diagnosis. The proposed method integrates representation learning, dimension reduction, and classifier modeling into a unified framework. Specifically, the framework first obtains multi-modal shared representations by learning a bi-directional mapping between original space and shared space. Within this shared space, we utilize several relational regularizers (including feature-feature, feature-label, and sample-sample regularizers) and auxiliary regularizers to encourage learning underlying associations inherent in multi-modal data and alleviate overfitting, respectively. Next, we project the shared representations into the target space for AD diagnosis. To validate the effectiveness of our proposed approach, we conduct extensive experiments on two independent datasets (i.e., ADNI-1 and ADNI-2), and the experimental results demonstrate that our proposed method outperforms several state-of-the-art methods.

Asymmetric multi-task attention network for prostate bed segmentation in computed tomography images

Post-prostatectomy radiotherapy requires accurate annotation of the prostate bed (PB), i.e., the residual tissue after the operative removal of the prostate gland, to minimize side effects on surrounding organs-at-risk (OARs). However, PB segmentation in computed tomography (CT) images is a challenging task, even for experienced physicians. This is because PB is almost a "virtual" target with non-contrast boundaries and highly variable shapes depending on neighboring OARs. In this work, we propose an asymmetric multi-task attention network (AMTA-Net) for the concurrent segmentation of PB and surrounding OARs. Our AMTA-Net mimics experts in delineating the non-contrast PB by explicitly leveraging its critical dependency on the neighboring OARs (i.e., the bladder and rectum), which are relatively easy to distinguish in CT images. Specifically, we first adopt a U-Net as the backbone network for the low-level (or prerequisite) task of the OAR segmentation. Then, we build an attention sub-network upon the backbone U-Net with a series of cascaded attention modules, which can hierarchically transfer the OAR features and adaptively learn discriminative representations for the high-level (or primary) task of the PB segmentation. We comprehensively evaluate the proposed AMTA-Net on a clinical dataset composed of 186 CT images. According to the experimental results, our AMTA-Net significantly outperforms current clinical state-of-the-arts (i.e., atlas-based segmentation methods), indicating the value of our method in reducing time and labor in the clinical workflow. Our AMTA-Net also presents better performance than the technical state-of-the-arts (i.e., the deep learning-based segmentation methods), especially for the most indistinguishable and clinically critical part of the PB boundaries. Source code is released at https://github.com/superxuang/amta-net.

Deep learning prediction of mild cognitive impairment conversion to Alzheimer's disease at 3 years after diagnosis using longitudinal and whole-brain 3D MRI

BACKGROUND

While there is no cure for Alzheimer's disease (AD), early diagnosis and accurate prognosis of AD may enable or encourage lifestyle changes, neurocognitive enrichment, and interventions to slow the rate of cognitive decline. The goal of our study was to develop and evaluate a novel deep learning algorithm to predict mild cognitive impairment (MCI) to AD conversion at three years after diagnosis using longitudinal and whole-brain 3D MRI.

METHODS

This retrospective study consisted of 320 normal cognition (NC), 554 MCI, and 237 AD patients. Longitudinal data include T1-weighted 3D MRI obtained at initial presentation with diagnosis of MCI and at 12-month follow up. Whole-brain 3D MRI volumes were used without a priori segmentation of regional structural volumes or cortical thicknesses. MRIs of the AD and NC cohort were used to train a deep learning classification model to obtain weights to be applied via transfer learning for prediction of MCI patient conversion to AD at three years post-diagnosis. Two (zero-shot and fine tuning) transfer learning methods were evaluated. Three different convolutional neural network (CNN) architectures (sequential, residual bottleneck, and wide residual) were compared. Data were split into 75% and 25% for training and testing, respectively, with 4-fold cross validation. Prediction accuracy was evaluated using balanced accuracy. Heatmaps were generated.

RESULTS

The sequential convolutional approach yielded slightly better performance than the residual-based architecture, the zero-shot transfer learning approach yielded better performance than fine tuning, and CNN using longitudinal data performed better than CNN using a single timepoint MRI in predicting MCI conversion to AD. The best CNN model for predicting MCI conversion to AD at three years after diagnosis yielded a balanced accuracy of 0.793. Heatmaps of the prediction model showed regions most relevant to the network including the lateral ventricles, periventricular white matter and cortical gray matter.

CONCLUSIONS

This is the first convolutional neural network model using longitudinal and whole-brain 3D MRIs without extracting regional brain volumes or cortical thicknesses to predict future MCI to AD conversion at 3 years after diagnosis. This approach could lead to early prediction of patients who are likely to progress to AD and thus may lead to better management of the disease.

Early Detection of Alzheimer's Disease Based on Clinical Trials, Three-Dimensional Imaging Data, and Personal Information Using Autoencoders

Background:

A timely diagnosis of Alzheimer's disease (AD) is crucial to obtain more practical treatments. In this article, a novel approach using Auto-Encoder Neural Networks (AENN) for early detection of AD was proposed.

Method:

The proposed method mainly deals with the classification of multimodal data and the imputation of missing data. The data under study involve the MiniMental State Examination, magnetic resonance imaging, positron emission tomography, cerebrospinal fluid data, and personal information. Natural logarithm was used for normalizing the data. The Auto-Encoder Neural Networks was used for imputing missing data. Principal component analysis algorithm was used for reducing dimensionality of data. Support Vector Machine (SVM) was used as classifier. The proposed method was evaluated using Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Then, 10fold crossvalidation was used to audit the detection accuracy of the method.

Results:

The effectiveness of the proposed approach was studied under several scenarios considering 705 cases of ADNI database. In three binary classification problems, that is AD vs. normal controls (NCs), mild cognitive impairment (MCI) vs. NC, and MCI vs. AD, we obtained the accuracies of 95.57%, 83.01%, and 78.67%, respectively.

Conclusion:

Experimental results revealed that the proposed method significantly outperformed most of the stateoftheart methods.

Synergistic learning of lung lobe segmentation and hierarchical multi-instance classification for automated severity assessment of COVID-19 in CT images

Understanding chest CT imaging of the coronavirus disease 2019 (COVID-19) will help detect infections early and assess the disease progression. Especially, automated severity assessment of COVID-19 in CT images plays an essential role in identifying cases that are in great need of intensive clinical care. However, it is often challenging to accurately assess the severity of this disease in CT images, due to variable infection regions in the lungs, similar imaging biomarkers, and large inter-case variations. To this end, we propose a synergistic learning framework for automated severity assessment of COVID-19 in 3D CT images, by jointly performing lung lobe segmentation and multi-instance classification. Considering that only a few infection regions in a CT image are related to the severity assessment, we first represent each input image by a bag that contains a set of 2D image patches (with each cropped from a specific slice). A multi-task multi-instance deep network (called M2 UNet) is then developed to assess the severity of COVID-19 patients and also segment the lung lobe simultaneously. Our M2 UNet consists of a patch-level encoder, a segmentation sub-network for lung lobe segmentation, and a classification sub-network for severity assessment (with a unique hierarchical multi-instance learning strategy). Here, the context information provided by segmentation can be implicitly employed to improve the performance of severity assessment. Extensive experiments were performed on a real COVID-19 CT image dataset consisting of 666 chest CT images, with results suggesting the effectiveness of our proposed method compared to several state-of-the-art methods.

Quantitative Radiomic Features as New Biomarkers for Alzheimer's Disease: An Amyloid PET Study

Growing evidence indicates that amyloid-beta (Aβ) accumulation is one of the most common neurobiological biomarkers in Alzheimer's disease (AD). The primary aim of this study was to explore whether the radiomic features of Aβ positron emission tomography (PET) images are used as predictors and provide a neurobiological foundation for AD. The radiomics features of Aβ PET imaging of each brain region of the Brainnetome Atlas were computed for classification and prediction using a support vector machine model. The results showed that the area under the receiver operating characteristic curve (AUC) was 0.93 for distinguishing AD (N = 291) from normal control (NC; N = 334). Additionally, the AUC was 0.83 for the prediction of mild cognitive impairment (MCI) converting (N = 88) (vs. no conversion, N = 100) to AD. In the MCI and AD groups, the systemic analysis demonstrated that the classification outputs were significantly associated with clinical measures (apolipoprotein E genotype, polygenic risk scores, polygenic hazard scores, cerebrospinal fluid Aβ, and Tau, cognitive ability score, the conversion time for progressive MCI subjects and cognitive changes). These findings provide evidence that the radiomic features of Aβ PET images can serve as new biomarkers for clinical applications in AD/MCI, further providing evidence for predicting whether MCI subjects will convert to AD.

Light-weight cross-view hierarchical fusion network for joint localization and identification in Alzheimer's disease with adaptive instance-declined pruning

Magnetic resonance imaging (MRI) has been widely used in assessing development of Alzheimer's disease (AD) by providing structural information of disease-associated regions (e.g. atrophic regions). In this paper, we propose a light-weight cross-view hierarchical fusion network (CvHF-net), consisting of local patch and global subject subnets, for joint localization and identification of the discriminative local patches and regions in the whole brain MRI, upon which feature representations are then jointly learned and fused to construct hierarchical classification models for AD diagnosis. Firstly, based on the extracted class-discriminative 3D patches, we employ the local patch subnets to utilize multiple 2D views to represent 3D patches by using an attention-aware hierarchical fusion structure in a divide-and-conquer manner. Since different local patches are with various abilities in AD identification, the global subject subnet is developed to bias the allocation of available resources towards the most informative parts among these local patches to obtain global information for AD identification. Besides, an instance declined pruning algorithm is embedded in the CvHF-net for adaptively selecting most discriminant patches in a task-driven manner. The proposed method was evaluated on the AD Neuroimaging Initiative dataset and the experimental results show that our proposed method can achieve good performance on AD diagnosis.

A survey on active learning and human-in-the-loop deep learning for medical image analysis

Fully automatic deep learning has become the state-of-the-art technique for many tasks including image acquisition, analysis and interpretation, and for the extraction of clinically useful information for computer-aided detection, diagnosis, treatment planning, intervention and therapy. However, the unique challenges posed by medical image analysis suggest that retaining a human end-user in any deep learning enabled system will be beneficial. In this review we investigate the role that humans might play in the development and deployment of deep learning enabled diagnostic applications and focus on techniques that will retain a significant input from a human end user. Human-in-the-Loop computing is an area that we see as increasingly important in future research due to the safety-critical nature of working in the medical domain. We evaluate four key areas that we consider vital for deep learning in the clinical practice: (1) Active Learning to choose the best data to annotate for optimal model performance; (2) Interaction with model outputs - using iterative feedback to steer models to optima for a given prediction and offering meaningful ways to interpret and respond to predictions; (3) Practical considerations - developing full scale applications and the key considerations that need to be made before deployment; (4) Future Prospective and Unanswered Questions - knowledge gaps and related research fields that will benefit human-in-the-loop computing as they evolve. We offer our opinions on the most promising directions of research and how various aspects of each area might be unified towards common goals.

A Mutual Multi-Scale Triplet Graph Convolutional Network for Classification of Brain Disorders Using Functional or Structural Connectivity

Brain connectivity alterations associated with mental disorders have been widely reported in both functional MRI (fMRI) and diffusion MRI (dMRI). However, extracting useful information from the vast amount of information afforded by brain networks remains a great challenge. Capturing network topology, graph convolutional networks (GCNs) have demonstrated to be superior in learning network representations tailored for identifying specific brain disorders. Existing graph construction techniques generally rely on a specific brain parcellation to define regions-of-interest (ROIs) to construct networks, often limiting the analysis into a single spatial scale. In addition, most methods focus on the pairwise relationships between the ROIs and ignore high-order associations between subjects. In this letter, we propose a mutual multi-scale triplet graph convolutional network (MMTGCN) to analyze functional and structural connectivity for brain disorder diagnosis. We first employ several templates with different scales of ROI parcellation to construct coarse-to-fine brain connectivity networks for each subject. Then, a triplet GCN (TGCN) module is developed to learn functional/structural representations of brain connectivity networks at each scale, with the triplet relationship among subjects explicitly incorporated into the learning process. Finally, we propose a template mutual learning strategy to train different scale TGCNs collaboratively for disease classification. Experimental results on 1,160 subjects from three datasets with fMRI or dMRI data demonstrate that our MMTGCN outperforms several state-of-the-art methods in identifying three types of brain disorders.

MetricUNet: Synergistic image- and voxel-level learning for precise prostate segmentation via online sampling

Fully convolutional networks (FCNs), including UNet and VNet, are widely-used network architectures for semantic segmentation in recent studies. However, conventional FCN is typically trained by the cross-entropy or Dice loss, which only calculates the error between predictions and ground-truth labels for pixels individually. This often results in non-smooth neighborhoods in the predicted segmentation. This problem becomes more serious in CT prostate segmentation as CT images are usually of low tissue contrast. To address this problem, we propose a two-stage framework, with the first stage to quickly localize the prostate region, and the second stage to precisely segment the prostate by a multi-task UNet architecture. We introduce a novel online metric learning module through voxel-wise sampling in the multi-task network. Therefore, the proposed network has a dual-branch architecture that tackles two tasks: (1) a segmentation sub-network aiming to generate the prostate segmentation, and (2) a voxel-metric learning sub-network aiming to improve the quality of the learned feature space supervised by a metric loss. Specifically, the voxel-metric learning sub-network samples tuples (including triplets and pairs) in voxel-level through the intermediate feature maps. Unlike conventional deep metric learning methods that generate triplets or pairs in image-level before the training phase, our proposed voxel-wise tuples are sampled in an online manner and operated in an end-to-end fashion via multi-task learning. To evaluate the proposed method, we implement extensive experiments on a real CT image dataset consisting 339 patients. The ablation studies show that our method can effectively learn more representative voxel-level features compared with the conventional learning methods with cross-entropy or Dice loss. And the comparisons show that the proposed method outperforms the state-of-the-art methods by a reasonable margin.

Deep Bayesian Hashing With Center Prior for Multi-Modal Neuroimage Retrieval

Multi-modal neuroimage retrieval has greatly facilitated the efficiency and accuracy of decision making in clinical practice by providing physicians with previous cases (with visually similar neuroimages) and corresponding treatment records. However, existing methods for image retrieval usually fail when applied directly to multi-modal neuroimage databases, since neuroimages generally have smaller inter-class variation and larger inter-modal discrepancy compared to natural images. To this end, we propose a deep Bayesian hash learning framework, called CenterHash, which can map multi-modal data into a shared Hamming space and learn discriminative hash codes from imbalanced multi-modal neuroimages. The key idea to tackle the small inter-class variation and large inter-modal discrepancy is to learn a common center representation for similar neuroimages from different modalities and encourage hash codes to be explicitly close to their corresponding center representations. Specifically, we measure the similarity between hash codes and their corresponding center representations and treat it as a center prior in the proposed Bayesian learning framework. A weighted contrastive likelihood loss function is also developed to facilitate hash learning from imbalanced neuroimage pairs. Comprehensive empirical evidence shows that our method can generate effective hash codes and yield state-of-the-art performance in cross-modal retrieval on three multi-modal neuroimage datasets.

Multi-View Separable Pyramid Network for AD Prediction at MCI Stage by 18F-FDG Brain PET Imaging

Alzheimer's Disease (AD), one of the main causes of death in elderly people, is characterized by Mild Cognitive Impairment (MCI) at prodromal stage. Nevertheless, only part of MCI subjects could progress to AD. The main objective of this paper is thus to identify those who will develop a dementia of AD type among MCI patients. ¹⁸F-FluoroDeoxyGlucose Positron Emission Tomography (¹⁸F-FDG PET) serves as a neuroimaging modality for early diagnosis as it can reflect neural activity via measuring glucose uptake at resting-state. In this paper, we design a deep network on ¹⁸F-FDG PET modality to address the problem of AD identification at early MCI stage. To this end, a Multi-view Separable Pyramid Network (MiSePyNet) is proposed, in which representations are learned from axial, coronal and sagittal views of PET scans so as to offer complementary information and then combined to make a decision jointly. Different from the widely and naturally used 3D convolution operations for 3D images, the proposed architecture is deployed with separable convolution from slice-wise to spatial-wise successively, which can retain the spatial information and reduce training parameters compared to 2D and 3D networks, respectively. Experiments on ADNI dataset show that the proposed method can yield better performance than both traditional and deep learning-based algorithms for predicting the progression of Mild Cognitive Impairment, with a classification accuracy of 83.05%.

Construction of MRI-Based Alzheimer's Disease Score Based on Efficient 3D Convolutional Neural Network: Comprehensive Validation on 7,902 Images from a Multi-Center Dataset

BACKGROUND

In recent years, many convolutional neural networks (CNN) have been proposed for the classification of Alzheimer's disease. Due to memory constraints, many of the proposed CNNs work at a 2D slice-level or 3D patch-level.

OBJECTIVE

Here, we propose a subject-level 3D CNN that can extract the neurodegenerative patterns of the whole brain MRI and converted into a probabilistic Dementia score.

METHODS

We propose an efficient and lightweight subject-level 3D CNN featuring dilated convolutions. We trained our network on the ADNI data on stable Dementia of the Alzheimer's type (sDAT) from stable normal controls (sNC). To comprehensively evaluate the generalizability of our proposed network, we performed four independent tests which includes testing on images from other ADNI individuals at various stages of the dementia, images acquired from other sites (AIBL), images acquired using different protocols (OASIS), and longitudinal images acquired over a short period of time (MIRIAD).

RESULTS

We achieved a 5-fold cross-validated balanced accuracy of 88%in differentiating sDAT from sNC, and an overall specificity of 79.5%and sensitivity 79.7%on the entire set of 7,902 independent test images.

CONCLUSION

Independent testing is essential for estimating the generalization ability of the network to unseen data, but is often lacking in studies using CNN for DAT classification. This makes it difficult to compare the performances achieved using different architectures. Our comprehensive evaluation highlighting the competitive performance of our network and potential promise for generalization.

Deep learning for brain disorders: from data processing to disease treatment

In order to reach precision medicine and improve patients' quality of life, machine learning is increasingly used in medicine. Brain disorders are often complex and heterogeneous, and several modalities such as demographic, clinical, imaging, genetics and environmental data have been studied to improve their understanding. Deep learning, a subpart of machine learning, provides complex algorithms that can learn from such various data. It has become state of the art in numerous fields, including computer vision and natural language processing, and is also growingly applied in medicine. In this article, we review the use of deep learning for brain disorders. More specifically, we identify the main applications, the concerned disorders and the types of architectures and data used. Finally, we provide guidelines to bridge the gap between research studies and clinical routine.

Ensemble of ROI-based convolutional neural network classifiers for staging the Alzheimer disease spectrum from magnetic resonance imaging

Patches from three orthogonal views of selected cerebral regions can be utilized to learn convolutional neural network (CNN) models for staging the Alzheimer disease (AD) spectrum including preclinical AD, mild cognitive impairment due to AD, and dementia due to AD and normal controls. Hippocampi, amygdalae and insulae were selected from the volumetric analysis of structured magnetic resonance images (MRIs). Three-view patches (TVPs) from these regions were fed to the CNN for training. MRIs were classified with the SoftMax-normalized scores of individual model predictions on TVPs. The significance of each region of interest (ROI) for staging the AD spectrum was evaluated and reported. The results of the ensemble classifier are compared with state-of-the-art methods using the same evaluation metrics. Patch-based ROI ensembles provide comparable diagnostic performance for AD staging. In this work, TVP-based ROI analysis using a CNN provides informative landmarks in cerebral MRIs and may have significance in clinical studies and computer-aided diagnosis system design.

Improving Alzheimer's stage categorization with Convolutional Neural Network using transfer learning and different magnetic resonance imaging modalities

BACKGROUND

Alzheimer's Disease (AD) is a neurodegenerative disease characterized by progressive loss of memory and general decline in cognitive functions. Multi-modal imaging such as structural MRI and DTI provide useful information for the classification of patients on the basis of brain biomarkers. Recently, CNN methods have emerged as powerful tools to improve classification using images.

NEW METHOD

In this paper, we propose a transfer learning scheme using Convolutional Neural Networks (CNNs) to automatically classify brain scans focusing only on a small ROI: e.g. a few slices of the hippocampal region. The network's architecture is similar to a LeNet-like CNN upon which models are built and fused for AD stage classification diagnosis. We evaluated various types of transfer learning through the following mechanisms: (i) cross-modal (sMRI and DTI) and (ii) cross-domain transfer learning (using MNIST) (iii) a hybrid transfer learning of both types.

RESULTS

Our method shows good performances even on small datasets and with a limited number of slices of small brain region. It increases accuracy with more than 5 points for the most difficult classification tasks, i.e., AD/MCI and MCI/NC.

COMPARISON WITH EXISTING METHODS

Our methodology provides good accuracy scores for classification over a shallow convolutional network. Besides, we focused only on a small region; i.e., the hippocampal region, where few slices are selected to feed the network. Also, we used cross-modal transfer learning.

CONCLUSIONS

Our proposed method is suitable for working with a shallow CNN network for low-resolution MRI and DTI scans. It yields to significant results even if the model is trained on small datasets, which is often the case in medical image analysis.

Multi-class diagnosis of Alzheimer's disease using cascaded three dimensional-convolutional neural network

Dementia is a social problem in the aging society of advanced countries. Presently, 46.8 million people affected with dementia worldwide, and it may increase to 130 million by 2050. Alzheimer's disease (AD) is the most common form of dementia. The cost of care for AD patients in 2015 was 818 billion US dollars and is expected to increase intensely due to the increasing number of patients due to the aging society. It isn't easy to cure AD, but early detection is crucial. This paper proposes a multi-class classification of AD, mild cognitive impairment (MCI), and normal control (NC) subjects using three dimensional-convolutional neural network with Support Vector Machine classifier. A cross-sectional study on structural MRI data of 465 subjects, including 132 AD patients, 181 MCI, and 152 NC, is performed in this paper. The highly complex and spatial atrophy patterns of the brain related to Alzheimer's Disease and MCI are extracted from structural MRI images using cascaded layers of the three dimensional convolutional neural network. The hectic process of segmentation and further extraction of handcrafted features is eliminated. The complete image is considered for the processing, thus incorporating every region of the brain for the classification. The features extracted using four cascaded layers of three dimensional-convolutional neural network are fed into the Support Vector Machine classifier. The proposed method achieved 97.77% accuracy which outperforms state of the art, and this algorithm is a promising indicator for the diagnosis of AD.

Hierarchical Structured Sparse Learning for Schizophrenia Identification

Fractional amplitude of low-frequency fluctuation (fALFF) has been widely used for resting-state functional magnetic resonance imaging (rs-fMRI) based schizophrenia (SZ) diagnosis. However, previous studies usually measure the fALFF within low-frequency fluctuation (from 0.01 to 0.08Hz), which cannot fully cover the complex neural activity pattern in the resting-state brain. In addition, existing studies usually ignore the fact that each specific frequency band can delineate the unique spontaneous fluctuations of neural activities in the brain. Accordingly, in this paper, we propose a novel hierarchical structured sparse learning method to sufficiently utilize the specificity and complementary structure information across four different frequency bands (from 0.01Hz to 0.25Hz) for SZ diagnosis. The proposed method can help preserve the partial group structures among multiple frequency bands and the specific characters in each frequency band. We further develop an efficient optimization algorithm to solve the proposed objective function. We validate the efficacy of our proposed method on a real SZ dataset. Also, to demonstrate the generality of the method, we apply our proposed method on a subset of Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Experimental results on both datasets demonstrate that our proposed method achieves promising performance in brain disease classification, compared with several state-of-the-art methods.

A Survey on Deep Learning for Neuroimaging-Based Brain Disorder Analysis

Deep learning has recently been used for the analysis of neuroimages, such as structural magnetic resonance imaging (MRI), functional MRI, and positron emission tomography (PET), and it has achieved significant performance improvements over traditional machine learning in computer-aided diagnosis of brain disorders. This paper reviews the applications of deep learning methods for neuroimaging-based brain disorder analysis. We first provide a comprehensive overview of deep learning techniques and popular network architectures by introducing various types of deep neural networks and recent developments. We then review deep learning methods for computer-aided analysis of four typical brain disorders, including Alzheimer's disease, Parkinson's disease, Autism spectrum disorder, and Schizophrenia, where the first two diseases are neurodegenerative disorders and the last two are neurodevelopmental and psychiatric disorders, respectively. More importantly, we discuss the limitations of existing studies and present possible future directions.

Binary Classification of Alzheimer's Disease Using sMRI Imaging Modality and Deep Learning

Alzheimer's disease (AD) is an irreversible devastative neurodegenerative disorder associated with progressive impairment of memory and cognitive functions. Its early diagnosis is crucial for the development of possible future treatment option(s). Structural magnetic resonance images (sMRI) play an important role to help in understanding the anatomical changes related to AD especially in its early stages. Conventional methods require the expertise of domain experts and extract hand-picked features such as gray matter substructures and train a classifier to distinguish AD subjects from healthy subjects. Different from these methods, this paper proposes to construct multiple deep 2D convolutional neural networks (2D-CNNs) to learn the various features from local brain images which are combined to make the final classification for AD diagnosis. The whole brain image was passed through two transfer learning architectures; Inception version 3 and Xception, as well as a custom Convolutional Neural Network (CNN) built with the help of separable convolutional layers which can automatically learn the generic features from imaging data for classification. Our study is conducted using cross-sectional T1-weighted structural MRI brain images from Open Access Series of Imaging Studies (OASIS) database to maintain the size and contrast over different MRI scans. Experimental results show that the transfer learning approaches exceed the performance of non-transfer learning-based approaches demonstrating the effectiveness of these approaches for the binary AD classification task.

Asymmetrical Multi-task Attention U-Net for the Segmentation of Prostate Bed in CT Image

Segmentation of the prostate bed, the residual tissue after the removal of the prostate gland, is an essential prerequisite for post-prostatectomy radiotherapy but also a challenging task due to its non-contrast boundaries and highly variable shapes relying on neighboring organs. In this work, we propose a novel deep learning-based method to automatically segment this "invisible target". As the main idea of our design, we expect to get reference from the surrounding normal structures (bladder&rectum) and take advantage of this information to facilitate the prostate bed segmentation. To achieve this goal, we first use a U-Net as the backbone network to perform the bladder&rectum segmentation, which serves as a low-level task that can provide references to the high-level task of the prostate bed segmentation. Based on the backbone network, we build a novel attention network with a series of cascaded attention modules to further extract discriminative features for the high-level prostate bed segmentation task. Since the attention network has one-sided dependency on the backbone network, simulating the clinical workflow to use normal structures to guide the segmentation of radiotherapy target, we name the final composition model asymmetrical multi-task attention U-Net. Extensive experiments on a clinical dataset consisting of 186 CT images demonstrate the effectiveness of this new design and the superior performance of the model in comparison to the conventional atlas-based methods for prostate bed segmentation. The source code is publicly available at https://github.com/superxuang/amta-net.

Multi-task Dynamic Transformer Network for Concurrent Bone Segmentation and Large-Scale Landmark Localization with Dental CBCT

Accurate bone segmentation and anatomical landmark localization are essential tasks in computer-aided surgical simulation for patients with craniomaxillofacial (CMF) deformities. To leverage the complementarity between the two tasks, we propose an efficient end-to-end deep network, i.e., multi-task dynamic transformer network (DTNet), to concurrently segment CMF bones and localize large-scale landmarks in one-pass from large volumes of cone-beam computed tomography (CBCT) data. Our DTNet was evaluated quantitatively using CBCTs of patients with CMF deformities. The results demonstrated that our method outperforms the other state-of-the-art methods in both tasks of the bony segmentation and the landmark digitization. Our DTNet features three main technical contributions. First, a collaborative two-branch architecture is designed to efficiently capture both fine-grained image details and complete global context for high-resolution volume-to-volume prediction. Second, leveraging anatomical dependencies between landmarks, regionalized dynamic learners (RDLs) are designed in the concept of "learns to learn" to jointly regress large-scale 3D heatmaps of all landmarks under limited computational costs. Third, adaptive transformer modules (ATMs) are designed for the flexible learning of task-specific feature embedding from common feature bases.

Deep learning of brain magnetic resonance images: A brief review

Magnetic resonance imaging (MRI) is one of the most popular techniques in brain science and is important for understanding brain function and neuropsychiatric disorders. However, the processing and analysis of MRI is not a trivial task with lots of challenges. Recently, deep learning has shown superior performance over traditional machine learning approaches in image analysis. In this survey, we give a brief review of the recent popular deep learning approaches and their applications in brain MRI analysis. Furthermore, popular brain MRI databases and deep learning tools are also introduced. The strength and weaknesses of different approaches are addressed, and challenges as well as future directions are also discussed.

Grab-AD: Generalizability and reproducibility of altered brain activity and diagnostic classification in Alzheimer's Disease

Alzheimer's disease (AD) is associated with disruptions in brain activity and networks. However, there is substantial inconsistency among studies that have investigated functional brain alterations in AD; such contradictions have hindered efforts to elucidate the core disease mechanisms. In this study, we aim to comprehensively characterize AD-associated functional brain alterations using one of the world's largest resting-state functional MRI (fMRI) biobank for the disorder. The biobank includes fMRI data from six neuroimaging centers, with a total of 252 AD patients, 221 mild cognitive impairment (MCI) patients and 215 healthy comparison individuals. Meta-analytic techniques were used to unveil reliable differences in brain function among the three groups. Relative to the healthy comparison group, AD was associated with significantly reduced functional connectivity and local activity in the default-mode network, basal ganglia and cingulate gyrus, along with increased connectivity or local activity in the prefrontal lobe and hippocampus (p < .05, Bonferroni corrected). Moreover, these functional alterations were significantly correlated with the degree of cognitive impairment (AD and MCI groups) and amyloid-β burden. Machine learning models were trained to recognize key fMRI features to predict individual diagnostic status and clinical score. Leave-one-site-out cross-validation established that diagnostic status (mean area under the receiver operating characteristic curve: 0.85) and clinical score (mean correlation coefficient between predicted and actual Mini-Mental State Examination scores: 0.56, p < .0001) could be predicted with high accuracy. Collectively, our findings highlight the potential for a reproducible and generalizable functional brain imaging biomarker to aid the early diagnosis of AD and track its progression.

Spatial-Temporal Dependency Modeling and Network Hub Detection for Functional MRI Analysis via Convolutional-Recurrent Network

Early identification of dementia at the stage of mild cognitive impairment (MCI) is crucial for timely diagnosis and intervention of Alzheimer's disease (AD). Although several pioneering studies have been devoted to automated AD diagnosis based on resting-state functional magnetic resonance imaging (rs-fMRI), their performance is somewhat limited due to non-effective mining of spatial-temporal dependency. Besides, few of these existing approaches consider the explicit detection and modeling of discriminative brain regions (i.e., network hubs) that are sensitive to AD progression. In this paper, we propose a unique Spatial-Temporal convolutional-recurrent neural Network (STNet) for automated prediction of AD progression and network hub detection from rs-fMRI time series. Our STNet incorporates the spatial-temporal information mining and AD-related hub detection into an end-to-end deep learning model. Specifically, we first partition rs-fMRI time series into a sequence of overlapping sliding windows. A sequence of convolutional components are then designed to capture the local-to-global spatially-dependent patterns within each sliding window, based on which we are able to identify discriminative hubs and characterize their unique contributions to disease diagnosis. A recurrent component with long short-term memory (LSTM) units is further employed to model the whole-brain temporal dependency from the spatially-dependent pattern sequences, thus capturing the temporal dynamics along time. We evaluate the proposed method on 174 subjects with 563 rs-fMRI scans from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, with results suggesting the effectiveness of our method in both tasks of disease progression prediction and AD-related hub detection.

Independent and reproducible hippocampal radiomic biomarkers for multisite Alzheimer's disease: diagnosis, longitudinal progress and biological basis

Hippocampal morphological change is one of the main hallmarks of Alzheimer's disease (AD). However, whether hippocampal radiomic features are robust as predictors of progression from mild cognitive impairment (MCI) to AD dementia and whether these features provide any neurobiological foundation remains unclear. The primary aim of this study was to verify whether hippocampal radiomic features can serve as robust magnetic resonance imaging (MRI) markers for AD. Multivariate classifier-based support vector machine (SVM) analysis provided individual-level predictions for distinguishing AD patients (n = 261) from normal controls (NCs; n = 231) with an accuracy of 88.21% and intersite cross-validation. Further analyses of a large, independent the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset (n = 1228) reinforced these findings. In MCI groups, a systemic analysis demonstrated that the identified features were significantly associated with clinical features (e.g., apolipoprotein E (APOE) genotype, polygenic risk scores, cerebrospinal fluid (CSF) Aβ, CSF Tau), and longitudinal changes in cognition ability; more importantly, the radiomic features had a consistently altered pattern with changes in the MMSE scores over 5 years of follow-up. These comprehensive results suggest that hippocampal radiomic features can serve as robust biomarkers for clinical application in AD/MCI, and further provide evidence for predicting whether an MCI subject would convert to AD based on the radiomics of the hippocampus. The results of this study are expected to have a substantial impact on the early diagnosis of AD/MCI.