Deep Learning With an Attention Mechanism for Differentiating the Origin of Brain Metastasis Using MR images

BACKGROUND

Brain metastasis (BM) is a serious neurological complication of cancer of different origins. The value of deep learning (DL) to identify multiple types of primary origins remains unclear.

PURPOSE

To distinguish primary site of BM and identify the best DL models.

STUDY TYPE

Retrospective.

POPULATION

A total of 449 BM derived from 214 patients (49.5% for female, mean age 58 years) (100 from small cell lung cancer [SCLC], 125 from non-small cell lung cancer [NSCLC], 116 from breast cancer [BC], and 108 from gastrointestinal cancer [GIC]) were included.

FIELD STRENGTH/SEQUENCE

A 3-T, T1 turbo spin echo (T1-TSE), T2-TSE, T2FLAIR-TSE, DWI echo-planar imaging (DWI-EPI) and contrast-enhanced T1-TSE (CE T1-TSE).

ASSESSMENT

Lesions were divided into training (n = 285, 153 patients), testing (n = 122, 93 patients), and independent testing cohorts (n = 42, 34 patients). Three-dimensional residual network (3D-ResNet), named 3D ResNet6 and 3D ResNet 18, was proposed for identifying the four origins based on single MRI and combined MRI (T1WI + T2-FLAIR + DWI, CE-T1WI + DWI, CE-T1WI + T2WI + DWI). DL model was used to distinguish lung cancer from non-lung cancer; then SCLC vs. NSCLC for lung cancer classification and BC vs. GIC for non-lung cancer classification was performed. A subjective visual analysis was implemented and compared with DL models. Gradient-weighted class activation mapping (Grad-CAM) was used to visualize the model by heatmaps.

STATISTICAL TESTS

The area under the receiver operating characteristics curve (AUC) assess each classification performance.

RESULTS

3D ResNet18 with Grad-CAM and AIC showed better performance than 3DResNet6, 3DResNet18 and the radiologist for distinguishing lung cancer from non-lung cancer, SCLC from NSCLC, and BC from GIC. For single MRI sequence, T1WI, DWI, and CE-T1WI performed best for lung cancer vs. non-lung cancer, SCLC vs. NSCLC, and BC vs. GIC classifications. The AUC ranged from 0.675 to 0.876 and from 0.684 to 0.800 regarding the testing and independent testing datasets, respectively. For combined MRI sequences, the combination of CE-T1WI + T2WI + DWI performed better for BC vs. GIC (AUCs of 0.788 and 0.848 on testing and independent testing datasets, respectively), while the combined MRI approach (T1WI + T2-FLAIR + DWI, CE-T1WI + DWI) could not achieve higher AUCs for lung cancer vs. non-lung cancer, SCLC vs. NSCLC. Grad-CAM helped for model visualization by heatmaps that focused on tumor regions.

DATA CONCLUSION

DL models may help to distinguish the origins of BM based on MRI data.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Deep learning to predict esophageal variceal bleeding based on endoscopic images

OBJECTIVE

Esophageal varix (EV) bleeding is a particularly serious complications of cirrhosis. Prediction of EV bleeding requires extensive endoscopy experience; it remains unreliable and inefficient. This retrospective cohort study evaluated the feasibility of using deep learning (DL) to predict the 12-month risk of EV bleeding based on endoscopic images.

METHODS

Six DL models were trained to perform binary classification of endoscopic images of EV bleeding. The models were subsequently validated using an external test dataset, then compared with classifications performed by two endoscopists.

RESULTS

In the validation dataset, EfficientNet had the highest accuracy (0.910), followed by ConvMixer (0.898) and Xception (0.875). In the test dataset, EfficientNet maintained the highest accuracy (0.893), which was better than the endoscopists (0.800 and 0.763). Notably, one endoscopist displayed higher recall (0.905), compared with EfficientNet (0.870). When their predictions were assisted by artificial intelligence, the accuracies of the two endoscopists increased by 17.3% and 19.0%. Moreover, statistical agreement among the models was dependent on model architecture.

CONCLUSIONS

This study demonstrated the feasibility of using DL to predict the 12-month risk of EV bleeding based on endoscopic images. The findings suggest that artificial intelligence-aided diagnosis will be a useful addition to cirrhosis management.

Explainable machine learning for diffraction patterns

Serial crystallography experiments at X-ray free-electron laser facilities produce massive amounts of data but only a fraction of these data are useful for downstream analysis. Thus, it is essential to differentiate between acceptable and unacceptable data, generally known as 'hit' and 'miss', respectively. Image classification methods from artificial intelligence, or more specifically convolutional neural networks (CNNs), classify the data into hit and miss categories in order to achieve data reduction. The quantitative performance established in previous work indicates that CNNs successfully classify serial crystallography data into desired categories [Ke, Brewster, Yu, Ushizima, Yang & Sauter (2018). J. Synchrotron Rad.25, 655-670], but no qualitative evidence on the internal workings of these networks has been provided. For example, there are no visualization methods that highlight the features contributing to a specific prediction while classifying data in serial crystallography experiments. Therefore, existing deep learning methods, including CNNs classifying serial crystallography data, are like a 'black box'. To this end, presented here is a qualitative study to unpack the internal workings of CNNs with the aim of visualizing information in the fundamental blocks of a standard network with serial crystallography data. The region(s) or part(s) of an image that mostly contribute to a hit or miss prediction are visualized.

Multisite Autism Spectrum Disorder Classification Using Convolutional Neural Network Classifier and Individual Morphological Brain Networks

Autism spectrum disorder (ASD) is a range of neurodevelopmental disorders with behavioral and cognitive impairment and brings huge burdens to the patients' families and the society. To accurately identify patients with ASD from typical controls is important for early detection and early intervention. However, almost all the current existing classification methods for ASD based on structural MRI (sMRI) mainly utilize the independent local morphological features and do not consider the covariance patterns of these features between regions. In this study, by combining the convolutional neural network (CNN) and individual structural covariance network, we proposed a new framework to classify ASD patients with sMRI data from the ABIDE consortium. Moreover, gradient-weighted class activation mapping (Grad-CAM) was applied to characterize the weight of features contributing to the classification. The experimental results showed that our proposed method outperforms the currently used methods for classifying ASD patients with the ABIDE data and achieves a high classification accuracy of 71.8% across different sites. Furthermore, the discriminative features were found to be mainly located in the prefrontal cortex and cerebellum, which may be the early biomarkers for the diagnosis of ASD. Our study demonstrated that CNN is an effective tool to build the framework for the diagnosis of ASD with individual structural covariance brain network.

Successful Identification of Nasopharyngeal Carcinoma in Nasopharyngeal Biopsies Using Deep Learning

Pathologic diagnosis of nasopharyngeal carcinoma (NPC) can be challenging since most cases are nonkeratinizing carcinoma with little differentiation and many admixed lymphocytes. Our aim was to evaluate the possibility to identify NPC in nasopharyngeal biopsies using deep learning. A total of 726 nasopharyngeal biopsies were included. Among them, 100 cases were randomly selected as the testing set, 20 cases as the validation set, and all other 606 cases as the training set. All three datasets had equal numbers of NPC cases and benign cases. Manual annotation was performed. Cropped square image patches of 256 × 256 pixels were used for patch-level training, validation, and testing. The final patch-level algorithm effectively identified NPC patches, with an area under the receiver operator characteristic curve (AUC) of 0.9900. Using gradient-weighted class activation mapping, we demonstrated that the identification of NPC patches was based on morphologic features of tumor cells. At the second stage, whole-slide images were sequentially cropped into patches, inferred with the patch-level algorithm, and reconstructed into images with a smaller size for training, validation, and testing. Finally, the AUC was 0.9848 for slide-level identification of NPC. Our result shows for the first time that deep learning algorithms can identify NPC.