A transformer-based deep-learning approach for classifying brain metastases into primary organ sites using clinical whole-brain MRI images

Enhancing lung cancer detection through integrated deep learning and transformer models

Lung cancer has been stated as one of the prevalent killers of cancer up to this present time and this clearly underlines the rationale for early diagnosis to enhance life expectancy of patients afflicted with the condition. The reasons behind the usage of the transformer and deep learning classifiers for the detection of lung cancer include accuracy, robustness along with the capability to handle and evaluate large data sets and much more. Such models can be more complex and can help to utilize multiple modalities of data to give extensive information that will be critical in ascertaining the right diagnosis at the right time. However, the existing works encounter several limitations including reliance on large annotated data, overfitting, high computation complexity, and interpretability. Third, the issue of the stability of these models' performance when applied to actual clinical datasets is still an open question; this is an even bigger issue that will greatly reduce the actual utilization of these models in clinical practice. To tackle these, we develop a novel Cancer Nexus Synergy (CanNS), which applies of A. Swin-Transformer UNet (SwiNet) Model for segmentation, Xception-LSTM GAN (XLG) CancerNet for classification, and Devilish Levy Optimization (DevLO) for fine-tuning parameters. This paper breaks new ground in that the presented elements are incorporated in a manner that co-operatively elevates the diagnostic capabilities while at the same time being computationally light and resilient. These are SwiNet for segmented analysis, XLG CancerNet for precise classification of the cases, and DevLO that optimizes the parameters of the lung cancer detection system, making the system more sensible and efficient. The performance outcomes indicate that the CanNS framework enhances the detection's accuracy, sensitivity, and specificity compared to the previous approaches.

Bayesian-Optimized Convolutional Neural Networks for Classifying Primary Tumor Origin of Brain Metastases from MRI

Background/Objectives: This study evaluates whether convolutional neural networks (CNNs) can be trained to determine the primary tumor origin from MRI images alone in patients with metastatic brain lesions. Methods: This retrospective, monocentric study involved the segmentation of 1175 brain lesions from MRI scans of 436 patients with histologically confirmed primary tumor origins. The four most common tumor types-lung adenocarcinoma, small cell lung cancer, breast cancer, and melanoma-were selected, and a class-balanced dataset was created through under-sampling. This resulted in 276 training datasets and 88 hold-out test datasets. Bayesian optimization was employed to determine the optimal CNN architecture, the most relevant imaging sequences, and whether the masking of images was necessary. We compared the performance of the CNN with that of two expert radiologists specializing in neuro-oncological imaging. Results: The best-performing CNN from the Bayesian optimization process used masked images across all available MRI sequences. It achieved Area-Under-the-Curve (AUC) values of 0.75 for melanoma, 0.65 for small cell lung cancer, 0.64 for breast cancer, and 0.57 for lung adenocarcinoma. Masked images likely improved performance by focusing the CNN on relevant regions and reducing noise from surrounding tissues. In comparison, Radiologist 1 achieved AUCs of 0.55, 0.52, 0.45, and 0.51, and Radiologist 2 achieved AUCs of 0.68, 0.55, 0.64, and 0.43 for the same tumor types, respectively. The CNN consistently showed higher accuracy, particularly for melanoma and breast cancer. Conclusions: Bayesian optimization enabled the creation of a CNN that outperformed expert radiologists in classifying the primary tumor origin of brain metastases from MRI.

Vision Transformers in Medical Imaging: a Comprehensive Review of Advancements and Applications Across Multiple Diseases

The rapid advancement of artificial intelligence techniques, particularly deep learning, has transformed medical imaging. This paper presents a comprehensive review of recent research that leverage vision transformer (ViT) models for medical image classification across various disciplines. The medical fields of focus include breast cancer, skin lesions, magnetic resonance imaging brain tumors, lung diseases, retinal and eye analysis, COVID-19, heart diseases, colon cancer, brain disorders, diabetic retinopathy, skin diseases, kidney diseases, lymph node diseases, and bone analysis. Each work is critically analyzed and interpreted with respect to its performance, data preprocessing methodologies, model architecture, transfer learning techniques, model interpretability, and identified challenges. Our findings suggest that ViT shows promising results in the medical imaging domain, often outperforming traditional convolutional neural networks (CNN). A comprehensive overview is presented in the form of figures and tables summarizing the key findings from each field. This paper provides critical insights into the current state of medical image classification using ViT and highlights potential future directions for this rapidly evolving research area.

Utilizing Vision Transformers for Predicting Early Response of Brain Metastasis to Magnetic Resonance Imaging-Guided Stage Gamma Knife Radiosurgery Treatment

BACKGROUND/OBJECTIVES

This study explores the application of vision transformers to predict early responses to stereotactic radiosurgery in patients with brain metastases using minimally pre-processed magnetic resonance imaging scans. The objective is to assess the potential of vision transformers as a predictive tool for clinical decision-making, particularly in the context of imbalanced datasets.

METHODS

We analyzed magnetic resonance imaging scans from 19 brain metastases patients, focusing on axial fluid-attenuated inversion recovery and high-resolution contrast-enhanced T1-weighted sequences. Patients were categorized into responders (complete or partial response) and non-responders (stable or progressive disease).

RESULTS

Despite the imbalanced nature of the dataset, our results demonstrate that vision transformers can predict early treatment responses with an overall accuracy of 99%. The model exhibited high precision (99% for progression and 100% for regression) and recall (99% for progression and 100% for regression). The use of the attention mechanism in the vision transformers allowed the model to focus on relevant features in the magnetic resonance imaging images, ensuring an unbiased performance even with the imbalanced data. Confusion matrix analysis further confirmed the model's reliability, with minimal misclassifications. Additionally, the model achieved a perfect area under the receiver operator characteristic curve (AUC = 1.00), effectively distinguishing between responders and non-responders.

CONCLUSIONS

These findings highlight the potential of vision transformers, aided by the attention mechanism, as a non-invasive, predictive tool for early response assessment in clinical oncology. The vision transformer (ViT) model employed in this study processes MRIs as sequences of patches, enabling the capture of localized tumor features critical for early response prediction. By leveraging patch-based feature learning, this approach enhances robustness, interpretability, and clinical applicability, addressing key challenges in tumor progression prediction following stereotactic radiosurgery (SRS). The model's robust performance, despite the dataset imbalance, underscores its ability to provide unbiased predictions. This approach could significantly enhance clinical decision-making and support personalized treatment strategies for brain metastases. Future research should validate these findings in larger, more diverse cohorts and explore the integration of additional data types to further optimize the model's clinical utility.

TopoTxR: A topology-guided deep convolutional network for breast parenchyma learning on DCE-MRIs

Characterization of breast parenchyma in dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a challenging task owing to the complexity of underlying tissue structures. Existing quantitative approaches, like radiomics and deep learning models, lack explicit quantification of intricate and subtle parenchymal structures, including fibroglandular tissue. To address this, we propose a novel topological approach that explicitly extracts multi-scale topological structures to better approximate breast parenchymal structures, and then incorporates these structures into a deep-learning-based prediction model via an attention mechanism. Our topology-informed deep learning model, TopoTxR, leverages topology to provide enhanced insights into tissues critical for disease pathophysiology and treatment response. We empirically validate TopoTxR using the VICTRE phantom breast dataset, showing that the topological structures extracted by our model effectively approximate the breast parenchymal structures. We further demonstrate TopoTxR's efficacy in predicting response to neoadjuvant chemotherapy. Our qualitative and quantitative analyses suggest differential topological behavior of breast tissue in treatment-naïve imaging, in patients who respond favorably to therapy as achieving pathological complete response (pCR) versus those who do not. In a comparative analysis with several baselines on the publicly available I-SPY 1 dataset (N = 161, including 47 patients with pCR and 114 without) and the Rutgers proprietary dataset (N = 120, with 69 patients achieving pCR and 51 not), TopoTxR demonstrates a notable improvement, achieving a 2.6% increase in accuracy and a 4.6% enhancement in AUC compared to the state-of-the-art method.

Role of artificial intelligence in brain tumour imaging

Artificial intelligence (AI) is a rapidly evolving field with many neuro-oncology applications. In this review, we discuss how AI can assist in brain tumour imaging, focusing on machine learning (ML) and deep learning (DL) techniques. We describe how AI can help in lesion detection, differential diagnosis, anatomic segmentation, molecular marker identification, prognostication, and pseudo-progression evaluation. We also cover AI applications in non-glioma brain tumours, such as brain metastasis, posterior fossa, and pituitary tumours. We highlight the challenges and limitations of AI implementation in radiology, such as data quality, standardization, and integration. Based on the findings in the aforementioned areas, we conclude that AI can potentially improve the diagnosis and treatment of brain tumours and provide a path towards personalized medicine and better patient outcomes.

Limited capability of MRI radiomics to predict primary tumor histology of brain metastases in external validation

Background

Growing research demonstrates the ability to predict histology or genetic information of various malignancies using radiomic features extracted from imaging data. This study aimed to investigate MRI-based radiomics in predicting the primary tumor of brain metastases through internal and external validation, using oversampling techniques to address the class imbalance.

Methods

This IRB-approved retrospective multicenter study included brain metastases from lung cancer, melanoma, breast cancer, colorectal cancer, and a combined heterogenous group of other primary entities (5-class classification). Local data were acquired between 2003 and 2021 from 231 patients (545 metastases). External validation was performed with 82 patients (280 metastases) and 258 patients (809 metastases) from the publicly available Stanford BrainMetShare and the University of California San Francisco Brain Metastases Stereotactic Radiosurgery datasets, respectively. Preprocessing included brain extraction, bias correction, coregistration, intensity normalization, and semi-manual binary tumor segmentation. Two-thousand five hundred and twenty-eight radiomic features were extracted from T1w (± contrast), fluid-attenuated inversion recovery (FLAIR), and wavelet transforms for each sequence (8 decompositions). Random forest classifiers were trained with selected features on original and oversampled data (5-fold cross-validation) and evaluated on internal/external holdout test sets using accuracy, precision, recall, F1 score, and area under the receiver-operating characteristic curve (AUC).

Results

Oversampling did not improve the overall unsatisfactory performance on the internal and external test sets. Incorrect data partitioning (oversampling before train/validation/test split) leads to a massive overestimation of model performance.

Conclusions

Radiomics models' capability to predict histologic or genomic data from imaging should be critically assessed; external validation is essential.

Magnetic resonance-based imaging biopsy with signatures including topological Betti number features for prediction of primary brain metastatic sites

This study incorporated topology Betti number (BN) features into the prediction of primary sites of brain metastases and the construction of magnetic resonance-based imaging biopsy (MRB) models. The significant features of the MRB model were selected from those obtained from gray-scale and three-dimensional wavelet-filtered images, BN and inverted BN (iBN) maps, and clinical variables (age and gender). The primary sites were predicted as either lung cancer or other cancers using MRB models, which were built using seven machine learning methods with significant features chosen by three feature selection methods followed by a combination strategy. Our study dealt with a dataset with relatively smaller brain metastases, which included effective diameters greater than 2 mm, with metastases ranging from 2 to 9 mm accounting for 17% of the dataset. The MRB models were trained by T1-weighted contrast-enhanced images of 494 metastases chosen from 247 patients and applied to 115 metastases from 62 test patients. The most feasible model attained an area under the receiver operating characteristic curve (AUC) of 0.763 for the test patients when using a signature including features of BN and iBN maps, gray-scale and wavelet-filtered images, and clinical variables. The AUCs of the model were 0.744 for non-small cell lung cancer and 0.861 for small cell lung cancer. The results suggest that the BN signature boosted the performance of MRB for the identification of primary sites of brain metastases including small tumors.

Deep Learning With an Attention Mechanism for Differentiating the Origin of Brain Metastasis Using MR images

BACKGROUND

Brain metastasis (BM) is a serious neurological complication of cancer of different origins. The value of deep learning (DL) to identify multiple types of primary origins remains unclear.

PURPOSE

To distinguish primary site of BM and identify the best DL models.

STUDY TYPE

Retrospective.

POPULATION

A total of 449 BM derived from 214 patients (49.5% for female, mean age 58 years) (100 from small cell lung cancer [SCLC], 125 from non-small cell lung cancer [NSCLC], 116 from breast cancer [BC], and 108 from gastrointestinal cancer [GIC]) were included.

FIELD STRENGTH/SEQUENCE

A 3-T, T1 turbo spin echo (T1-TSE), T2-TSE, T2FLAIR-TSE, DWI echo-planar imaging (DWI-EPI) and contrast-enhanced T1-TSE (CE T1-TSE).

ASSESSMENT

Lesions were divided into training (n = 285, 153 patients), testing (n = 122, 93 patients), and independent testing cohorts (n = 42, 34 patients). Three-dimensional residual network (3D-ResNet), named 3D ResNet6 and 3D ResNet 18, was proposed for identifying the four origins based on single MRI and combined MRI (T1WI + T2-FLAIR + DWI, CE-T1WI + DWI, CE-T1WI + T2WI + DWI). DL model was used to distinguish lung cancer from non-lung cancer; then SCLC vs. NSCLC for lung cancer classification and BC vs. GIC for non-lung cancer classification was performed. A subjective visual analysis was implemented and compared with DL models. Gradient-weighted class activation mapping (Grad-CAM) was used to visualize the model by heatmaps.

STATISTICAL TESTS

The area under the receiver operating characteristics curve (AUC) assess each classification performance.

RESULTS

3D ResNet18 with Grad-CAM and AIC showed better performance than 3DResNet6, 3DResNet18 and the radiologist for distinguishing lung cancer from non-lung cancer, SCLC from NSCLC, and BC from GIC. For single MRI sequence, T1WI, DWI, and CE-T1WI performed best for lung cancer vs. non-lung cancer, SCLC vs. NSCLC, and BC vs. GIC classifications. The AUC ranged from 0.675 to 0.876 and from 0.684 to 0.800 regarding the testing and independent testing datasets, respectively. For combined MRI sequences, the combination of CE-T1WI + T2WI + DWI performed better for BC vs. GIC (AUCs of 0.788 and 0.848 on testing and independent testing datasets, respectively), while the combined MRI approach (T1WI + T2-FLAIR + DWI, CE-T1WI + DWI) could not achieve higher AUCs for lung cancer vs. non-lung cancer, SCLC vs. NSCLC. Grad-CAM helped for model visualization by heatmaps that focused on tumor regions.

DATA CONCLUSION

DL models may help to distinguish the origins of BM based on MRI data.

EVIDENCE LEVEL

3 TECHNICAL EFFICACY: Stage 2.

Vision transformer architecture and applications in digital health: a tutorial and survey

The vision transformer (ViT) is a state-of-the-art architecture for image recognition tasks that plays an important role in digital health applications. Medical images account for 90% of the data in digital medicine applications. This article discusses the core foundations of the ViT architecture and its digital health applications. These applications include image segmentation, classification, detection, prediction, reconstruction, synthesis, and telehealth such as report generation and security. This article also presents a roadmap for implementing the ViT in digital health systems and discusses its limitations and challenges.