AI-Enhanced Diagnosis of Challenging Lesions in Breast MRI: A Methodology and Application Primer

MRI-Based Breast Cancer Classification and Localization by Multiparametric Feature Extraction and Combination Using Deep Learning

BACKGROUND

Deep learning (DL) have been reported feasible in breast MRI. However, the effectiveness of DL method in mpMRI combinations for breast cancer detection has not been well investigated.

PURPOSE

To implement a DL method for breast cancer classification and detection using feature extraction and combination from multiple sequences.

STUDY TYPE

Retrospective.

POPULATION

A total of 569 local cases as internal cohort (50.2 ± 11.2 years; 100% female), divided among training (218), validation (73) and testing (278); 125 cases from a public dataset as the external cohort (53.6 ± 11.5 years; 100% female).

FIELD STRENGTH/SEQUENCE

T1-weighted imaging and dynamic contrast-enhanced MRI (DCE-MRI) with gradient echo sequences, T2-weighted imaging (T2WI) with spin-echo sequences, diffusion-weighted imaging with single-shot echo-planar sequence and at 1.5-T.

ASSESSMENT

A convolutional neural network and long short-term memory cascaded network was implemented for lesion classification with histopathology as the ground truth for malignant and benign categories and contralateral breasts as healthy category in internal/external cohorts. BI-RADS categories were assessed by three independent radiologists as comparison, and class activation map was employed for lesion localization in internal cohort. The classification and localization performances were assessed with DCE-MRI and non-DCE sequences, respectively.

STATISTICAL TESTS

Sensitivity, specificity, area under the curve (AUC), DeLong test, and Cohen's kappa for lesion classification. Sensitivity and mean squared error for localization. A P-value <0.05 was considered statistically significant.

RESULTS

With the optimized mpMRI combinations, the lesion classification achieved an AUC = 0.98/0.91, sensitivity = 0.96/0.83 in the internal/external cohorts, respectively. Without DCE-MRI, the DL-based method was superior to radiologists' readings (AUC 0.96 vs. 0.90). The lesion localization achieved sensitivities of 0.97/0.93 with DCE-MRI/T2WI alone, respectively.

DATA CONCLUSION

The DL method achieved high accuracy for lesion detection in the internal/external cohorts. The classification performance with a contrast agent-free combination is comparable to DCE-MRI alone and the radiologists' reading in AUC and sensitivity.

EVIDENCE LEVEL

3.

TECHNICAL EFFICACY

Stage 2.

Deep Learning Model Based on Dual-Modal Ultrasound and Molecular Data for Predicting Response to Neoadjuvant Chemotherapy in Breast Cancer

RATIONALE AND OBJECTIVES

To carry out radiomics analysis/deep convolutional neural network (CNN) based on B-mode ultrasound (BUS) and shear wave elastography (SWE) to predict response to neoadjuvant chemotherapy (NAC) in breast cancer patients.

MATERIALS AND METHODS

In this prospective study, 255 breast cancer patients who received NAC between September 2016 and December 2021 were included. Radiomics models were designed using a support vector machine classifier based on US images obtained before treatment, including BUS and SWE. And CNN models also were developed using ResNet architecture. The final predictive model was developed by combining the dual-modal US and independently associated clinicopathologic characteristics. The predictive performances of the models were assessed with five-fold cross-validation.

RESULTS

Pretreatment SWE performed better than BUS in predicting the response to NAC for breast cancer for both the CNN and radiomics models (P < 0.001). The predictive results of the CNN models were significantly better than the radiomics models, with AUCs of 0.72 versus 0.69 for BUS and 0.80 versus 0.77 for SWE, respectively (P = 0.003). The CNN model based on the dual-modal US and molecular data exhibited outstanding performance in predicting NAC response, with an accuracy of 83.60% ± 2.63%, a sensitivity of 87.76% ± 6.44%, and a specificity of 77.45% ± 4.38%.

CONCLUSION

The pretreatment CNN model based on the dual-modal US and molecular data achieved excellent performance for predicting the response to chemotherapy in breast cancer. Therefore, this model has the potential to serve as a non-invasive objective biomarker to predict NAC response and aid clinicians with individual treatments.

Predicting Breast Cancer Subtypes Using Magnetic Resonance Imaging Based Radiomics With Automatic Segmentation

OBJECTIVE

The aim of the study is to demonstrate whether radiomics based on an automatic segmentation method is feasible for predicting molecular subtypes.

METHODS

This retrospective study included 516 patients with confirmed breast cancer. An automatic segmentation-3-dimensional UNet-based Convolutional Neural Networks, trained on our in-house data set-was applied to segment the regions of interest. A set of 1316 radiomics features per region of interest was extracted. Eighteen cross-combination radiomics methods-with 6 feature selection methods and 3 classifiers-were used for model selection. Model classification performance was assessed using the area under the receiver operating characteristic curve (AUC), accuracy, sensitivity, and specificity.

RESULTS

The average dice similarity coefficient value of the automatic segmentation was 0.89. The radiomics models were predictive of 4 molecular subtypes with the best average: AUC = 0.8623, accuracy = 0.6596, sensitivity = 0.6383, and specificity = 0.8775. For luminal versus nonluminal subtypes, AUC = 0.8788 (95% confidence interval [CI], 0.8505-0.9071), accuracy = 0.7756, sensitivity = 0.7973, and specificity = 0.7466. For human epidermal growth factor receptor 2 (HER2)-enriched versus non-HER2-enriched subtypes, AUC = 0.8676 (95% CI, 0.8370-0.8982), accuracy = 0.7737, sensitivity = 0.8859, and specificity = 0.7283. For triple-negative breast cancer versus non-triple-negative breast cancer subtypes, AUC = 0.9335 (95% CI, 0.9027-0.9643), accuracy = 0.9110, sensitivity = 0.4444, and specificity = 0.9865.

CONCLUSIONS

Radiomics based on automatic segmentation of magnetic resonance imaging can predict breast cancer of 4 molecular subtypes noninvasively and is potentially applicable in large samples.

Machine learning for predicting breast-conserving surgery candidates after neoadjuvant chemotherapy based on DCE-MRI

Purpose

This study aimed to investigate a machine learning method for predicting breast-conserving surgery (BCS) candidates, from patients who received neoadjuvant chemotherapy (NAC) by using dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) obtained before and after NAC.

Materials and methods

This retrospective study included 75 patients who underwent NAC and breast surgery. First, 3,390 features were comprehensively extracted from pre- and post-NAC DCE-MRIs. Then patients were then divided into two groups: type 1, patients with pathologic complete response (pCR) and single lesion shrinkage; type 2, major residual lesion with satellite foci, multifocal residual, stable disease (SD), and progressive disease (PD). The logistic regression (LR) was used to build prediction models to identify the two groups. Prediction performance was assessed using the area under the curve (AUC), accuracy, sensitivity, and specificity.

Results

Radiomics features were significantly related to breast cancer shrinkage after NAC. The combination model achieved an AUC of 0.82, and the pre-NAC model was 0.64, the post-NAC model was 0.70, and the pre-post-NAC model was 0.80. In the combination model, 15 features, including nine wavelet-based features, four Laplacian-of-Gauss (LoG) features, and two original features, were filtered. Among these selected were four features from pre-NAC DCE-MRI, six were from post-NAC DCE-MRI, and five were from pre-post-NAC features.

Conclusion

The model combined with pre- and post-NAC DCE-MRI can effectively predict candidates to undergo BCS and provide AI-based decision support for clinicians with ensured safety. High-order (LoG- and wavelet-based) features play an important role in our machine learning model. The features from pre-post-NAC DCE-MRI had better predictive performance.

Deep learning applications to breast cancer detection by magnetic resonance imaging: a literature review

Deep learning analysis of radiological images has the potential to improve diagnostic accuracy of breast cancer, ultimately leading to better patient outcomes. This paper systematically reviewed the current literature on deep learning detection of breast cancer based on magnetic resonance imaging (MRI). The literature search was performed from 2015 to Dec 31, 2022, using Pubmed. Other database included Semantic Scholar, ACM Digital Library, Google search, Google Scholar, and pre-print depositories (such as Research Square). Articles that were not deep learning (such as texture analysis) were excluded. PRISMA guidelines for reporting were used. We analyzed different deep learning algorithms, methods of analysis, experimental design, MRI image types, types of ground truths, sample sizes, numbers of benign and malignant lesions, and performance in the literature. We discussed lessons learned, challenges to broad deployment in clinical practice and suggested future research directions.

Diagnostic efficiency of multi-modal MRI based deep learning with Sobel operator in differentiating benign and malignant breast mass lesions-a retrospective study

Purpose

To compare the diagnostic efficiencies of deep learning single-modal and multi-modal for the classification of benign and malignant breast mass lesions.

Methods

We retrospectively collected data from 203 patients (207 lesions, 101 benign and 106 malignant) with breast tumors who underwent breast magnetic resonance imaging (MRI) before surgery or biopsy between January 2014 and October 2020. Mass segmentation was performed based on the three dimensions-region of interest (3D-ROI) minimum bounding cube at the edge of the lesion. We established single-modal models based on a convolutional neural network (CNN) including T2WI and non-fs T1WI, the dynamic contrast-enhanced (DCE-MRI) first phase was pre-contrast T1WI (d1), and Phases 2, 4, and 6 were post-contrast T1WI (d2, d4, d6); and Multi-modal fusion models with a Sobel operator (four_mods:T2WI, non-fs-T1WI, d1, d2). Training set (n = 145), validation set (n = 22), and test set (n = 40). Five-fold cross validation was performed. Accuracy, sensitivity, specificity, negative predictive value, positive predictive value, and area under the ROC curve (AUC) were used as evaluation indicators. Delong's test compared the diagnostic performance of the multi-modal and single-modal models.

Results

All models showed good performance, and the AUC values were all greater than 0.750. Among the single-modal models, T2WI, non-fs-T1WI, d1, and d2 had specificities of 77.1%, 77.2%, 80.2%, and 78.2%, respectively. d2 had the highest accuracy of 78.5% and showed the best diagnostic performance with an AUC of 0.827. The multi-modal model with the Sobel operator performed better than single-modal models, with an AUC of 0.887, sensitivity of 79.8%, specificity of 86.1%, and positive prediction value of 85.6%. Delong's test showed that the diagnostic performance of the multi-modal fusion models was higher than that of the six single-modal models (T2WI, non-fs-T1WI, d1, d2, d4, d6); the difference was statistically significant (p = 0.043, 0.017, 0.006, 0.017, 0.020, 0.004, all were greater than 0.05).

Conclusions

Multi-modal fusion deep learning models with a Sobel operator had excellent diagnostic value in the classification of breast masses, and further increase the efficiency of diagnosis.

Ethics of Artificial Intelligence in Breast Imaging

There is great interest in the development of artificial intelligence (AI) applications for medical imaging in general and specifically in breast imaging. Because of the scale of application and the potential for harm, there has been a parallel interest in assuring that these new technologies are scrutinized and applied in ethical ways. The four principles of autonomy, beneficence, non-maleficence, and justice are widely accepted as a framework for bioethical analysis. We incorporate a fifth principle of explicability (adapted from Floridi and Cowls) because of the unique considerations of AI. We review definitions of each of these principles and provide examples of their practical application to breast imaging.

A review of the current state of the computer-aided diagnosis (CAD) systems for breast cancer diagnosis

Breast cancer is one of the most common cancers affecting females worldwide. Early detection and diagnosis of breast cancer may aid in timely treatment, reducing the mortality rate to a great extent. To diagnose breast cancer, computer-aided diagnosis (CAD) systems employ a variety of imaging modalities such as mammography, computerized tomography, magnetic resonance imaging, ultrasound, and histological imaging. CAD and breast-imaging specialists are in high demand for early detection and diagnosis. This system has the potential to enhance the partiality of traditional histopathological image analysis. This review aims to highlight the recent advancements and the current state of CAD systems for breast cancer detection using different modalities.

An artificial intelligence system using maximum intensity projection MR images facilitates classification of non-mass enhancement breast lesions

OBJECTIVES

To build an artificial intelligence (AI) system to classify benign and malignant non-mass enhancement (NME) lesions using maximum intensity projection (MIP) of early post-contrast subtracted breast MR images.

METHODS

This retrospective study collected 965 pure NME lesions (539 benign and 426 malignant) confirmed by histopathology or follow-up in 903 women. The 754 NME lesions acquired by one MR scanner were randomly split into the training set, validation set, and test set A (482/121/151 lesions). The 211 NME lesions acquired by another MR scanner were used as test set B. The AI system was developed using ResNet-50 with the axial and sagittal MIP images. One senior and one junior radiologist reviewed the MIP images of each case independently and rated its Breast Imaging Reporting and Data System category. The performance of the AI system and the radiologists was evaluated using the area under the receiver operating characteristic curve (AUC).

RESULTS

The AI system yielded AUCs of 0.859 and 0.816 in the test sets A and B, respectively. The AI system achieved comparable performance as the senior radiologist (p = 0.558, p = 0.041) and outperformed the junior radiologist (p < 0.001, p = 0.009) in both test sets A and B. After AI assistance, the AUC of the junior radiologist increased from 0.740 to 0.862 in test set A (p < 0.001) and from 0.732 to 0.843 in test set B (p < 0.001).

CONCLUSION

Our MIP-based AI system yielded good applicability in classifying NME lesions in breast MRI and can assist the junior radiologist achieve better performance.

KEY POINTS

• Our MIP-based AI system yielded good applicability in the dataset both from the same and a different MR scanner in predicting malignant NME lesions. • The AI system achieved comparable diagnostic performance with the senior radiologist and outperformed the junior radiologist. • This AI system can assist the junior radiologist achieve better performance in the classification of NME lesions in MRI.

Breast Cancer Classification on Multiparametric MRI - Increased Performance of Boosting Ensemble Methods

Introduction: This study aims to assess the utility of Boosting ensemble classification methods for increasing the diagnostic performance of multiparametric Magnetic Resonance Imaging (mpMRI) radiomic models, in differentiating benign and malignant breast lesions. Methods: The dataset includes mpMR images of 140 female patients with mass-like breast lesions (70 benign and 70 malignant), consisting of Dynamic Contrast Enhanced (DCE) and T2-weighted sequences, and the Apparent Diffusion Coefficient (ADC) calculated from the Diffusion Weighted Imaging (DWI) sequence. Tumor masks were manually defined in all consecutive slices of the respective MRI volumes and 3D radiomic features were extracted with the Pyradiomics package. Feature dimensionality reduction was based on statistical tests and the Boruta wrapper. Hierarchical Clustering on Spearman's rank correlation coefficients between features and Random Forest classification for obtaining feature importance, were implemented for selecting the final feature subset. Adaptive Boosting (AdaBoost), Gradient Boosting (GB), Extreme Gradient Boosting (XGBoost) and Light Gradient Boosting Machine (LightGBM) classifiers, were trained and tested with bootstrap validation in differentiating breast lesions. A Support Vector Machine (SVM) classifier was also exploited for comparison. The Receiver Operator Characteristic (ROC) curves and DeLong's test were utilized to evaluate the classification performances. Results: The final feature subset consisted of 5 features derived from the lesion shape and the first order histogram of DCE and ADC images volumes. XGboost and LGBM achieved statistically significantly higher average classification performances [AUC = 0.95 and 0.94 respectively], followed by Adaboost [AUC = 0.90], GB [AUC = 0.89] and SVM [AUC = 0.88]. Conclusion: Overall, the integration of Ensemble Learning methods within mpMRI radiomic analysis can improve the performance of computer-assisted diagnosis of breast cancer lesions.

Vascularity and Dynamic Contrast-Enhanced Breast Magnetic Resonance Imaging

Angiogenesis is a key step in the initiation and progression of an invasive breast cancer. High microvessel density by morphological characterization predicts metastasis and poor survival in women with invasive breast cancers. However, morphologic characterization is subject to variability and only can evaluate a limited portion of an invasive breast cancer. Consequently, breast Magnetic Resonance Imaging (MRI) is currently being evaluated to assess vascularity. Recently, through the new field of radiomics, dynamic contrast enhanced (DCE)-MRI is being used to evaluate vascular density, vascular morphology, and detection of aggressive breast cancer biology. While DCE-MRI is a highly sensitive tool, there are specific features that limit computational evaluation of blood vessels. These include (1) DCE-MRI evaluates gadolinium contrast and does not directly evaluate biology, (2) the resolution of DCE-MRI is insufficient for imaging small blood vessels, and (3) DCE-MRI images are very difficult to co-register. Here we review computational approaches for detection and analysis of blood vessels in DCE-MRI images and present some of the strategies we have developed for co-registry of DCE-MRI images and early detection of vascularization.

BI-RADS Reading of Non-Mass Lesions on DCE-MRI and Differential Diagnosis Performed by Radiomics and Deep Learning

Background

A wide variety of benign and malignant processes can manifest as non-mass enhancement (NME) in breast MRI. Compared to mass lesions, there are no distinct features that can be used for differential diagnosis. The purpose is to use the BI-RADS descriptors and models developed using radiomics and deep learning to distinguish benign from malignant NME lesions.

Materials and Methods

A total of 150 patients with 104 malignant and 46 benign NME were analyzed. Three radiologists performed reading for morphological distribution and internal enhancement using the 5th BI-RADS lexicon. For each case, the 3D tumor mask was generated using Fuzzy-C-Means segmentation. Three DCE parametric maps related to wash-in, maximum, and wash-out were generated, and PyRadiomics was applied to extract features. The radiomics model was built using five machine learning algorithms. ResNet50 was implemented using three parametric maps as input. Approximately 70% of earlier cases were used for training, and 30% of later cases were held out for testing.

Results

The diagnostic BI-RADS in the original MRI report showed that 104/104 malignant and 36/46 benign lesions had a BI-RADS score of 4A–5. For category reading, the kappa coefficient was 0.83 for morphological distribution (excellent) and 0.52 for internal enhancement (moderate). Segmental and Regional distribution were the most prominent for the malignant group, and focal distribution for the benign group. Eight radiomics features were selected by support vector machine (SVM). Among the five machine learning algorithms, SVM yielded the highest accuracy of 80.4% in training and 77.5% in testing datasets. ResNet50 had a better diagnostic performance, 91.5% in training and 83.3% in testing datasets.

Conclusion

Diagnosis of NME was challenging, and the BI-RADS scores and descriptors showed a substantial overlap. Radiomics and deep learning may provide a useful CAD tool to aid in diagnosis.

Radiomics in breast MRI: current progress toward clinical application in the era of artificial intelligence

Breast magnetic resonance imaging (MRI) is the most sensitive imaging modality for breast cancer diagnosis and is widely used clinically. Dynamic contrast-enhanced MRI is the basis for breast MRI, but ultrafast images, T2-weighted images, and diffusion-weighted images are also taken to improve the characteristics of the lesion. Such multiparametric MRI with numerous morphological and functional data poses new challenges to radiologists, and thus, new tools for reliable, reproducible, and high-volume quantitative assessments are warranted. In this context, radiomics, which is an emerging field of research involving the conversion of digital medical images into mineable data for clinical decision-making and outcome prediction, has been gaining ground in oncology. Recent development in artificial intelligence has promoted radiomics studies in various fields including breast cancer treatment and numerous studies have been conducted. However, radiomics has shown a translational gap in clinical practice, and many issues remain to be solved. In this review, we will outline the steps of radiomics workflow and investigate clinical application of radiomics focusing on breast MRI based on published literature, as well as current discussion about limitations and challenges in radiomics.

AI-enhanced breast imaging: Where are we and where are we heading?

Significant advances in imaging analysis and the development of high-throughput methods that can extract and correlate multiple imaging parameters with different clinical outcomes have led to a new direction in medical research. Radiomics and artificial intelligence (AI) studies are rapidly evolving and have many potential applications in breast imaging, such as breast cancer risk prediction, lesion detection and classification, radiogenomics, and prediction of treatment response and clinical outcomes. AI has been applied to different breast imaging modalities, including mammography, ultrasound, and magnetic resonance imaging, in different clinical scenarios. The application of AI tools in breast imaging has an unprecedented opportunity to better derive clinical value from imaging data and reshape the way we care for our patients. The aim of this study is to review the current knowledge and future applications of AI-enhanced breast imaging in clinical practice.

AI-enhanced simultaneous multiparametric 18F-FDG PET/MRI for accurate breast cancer diagnosis

Purpose

To assess whether a radiomics and machine learning (ML) model combining quantitative parameters and radiomics features extracted from simultaneous multiparametric ¹⁸F-FDG PET/MRI can discriminate between benign and malignant breast lesions.

Methods

A population of 102 patients with 120 breast lesions (101 malignant and 19 benign) detected on ultrasound and/or mammography was prospectively enrolled. All patients underwent hybrid ¹⁸F-FDG PET/MRI for diagnostic purposes. Quantitative parameters were extracted from DCE (MTT, VD, PF), DW (mean ADC of breast lesions and contralateral breast parenchyma), PET (SUVmax, SUVmean, and SUVminimum of breast lesions, as well as SUVmean of the contralateral breast parenchyma), and T2-weighted images. Radiomics features were extracted from DCE, T2-weighted, ADC, and PET images. Different diagnostic models were developed using a fine Gaussian support vector machine algorithm which explored different combinations of quantitative parameters and radiomics features to obtain the highest accuracy in discriminating between benign and malignant breast lesions using fivefold cross-validation. The performance of the best radiomics and ML model was compared with that of expert reader review using McNemar’s test.

Results

Eight radiomics models were developed. The integrated model combining MTT and ADC with radiomics features extracted from PET and ADC images obtained the highest accuracy for breast cancer diagnosis (AUC 0.983), although its accuracy was not significantly higher than that of expert reader review (AUC 0.868) (p = 0.508).

Conclusion

A radiomics and ML model combining quantitative parameters and radiomics features extracted from simultaneous multiparametric ¹⁸F-FDG PET/MRI images can accurately discriminate between benign and malignant breast lesions.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05492-z.

Artificial Intelligence and Big Data

Medical images play an important role in ophthalmology and radiology. Medical image analysis has greatly benefited from the application of "deep learning" techniques in clinical and experimental radiology. Clinical applications and their relevance for radiological imaging in ophthalmology are presented.

Automatic lesion detection, segmentation and characterization via 3D multiscale morphological sifting in breast MRI

Previous studies on computer aided detection/diagnosis (CAD) in 4D breast magnetic resonance imaging (MRI) usually regard lesion detection, segmentation and characterization as separate tasks, and typically require users to manually select 2D MRI slices or regions of interest as the input. In this work, we present a breast MRI CAD system that can handle 4D multimodal breast MRI data, and integrate lesion detection, segmentation and characterization with no user intervention. The proposed CAD system consists of three major stages: region candidate generation, feature extraction and region candidate classification. Breast lesions are firstly extracted as region candidates using the novel 3D multiscale morphological sifting (MMS). The 3D MMS, which uses linear structuring elements to extract lesion-like patterns, can segment lesions from breast images accurately and efficiently. Analytical features are then extracted from all available 4D multimodal breast MRI sequences, including T1-, T2-weighted and DCE sequences, to represent the signal intensity, texture, morphological and enhancement kinetic characteristics of the region candidates. The region candidates are lastly classified as lesion or normal tissue by the random under-sampling boost (RUSboost), and as malignant or benign lesion by the random forest. Evaluated on a breast MRI dataset which contains a total of 117 cases with 141 biopsy-proven lesions (95 malignant and 46 benign lesions), the proposed system achieves a true positive rate (TPR) of 0.90 at 3.19 false positives per patient (FPP) for lesion detection and a TPR of 0.91 at a FPP of 2.95 for identifying malignant lesions without any user intervention. The average dice similarity index (DSI) is0.72±0.15for lesion segmentation. Compared with previously proposed lesion detection, detection-segmentation and detection-characterization systems evaluated on the same breast MRI dataset, the proposed CAD system achieves a favourable performance in breast lesion detection and characterization.