Multi-input deep learning architecture for predicting breast tumor response to chemotherapy using quantitative MR images

Incorporating radiomic MRI models for presurgical response assessment in patients with early breast cancer undergoing neoadjuvant systemic therapy: Collaborative insights from breast oncologists and radiologists

The assessment of neoadjuvant treatment's response is critical for selecting the most suitable therapeutic options for patients with breast cancer to reduce the need for invasive local therapies. Breast magnetic resonance imaging (MRI) is so far one of the most accurate approaches for assessing pathological complete response, although this is limited by the qualitative and subjective nature of radiologists' assessment, often making it insufficient for deciding whether to forgo additional locoregional therapy measures. To increase the accuracy and prediction of radiomic MRI with the aid of machine learning models and deep learning methods, as part of artificial intelligence, have been used to analyse the different subtypes of breast cancer and the specific changes observed before and after therapy. This review discusses recent advancements in radiomic MRI models for presurgical response assessment for patients with early breast cancer receiving preoperative treatments, with a focus on their implications for clinical practice.

Integrating Machine Learning and Deep Learning for Predicting Non-Surgical Root Canal Treatment Outcomes Using Two-Dimensional Periapical Radiographs

Background/Objectives: In a previous study, we utilized categorical variables and machine learning (ML) algorithms to predict the success of non-surgical root canal treatments (NSRCTs) in apical periodontitis (AP), classifying the outcome as either success (healed) or failure (not healed). Given the importance of radiographic imaging in diagnosis, the present study evaluates the efficacy of deep learning (DL) in predicting NSRCT outcomes using two-dimensional (2D) periapical radiographs, comparing its performance with ML models. Methods: The DL model was trained and validated using leave-one-out cross-validation (LOOCV). Its output was incorporated into the set of categorical variables, and the ML study was reproduced using backward stepwise selection (BSS). The chi-square test was applied to assess the association between this new variable and NSRCT outcomes. Finally, after identifying the best-performing method from the ML study reproduction, statistical comparisons were conducted between this method, clinical professionals, and the image-based model using Fisher's exact test. Results: The association study yielded a p-value of 0.000000127, highlighting the predictive capability of 2D radiographs. After incorporating the DL-based predictive variable, the ML algorithm that demonstrated the best performance was logistic regression (LR), differing from the previous study, where random forest (RF) was the top performer. When comparing the deep learning-logistic regression (DL-LR) model with the clinician's prognosis (DP), DL-LR showed superior performance with a statistically significant difference (p-value < 0.05) in sensitivity, NPV, and accuracy. The same trend was observed in the DL vs. DP comparison. However, no statistically significant differences were found in the comparisons of RF vs. DL-LR, RF vs. DL, or DL vs. DL-LR. Conclusions: The findings of this study suggest that image-based artificial intelligence models exhibit superior predictive capability compared with those relying exclusively on categorical data. Moreover, they outperform clinician prognosis.

A deep learning approach for early prediction of breast cancer neoadjuvant chemotherapy response on multistage bimodal ultrasound images

Neoadjuvant chemotherapy (NAC) is a systemic and systematic chemotherapy regimen for breast cancer patients before surgery. However, NAC is not effective for everyone, and the process is excruciating. Therefore, accurate early prediction of the efficacy of NAC is essential for the clinical diagnosis and treatment of patients. In this study, a novel convolutional neural network model with bimodal layer-wise feature fusion module (BLFFM) and temporal hybrid attention module (THAM) is proposed, which uses multistage bimodal ultrasound images as input for early prediction of the efficacy of neoadjuvant chemotherapy in locally advanced breast cancer (LABC) patients. The BLFFM can effectively mine the highly complex correlation and complementary feature information between gray-scale ultrasound (GUS) and color Doppler blood flow imaging (CDFI). The THAM is able to focus on key features of lesion progression before and after one cycle of NAC. The GUS and CDFI videos of 101 patients collected from cooperative medical institutions were preprocessed to obtain 3000 sets of multistage bimodal ultrasound image combinations for experiments. The experimental results show that the proposed model is effective and outperforms the compared models. The code will be published on the https://github.com/jinzhuwei/BLTA-CNN .

The top 100 most-cited articles on artificial intelligence in breast radiology: a bibliometric analysis

INTRODUCTION

Artificial intelligence (AI) in radiology is a rapidly evolving field. In breast imaging, AI has already been applied in a real-world setting and multiple studies have been conducted in the area. The aim of this analysis is to identify the most influential publications on the topic of artificial intelligence in breast imaging.

METHODS

A retrospective bibliometric analysis was conducted on artificial intelligence in breast radiology using the Web of Science database. The search strategy involved searching for the keywords 'breast radiology' or 'breast imaging' and the various keywords associated with AI such as 'deep learning', 'machine learning,' and 'neural networks'.

RESULTS

From the top 100 list, the number of citations per article ranged from 30 to 346 (average 85). The highest cited article titled 'Artificial Neural Networks In Mammography-Application To Decision-Making In The Diagnosis Of Breast-Cancer' was published in Radiology in 1993. Eighty-three of the articles were published in the last 10 years. The journal with the greatest number of articles was Radiology (n = 22). The most common country of origin was the United States (n = 51). Commonly occurring topics published were the use of deep learning models for breast cancer detection in mammography or ultrasound, radiomics in breast cancer, and the use of AI for breast cancer risk prediction.

CONCLUSION

This study provides a comprehensive analysis of the top 100 most-cited papers on the subject of artificial intelligence in breast radiology and discusses the current most influential papers in the field.

CLINICAL RELEVANCE STATEMENT

This article provides a concise summary of the top 100 most-cited articles in the field of artificial intelligence in breast radiology. It discusses the most impactful articles and explores the recent trends and topics of research in the field.

KEY POINTS

Multiple studies have been conducted on AI in breast radiology. The most-cited article was published in the journal Radiology in 1993. This study highlights influential articles and topics on AI in breast radiology.

An explainable longitudinal multi-modal fusion model for predicting neoadjuvant therapy response in women with breast cancer

Multi-modal image analysis using deep learning (DL) lays the foundation for neoadjuvant treatment (NAT) response monitoring. However, existing methods prioritize extracting multi-modal features to enhance predictive performance, with limited consideration on real-world clinical applicability, particularly in longitudinal NAT scenarios with multi-modal data. Here, we propose the Multi-modal Response Prediction (MRP) system, designed to mimic real-world physician assessments of NAT responses in breast cancer. To enhance feasibility, MRP integrates cross-modal knowledge mining and temporal information embedding strategy to handle missing modalities and remain less affected by different NAT settings. We validated MRP through multi-center studies and multinational reader studies. MRP exhibited comparable robustness to breast radiologists, outperforming humans in predicting pathological complete response in the Pre-NAT phase (ΔAUROC 14% and 10% on in-house and external datasets, respectively). Furthermore, we assessed MRP's clinical utility impact on treatment decision-making. MRP may have profound implications for enrolment into NAT trials and determining surgery extensiveness.

Advancing personalized oncology: a systematic review on the integration of artificial intelligence in monitoring neoadjuvant treatment for breast cancer patients

PURPOSE

Despite suffering from the same disease, each patient exhibits a distinct microbiological profile and variable reactivity to prescribed treatments. Most doctors typically use a standardized treatment approach for all patients suffering from a specific disease. Consequently, the challenge lies in the effectiveness of this standardized treatment and in adapting it to each individual patient. Personalized medicine is an emerging field in which doctors use diagnostic tests to identify the most effective medical treatments for each patient. Prognosis, disease monitoring, and treatment planning rely on manual, error-prone methods. Artificial intelligence (AI) uses predictive techniques capable of automating prognostic and monitoring processes, thus reducing the error rate associated with conventional methods.

METHODS

This paper conducts an analysis of current literature, encompassing the period from January 2015 to 2023, based on Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA).

RESULTS

In assessing 25 pertinent studies concerning predicting neoadjuvant treatment (NAT) response in breast cancer (BC) patients, the studies explored various imaging modalities (Magnetic Resonance Imaging, Ultrasound, etc.), evaluating results based on accuracy, sensitivity, and area under the curve. Additionally, the technologies employed, such as machine learning (ML), deep learning (DL), statistics, and hybrid models, were scrutinized. The presentation of datasets used for predicting complete pathological response (PCR) was also considered.

CONCLUSION

This paper seeks to unveil crucial insights into the application of AI techniques in personalized oncology, particularly in the monitoring and prediction of responses to NAT for BC patients. Finally, the authors suggest avenues for future research into AI-based monitoring systems.

Explainable artificial intelligence in breast cancer detection and risk prediction: A systematic scoping review

With the advances in artificial intelligence (AI), data-driven algorithms are becoming increasingly popular in the medical domain. However, due to the nonlinear and complex behavior of many of these algorithms, decision-making by such algorithms is not trustworthy for clinicians and is considered a black-box process. Hence, the scientific community has introduced explainable artificial intelligence (XAI) to remedy the problem. This systematic scoping review investigates the application of XAI in breast cancer detection and risk prediction. We conducted a comprehensive search on Scopus, IEEE Explore, PubMed, and Google Scholar (first 50 citations) using a systematic search strategy. The search spanned from January 2017 to July 2023, focusing on peer-reviewed studies implementing XAI methods in breast cancer datasets. Thirty studies met our inclusion criteria and were included in the analysis. The results revealed that SHapley Additive exPlanations (SHAP) is the top model-agnostic XAI technique in breast cancer research in terms of usage, explaining the model prediction results, diagnosis and classification of biomarkers, and prognosis and survival analysis. Additionally, the SHAP model primarily explained tree-based ensemble machine learning models. The most common reason is that SHAP is model agnostic, which makes it both popular and useful for explaining any model prediction. Additionally, it is relatively easy to implement effectively and completely suits performant models, such as tree-based models. Explainable AI improves the transparency, interpretability, fairness, and trustworthiness of AI-enabled health systems and medical devices and, ultimately, the quality of care and outcomes.

A Survey on Explainable Artificial Intelligence (XAI) Techniques for Visualizing Deep Learning Models in Medical Imaging

The combination of medical imaging and deep learning has significantly improved diagnostic and prognostic capabilities in the healthcare domain. Nevertheless, the inherent complexity of deep learning models poses challenges in understanding their decision-making processes. Interpretability and visualization techniques have emerged as crucial tools to unravel the black-box nature of these models, providing insights into their inner workings and enhancing trust in their predictions. This survey paper comprehensively examines various interpretation and visualization techniques applied to deep learning models in medical imaging. The paper reviews methodologies, discusses their applications, and evaluates their effectiveness in enhancing the interpretability, reliability, and clinical relevance of deep learning models in medical image analysis.

Deep learning analysis of serial digital breast tomosynthesis images in a prospective cohort of breast cancer patients who received neoadjuvant chemotherapy

PURPOSE

Different imaging tools, including digital breast tomosynthesis (DBT), are frequently used for evaluating tumor response during neoadjuvant chemotherapy (NACT). This study aimed to explore whether using artificial intelligence (AI) for serial DBT acquisitions during NACT for breast cancer can predict pathological complete response (pCR) after completion of NACT.

METHODS

A total of 149 women (mean age 53 years, pCR rate 22 %) with breast cancer treated with NACT at Skane University Hospital, Sweden, between 2014 and 2019, were prospectively included in this observational cohort study (ClinicalTrials.gov: NCT02306096). DBT images from both the cancer and contralateral healthy breasts acquired at three time points: pre-NACT, after two cycles of NACT, and after the completion of six cycles of NACT (pre-surgery) were analyzed. The deep learning AI system used to predict pCR consisted of a backbone 3D ResNet and an attention and prediction module. The GradCAM method was used to obtain insights into the model decision basis through a quantitative analysis of the importance maps on the validation set. Moreover, specific model choices were motivated by ablation studies.

RESULTS

The AI model reached an AUC of 0.83 (95% CI: 0.63-1.00) (test set). The spatial correlation of importance maps for input volumes from the same patient but at different time points was high, possibly indicating that the model focuses on the same areas during decision-making.

CONCLUSIONS

We demonstrate a high discriminative performance of our algorithm for predicting pCR/non-pCR. Availability of larger datasets would permit more comprehensive training of the models and more rigorous evaluation of their prediction performance for future patients.

Anti-HER2 therapy response assessment for guiding treatment (de-)escalation in early HER2-positive breast cancer using a novel deep learning radiomics model

OBJECTIVES

Anti-HER2 targeted therapy significantly reduces risk of relapse in HER2 + breast cancer. New measures are needed for a precise risk stratification to guide (de-)escalation of anti-HER2 strategy.

METHODS

A total of 726 HER2 + cases who received no/single/dual anti-HER2 targeted therapies were split into three respective cohorts. A deep learning model (DeepTEPP) based on preoperative breast magnetic resonance (MR) was developed. Patients were scored and categorized into low-, moderate-, and high-risk groups. Recurrence-free survival (RFS) was compared in patients with different risk groups according to the anti-HER2 treatment they received, to validate the value of DeepTEPP in predicting treatment efficacy and guiding anti-HER2 strategy.

RESULTS

DeepTEPP was capable of risk stratification and guiding anti-HER2 treatment strategy: DeepTEPP-Low patients (60.5%) did not derive significant RFS benefit from trastuzumab (p = 0.144), proposing an anti-HER2 de-escalation. DeepTEPP-Moderate patients (19.8%) significantly benefited from trastuzumab (p = 0.048), but did not obtain additional improvements from pertuzumab (p = 0.125). DeepTEPP-High patients (19.7%) significantly benefited from dual HER2 blockade (p = 0.045), suggesting an anti-HER2 escalation.

CONCLUSIONS

DeepTEPP represents a pioneering MR-based deep learning model that enables the non-invasive prediction of adjuvant anti-HER2 effectiveness, thereby providing valuable guidance for anti-HER2 (de-)escalation strategies. DeepTEPP provides an important reference for choosing the appropriate individualized treatment in HER2 + breast cancer patients, warranting prospective validation.

CLINICAL RELEVANCE STATEMENT

We built an MR-based deep learning model DeepTEPP, which enables the non-invasive prediction of adjuvant anti-HER2 effectiveness, thus guiding anti-HER2 (de-)escalation strategies in early HER2-positive breast cancer patients.

KEY POINTS

• DeepTEPP is able to predict anti-HER2 effectiveness and to guide treatment (de-)escalation. • DeepTEPP demonstrated an impressive prognostic efficacy for recurrence-free survival and overall survival. • To our knowledge, this is one of the very few, also the largest study to test the efficacy of a deep learning model extracted from breast MR images on HER2-positive breast cancer survival and anti-HER2 therapy effectiveness prediction.

Modality-Specific Information Disentanglement From Multi-Parametric MRI for Breast Tumor Segmentation and Computer-Aided Diagnosis

Breast cancer is becoming a significant global health challenge, with millions of fatalities annually. Magnetic Resonance Imaging (MRI) can provide various sequences for characterizing tumor morphology and internal patterns, and becomes an effective tool for detection and diagnosis of breast tumors. However, previous deep-learning based tumor segmentation methods from multi-parametric MRI still have limitations in exploring inter-modality information and focusing task-informative modality/modalities. To address these shortcomings, we propose a Modality-Specific Information Disentanglement (MoSID) framework to extract both inter- and intra-modality attention maps as prior knowledge for guiding tumor segmentation. Specifically, by disentangling modality-specific information, the MoSID framework provides complementary clues for the segmentation task, by generating modality-specific attention maps to guide modality selection and inter-modality evaluation. Our experiments on two 3D breast datasets and one 2D prostate dataset demonstrate that the MoSID framework outperforms other state-of-the-art multi-modality segmentation methods, even in the cases of missing modalities. Based on the segmented lesions, we further train a classifier to predict the patients' response to radiotherapy. The prediction accuracy is comparable to the case of using manually-segmented tumors for treatment outcome prediction, indicating the robustness and effectiveness of the proposed segmentation method. The code is available at https://github.com/Qianqian-Chen/MoSID.

Artificial Intelligence-Enhanced Breast MRI: Applications in Breast Cancer Primary Treatment Response Assessment and Prediction

ABSTRACT

Primary systemic therapy (PST) is the treatment of choice in patients with locally advanced breast cancer and is nowadays also often used in patients with early-stage breast cancer. Although imaging remains pivotal to assess response to PST accurately, the use of imaging to predict response to PST has the potential to not only better prognostication but also allow the de-escalation or omission of potentially toxic treatment with undesirable adverse effects, the accelerated implementation of new targeted therapies, and the mitigation of surgical delays in selected patients. In response to the limited ability of radiologists to predict response to PST via qualitative, subjective assessments of tumors on magnetic resonance imaging (MRI), artificial intelligence-enhanced MRI with classical machine learning, and in more recent times, deep learning, have been used with promising results to predict response, both before the start of PST and in the early stages of treatment. This review provides an overview of the current applications of artificial intelligence to MRI in assessing and predicting response to PST, and discusses the challenges and limitations of their clinical implementation.

Deep Learning-Based Segmentation of Locally Advanced Breast Cancer on MRI in Relation to Residual Cancer Burden: A Multi-Institutional Cohort Study

BACKGROUND

While several methods have been proposed for automated assessment of breast-cancer response to neoadjuvant chemotherapy on breast MRI, limited information is available about their performance across multiple institutions.

PURPOSE

To assess the value and robustness of deep learning-derived volumes of locally advanced breast cancer (LABC) on MRI to infer the presence of residual disease after neoadjuvant chemotherapy.

STUDY TYPE

Retrospective.

SUBJECTS

Training cohort: 102 consecutive female patients with LABC scheduled for neoadjuvant chemotherapy (NAC) from a single institution (age: 25-73 years). Independent testing cohort: 55 consecutive female patients with LABC from four institutions (age: 25-72 years).

FIELD STRENGTH/SEQUENCE

Training cohort: single vendor 1.5 T or 3.0 T. Testing cohort: multivendor 3.0 T. Gradient echo dynamic contrast-enhanced sequences.

ASSESSMENT

A convolutional neural network (nnU-Net) was trained to segment LABC. Based on resulting tumor volumes, an extremely randomized tree model was trained to assess residual cancer burden (RCB)-0/I vs. RCB-II/III. An independent model was developed using functional tumor volume (FTV). Models were tested on an independent testing cohort and response assessment performance and robustness across multiple institutions were assessed.

STATISTICAL TESTS

The receiver operating characteristic (ROC) was used to calculate the area under the ROC curve (AUC). DeLong's method was used to compare AUCs. Correlations were calculated using Pearson's method. P values <0.05 were considered significant.

RESULTS

Automated segmentation resulted in a median (interquartile range [IQR]) Dice score of 0.87 (0.62-0.93), with similar volumetric measurements (R = 0.95, P < 0.05). Automated volumetric measurements were significantly correlated with FTV (R = 0.80). Tumor volume-derived from deep learning of DCE-MRI was associated with RCB, yielding an AUC of 0.76 to discriminate between RCB-0/I and RCB-II/III, performing similar to the FTV-based model (AUC = 0.77, P = 0.66). Performance was comparable across institutions (IQR AUC: 0.71-0.84).

DATA CONCLUSION

Deep learning-based segmentation estimates changes in tumor load on DCE-MRI that are associated with RCB after NAC and is robust against variations between institutions.

EVIDENCE LEVEL

2.

TECHNICAL EFFICACY

Stage 4.

Exploring Neoadjuvant Chemotherapy, Predictive Models, Radiomic, and Pathological Markers in Breast Cancer: A Comprehensive Review

Breast cancer retains its position as the most prevalent form of malignancy among females on a global scale. The careful selection of appropriate treatment for each patient holds paramount importance in effectively managing breast cancer. Neoadjuvant chemotherapy (NACT) plays a pivotal role in the comprehensive treatment of this disease. Administering chemotherapy before surgery, NACT becomes a powerful tool in reducing tumor size, potentially enabling fewer invasive surgical procedures and even rendering initially inoperable tumors amenable to surgery. However, a significant challenge lies in the varying responses exhibited by different patients towards NACT. To address this challenge, researchers have focused on developing prediction models that can identify those who would benefit from NACT and those who would not. Such models have the potential to reduce treatment costs and contribute to a more efficient and accurate management of breast cancer. Therefore, this review has two objectives: first, to identify the most effective radiomic markers correlated with NACT response, and second, to explore whether integrating radiomic markers extracted from radiological images with pathological markers can enhance the predictive accuracy of NACT response. This review will delve into addressing these research questions and also shed light on the emerging research direction of leveraging artificial intelligence techniques for predicting NACT response, thereby shaping the future landscape of breast cancer treatment.

Deep Learning for Fully Automatic Tumor Segmentation on Serially Acquired Dynamic Contrast-Enhanced MRI Images of Triple-Negative Breast Cancer

Accurate tumor segmentation is required for quantitative image analyses, which are increasingly used for evaluation of tumors. We developed a fully automated and high-performance segmentation model of triple-negative breast cancer using a self-configurable deep learning framework and a large set of dynamic contrast-enhanced MRI images acquired serially over the patients' treatment course. Among all models, the top-performing one that was trained with the images across different time points of a treatment course yielded a Dice similarity coefficient of 93% and a sensitivity of 96% on baseline images. The top-performing model also produced accurate tumor size measurements, which is valuable for practical clinical applications.

Predicting pathological complete response to neoadjuvant systemic therapy for triple-negative breast cancers using deep learning on multiparametric MRIs

We trained and validated a deep learning model that can predict the treatment response to neoadjuvant systemic therapy (NAST) for patients with triple negative breast cancer (TNBC). Dynamic contrast enhanced (DCE) MRI and diffusion-weighted imaging (DWI) of the pre-treatment (baseline) and after four cycles (C4) of doxorubicin/cyclophosphamide treatment were used as inputs to the model for prediction of pathologic complete response (pCR). Based on the standard pCR definition that includes disease status in either breast or axilla, the model achieved areas under the receiver operating characteristic curves (AUCs) of 0.96 ± 0.05, 0.78 ± 0.09, 0.88 ± 0.02, and 0.76 ± 0.03, for the training, validation, testing, and prospective testing groups, respectively. For the pCR status of breast only, the retrained model achieved prediction AUCs of 0.97 ± 0.04, 0.82 ± 0.10, 0.86 ± 0.03, and 0.83 ± 0.02, for the training, validation, testing, and prospective testing groups, respectively. Thus, the developed deep learning model is highly promising for predicting the treatment response to NAST of TNBC.Clinical Relevance- Deep learning based on serial and multiparametric MRIs can potentially distinguish TNBC patients with pCR from non-pCR at the early stage of neoadjuvant systemic therapy, potentially enabling more personalized treatment of TNBC patients.

Predicting pathological complete response based on weakly and semi-supervised joint learning from breast cancer MRI

Neoadjuvant chemotherapy (NAC) is the standard treatment for breast cancer patients. Patients achieving complete pathological response (pCR) after NAC usually have a good prognosis. However, automatic pCR prediction has been a challenging problem due to lacking well annotations in 3D MRI. Thus far, unifying different annotation information to predict the tumor's early response to NAC has not been systematically addressed. This paper proposes a weakly and semi-supervised joint learning method that integrates attentional features from multi-parametric MRI with radiomic features for predicting pCR to NAC in breast cancer patients. The attention-based multi-instance learning (MIL) is first developed to generate informative MRI bag-level features and mine key instances. The mean-teacher framework is then employed to segment tumor regions in a semi-supervised setting for extracting radiomic features. We perform experiments on 442 patients' data and show that our method achieves an AUC value of 0.85 in pCR prediction, which is superior to comparative methods. It is also shown that learning from multi-parametric MRI outperforms that of single-parameter MRI in pCR prediction.

Recent Advances in Explainable Artificial Intelligence for Magnetic Resonance Imaging

Advances in artificial intelligence (AI), especially deep learning (DL), have facilitated magnetic resonance imaging (MRI) data analysis, enabling AI-assisted medical image diagnoses and prognoses. However, most of the DL models are considered as "black boxes". There is an unmet need to demystify DL models so domain experts can trust these high-performance DL models. This has resulted in a sub-domain of AI research called explainable artificial intelligence (XAI). In the last decade, many experts have dedicated their efforts to developing novel XAI methods that are competent at visualizing and explaining the logic behind data-driven DL models. However, XAI techniques are still in their infancy for medical MRI image analysis. This study aims to outline the XAI applications that are able to interpret DL models for MRI data analysis. We first introduce several common MRI data modalities. Then, a brief history of DL models is discussed. Next, we highlight XAI frameworks and elaborate on the principles of multiple popular XAI methods. Moreover, studies on XAI applications in MRI image analysis are reviewed across the tissues/organs of the human body. A quantitative analysis is conducted to reveal the insights of MRI researchers on these XAI techniques. Finally, evaluations of XAI methods are discussed. This survey presents recent advances in the XAI domain for explaining the DL models that have been utilized in MRI applications.

An Endodontic forecasting model based on the analysis of preoperative dental radiographs: A pilot study on an endodontic predictive deep neural network

INTRODUCTION

This study aimed to evaluate the use of deep convolutional neural network (DCNN) algorithms to detect clinical features and predict the three years outcome of endodontic treatment on preoperative periapical radiographs.

METHODS

A database of single-root premolars that received endodontic treatment or retreatment by endodontists with presence of three years outcome was prepared (n = 598). We constructed a 17-layered DCNN with a self-attention layer (PRESSAN-17), and the model was trained, validated, and tested to 1) detect seven clinical features, i.e., full coverage restoration (FCR), presence of proximal teeth (PRX), coronal defect (COD), root rest (RRS), canal visibility (CAV), previous root filling (PRF), and periapical radiolucency (PAR), and 2) predict the three years endodontic prognosis by analyzing preoperative periapical radiographs as an input. During the prognostication test, a conventional DCNN without a self-attention layer (RESNET-18) was tested for comparison. Accuracy and area under the receiver-operating-characteristic (ROC) curve (AUC) were mainly evaluated for performance comparison. Gradient-weighted class activation mapping (Grad-CAM) was used to visualize weighted heatmaps.

RESULTS

PRESSAN-17 detected FCR (AUC = 0.975), PRX (0.866), COD (0.672), RRS (0.989), PRF (0.879) and PAR (0.690) significantly, compared to the no-information rate (p<0.05). Comparing the mean accuracy of 5-fold validation of two models, PRESSAN-17 (67.0%) showed a significant difference to RESNET-18 (63.4%, p<0.05). Also, the area under average ROC of PRESSAN-17 was 0.638, which was significantly different compared to the no-information rate. Grad-CAM demonstrated that PRESSAN-17 correctly identified clinical features.

CONCLUSIONS

Deep convolutional neural networks may aid in the prognostication of endodontic treatment outcome.

Survey of explainable artificial intelligence techniques for biomedical imaging with deep neural networks

Artificial Intelligence (AI) techniques of deep learning have revolutionized the disease diagnosis with their outstanding image classification performance. In spite of the outstanding results, the widespread adoption of these techniques in clinical practice is still taking place at a moderate pace. One of the major hindrance is that a trained Deep Neural Networks (DNN) model provides a prediction, but questions about why and how that prediction was made remain unanswered. This linkage is of utmost importance for the regulated healthcare domain to increase the trust in the automated diagnosis system by the practitioners, patients and other stakeholders. The application of deep learning for medical imaging has to be interpreted with caution due to the health and safety concerns similar to blame attribution in the case of an accident involving autonomous cars. The consequences of both a false positive and false negative cases are far reaching for patients' welfare and cannot be ignored. This is exacerbated by the fact that the state-of-the-art deep learning algorithms comprise of complex interconnected structures, millions of parameters, and a 'black box' nature, offering little understanding of their inner working unlike the traditional machine learning algorithms. Explainable AI (XAI) techniques help to understand model predictions which help develop trust in the system, accelerate the disease diagnosis, and meet adherence to regulatory requirements. This survey provides a comprehensive review of the promising field of XAI for biomedical imaging diagnostics. We also provide a categorization of the XAI techniques, discuss the open challenges, and provide future directions for XAI which would be of interest to clinicians, regulators and model developers.

Clinical applications of deep learning in breast MRI

Deep learning (DL) is one of the most powerful data-driven machine-learning techniques in artificial intelligence (AI). It can automatically learn from raw data without manual feature selection. DL models have led to remarkable advances in data extraction and analysis for medical imaging. Magnetic resonance imaging (MRI) has proven useful in delineating the characteristics and extent of breast lesions and tumors. This review summarizes the current state-of-the-art applications of DL models in breast MRI. Many recent DL models were examined in this field, along with several advanced learning approaches and methods for data normalization and breast and lesion segmentation. For clinical applications, DL-based breast MRI models were proven useful in five aspects: diagnosis of breast cancer, classification of molecular types, classification of histopathological types, prediction of neoadjuvant chemotherapy response, and prediction of lymph node metastasis. For subsequent studies, further improvement in data acquisition and preprocessing is necessary, additional DL techniques in breast MRI should be investigated, and wider clinical applications need to be explored.

Prediction of pathologic complete response to neoadjuvant systemic therapy in triple negative breast cancer using deep learning on multiparametric MRI

Triple-negative breast cancer (TNBC) is an aggressive subtype of breast cancer. Neoadjuvant systemic therapy (NAST) followed by surgery are currently standard of care for TNBC with 50-60% of patients achieving pathologic complete response (pCR). We investigated ability of deep learning (DL) on dynamic contrast enhanced (DCE) MRI and diffusion weighted imaging acquired early during NAST to predict TNBC patients' pCR status in the breast. During the development phase using the images of 130 TNBC patients, the DL model achieved areas under the receiver operating characteristic curves (AUCs) of 0.97 ± 0.04 and 0.82 ± 0.10 for the training and the validation, respectively. The model achieved an AUC of 0.86 ± 0.03 when evaluated in the independent testing group of 32 patients. In an additional prospective blinded testing group of 48 patients, the model achieved an AUC of 0.83 ± 0.02. These results demonstrated that DL based on multiparametric MRI can potentially differentiate TNBC patients with pCR or non-pCR in the breast early during NAST.

Deep Learning Prediction of Pathologic Complete Response in Breast Cancer Using MRI and Other Clinical Data: A Systematic Review

Breast cancer patients who have pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) are more likely to have better clinical outcomes. The ability to predict which patient will respond to NAC early in the treatment course is important because it could help to minimize unnecessary toxic NAC and to modify regimens mid-treatment to achieve better efficacy. Machine learning (ML) is increasingly being used in radiology and medicine because it can identify relationships amongst complex data elements to inform outcomes without the need to specify such relationships a priori. One of the most popular deep learning methods that applies to medical images is the Convolutional Neural Networks (CNN). In contrast to supervised ML, deep learning CNN can operate on the whole images without requiring radiologists to manually contour the tumor on images. Although there have been many review papers on supervised ML prediction of pCR, review papers on deep learning prediction of pCR are sparse. Deep learning CNN could also incorporate multiple image types, clinical data such as demographics and molecular subtypes, as well as data from multiple treatment time points to predict pCR. The goal of this study is to perform a systematic review of deep learning methods that use whole-breast MRI images without annotation or tumor segmentation to predict pCR in breast cancer.

MRI-Based Digital Models Forecast Patient-Specific Treatment Responses to Neoadjuvant Chemotherapy in Triple-Negative Breast Cancer

Triple-negative breast cancer (TNBC) is persistently refractory to therapy, and methods to improve targeting and evaluation of responses to therapy in this disease are needed. Here, we integrate quantitative MRI data with biologically based mathematical modeling to accurately predict the response of TNBC to neoadjuvant systemic therapy (NAST) on an individual basis. Specifically, 56 patients with TNBC enrolled in the ARTEMIS trial (NCT02276443) underwent standard-of-care doxorubicin/cyclophosphamide (A/C) and then paclitaxel for NAST, where dynamic contrast-enhanced MRI and diffusion-weighted MRI were acquired before treatment and after two and four cycles of A/C. A biologically based model was established to characterize tumor cell movement, proliferation, and treatment-induced cell death. Two evaluation frameworks were investigated using: (i) images acquired before and after two cycles of A/C for calibration and predicting tumor status after A/C, and (ii) images acquired before, after two cycles, and after four cycles of A/C for calibration and predicting response following NAST. For Framework 1, the concordance correlation coefficients between the predicted and measured patient-specific, post-A/C changes in tumor cellularity and volume were 0.95 and 0.94, respectively. For Framework 2, the biologically based model achieved an area under the receiver operator characteristic curve of 0.89 (sensitivity/specificity = 0.72/0.95) for differentiating pathological complete response (pCR) from non-pCR, which is statistically superior (P &lt; 0.05) to the value of 0.78 (sensitivity/specificity = 0.72/0.79) achieved by tumor volume measured after four cycles of A/C. Overall, this model successfully captured patient-specific, spatiotemporal dynamics of TNBC response to NAST, providing highly accurate predictions of NAST response.

SIGNIFICANCE

Integrating MRI data with biologically based mathematical modeling successfully predicts breast cancer response to chemotherapy, suggesting digital twins could facilitate a paradigm shift from simply assessing response to predicting and optimizing therapeutic efficacy.

Development and validation of a deep learning model for breast lesion segmentation and characterization in multiparametric MRI

Importance

The utilization of artificial intelligence for the differentiation of benign and malignant breast lesions in multiparametric MRI (mpMRI) assists radiologists to improve diagnostic performance.

Objectives

To develop an automated deep learning model for breast lesion segmentation and characterization and to evaluate the characterization performance of AI models and radiologists.

Materials and methods

For lesion segmentation, 2,823 patients were used for the training, validation, and testing of the VNet-based segmentation models, and the average Dice similarity coefficient (DSC) between the manual segmentation by radiologists and the mask generated by VNet was calculated. For lesion characterization, 3,303 female patients with 3,607 pathologically confirmed lesions (2,213 malignant and 1,394 benign lesions) were used for the three ResNet-based characterization models (two single-input and one multi-input models). Histopathology was used as the diagnostic criterion standard to assess the characterization performance of the AI models and the BI-RADS categorized by the radiologists, in terms of sensitivity, specificity, accuracy, and the area under the receiver operating characteristic curve (AUC). An additional 123 patients with 136 lesions (81 malignant and 55 benign lesions) from another institution were available for external testing.

Results

Of the 5,811 patients included in the study, the mean age was 46.14 (range 11–89) years. In the segmentation task, a DSC of 0.860 was obtained between the VNet-generated mask and manual segmentation by radiologists. In the characterization task, the AUCs of the multi-input and the other two single-input models were 0.927, 0.821, and 0.795, respectively. Compared to the single-input DWI or DCE model, the multi-input DCE and DWI model obtained a significant increase in sensitivity, specificity, and accuracy (0.831 vs. 0.772/0.776, 0.874 vs. 0.630/0.709, 0.846 vs. 0.721/0.752). Furthermore, the specificity of the multi-input model was higher than that of the radiologists, whether using BI-RADS category 3 or 4 as a cutoff point (0.874 vs. 0.404/0.841), and the accuracy was intermediate between the two assessment methods (0.846 vs. 0.773/0.882). For the external testing, the performance of the three models remained robust with AUCs of 0.812, 0.831, and 0.885, respectively.

Conclusions

Combining DCE with DWI was superior to applying a single sequence for breast lesion characterization. The deep learning computer-aided diagnosis (CADx) model we developed significantly improved specificity and achieved comparable accuracy to the radiologists with promise for clinical application to provide preliminary diagnoses.

Evaluation of cancer outcome assessment using MRI: A review of deep-learning methods

Accurate evaluation of tumor response to treatment is critical to allow personalized treatment regimens according to the predicted response and to support clinical trials investigating new therapeutic agents by providing them with an accurate response indicator. Recent advances in medical imaging, computer hardware, and machine-learning algorithms have resulted in the increased use of these tools in the field of medicine as a whole and specifically in cancer imaging for detection and characterization of malignant lesions, prognosis, and assessment of treatment response. Among the currently available imaging techniques, magnetic resonance imaging (MRI) plays an important role in the evaluation of treatment assessment of many cancers, given its superior soft-tissue contrast and its ability to allow multiplanar imaging and functional evaluation. In recent years, deep learning (DL) has become an active area of research, paving the way for computer-assisted clinical and radiological decision support. DL can uncover associations between imaging features that cannot be visually identified by the naked eye and pertinent clinical outcomes. The aim of this review is to highlight the use of DL in the evaluation of tumor response assessed on MRI. In this review, we will first provide an overview of common DL architectures used in medical imaging research in general. Then, we will review the studies to date that have applied DL to magnetic resonance imaging for the task of treatment response assessment. Finally, we will discuss the challenges and opportunities of using DL within the clinical workflow.

A Survey on Deep Learning for Precision Oncology

Precision oncology, which ensures optimized cancer treatment tailored to the unique biology of a patient’s disease, has rapidly developed and is of great clinical importance. Deep learning has become the main method for precision oncology. This paper summarizes the recent deep-learning approaches relevant to precision oncology and reviews over 150 articles within the last six years. First, we survey the deep-learning approaches categorized by various precision oncology tasks, including the estimation of dose distribution for treatment planning, survival analysis and risk estimation after treatment, prediction of treatment response, and patient selection for treatment planning. Secondly, we provide an overview of the studies per anatomical area, including the brain, bladder, breast, bone, cervix, esophagus, gastric, head and neck, kidneys, liver, lung, pancreas, pelvis, prostate, and rectum. Finally, we highlight the challenges and discuss potential solutions for future research directions.

Robustness Evaluation of a Deep Learning Model on Sagittal and Axial Breast DCE-MRIs to Predict Pathological Complete Response to Neoadjuvant Chemotherapy

To date, some artificial intelligence (AI) methods have exploited Dynamic Contrast-Enhanced Magnetic Resonance Imaging (DCE-MRI) to identify finer tumor properties as potential earlier indicators of pathological Complete Response (pCR) in breast cancer patients undergoing neoadjuvant chemotherapy (NAC). However, they work either for sagittal or axial MRI protocols. More flexible AI tools, to be used easily in clinical practice across various institutions in accordance with its own imaging acquisition protocol, are required. Here, we addressed this topic by developing an AI method based on deep learning in giving an early prediction of pCR at various DCE-MRI protocols (axial and sagittal). Sagittal DCE-MRIs refer to 151 patients (42 pCR; 109 non-pCR) from the public I-SPY1 TRIAL database (DB); axial DCE-MRIs are related to 74 patients (22 pCR; 52 non-pCR) from a private DB provided by Istituto Tumori “Giovanni Paolo II” in Bari (Italy). By merging the features extracted from baseline MRIs with some pre-treatment clinical variables, accuracies of 84.4% and 77.3% and AUC values of 80.3% and 78.0% were achieved on the independent tests related to the public DB and the private DB, respectively. Overall, the presented method has shown to be robust regardless of the specific MRI protocol.

Deep learning in breast imaging

Millions of breast imaging exams are performed each year in an effort to reduce the morbidity and mortality of breast cancer. Breast imaging exams are performed for cancer screening, diagnostic work-up of suspicious findings, evaluating extent of disease in recently diagnosed breast cancer patients, and determining treatment response. Yet, the interpretation of breast imaging can be subjective, tedious, time-consuming, and prone to human error. Retrospective and small reader studies suggest that deep learning (DL) has great potential to perform medical imaging tasks at or above human-level performance, and may be used to automate aspects of the breast cancer screening process, improve cancer detection rates, decrease unnecessary callbacks and biopsies, optimize patient risk assessment, and open up new possibilities for disease prognostication. Prospective trials are urgently needed to validate these proposed tools, paving the way for real-world clinical use. New regulatory frameworks must also be developed to address the unique ethical, medicolegal, and quality control issues that DL algorithms present. In this article, we review the basics of DL, describe recent DL breast imaging applications including cancer detection and risk prediction, and discuss the challenges and future directions of artificial intelligence-based systems in the field of breast cancer.

Dual-Branch Convolutional Neural Network Based on Ultrasound Imaging in the Early Prediction of Neoadjuvant Chemotherapy Response in Patients With Locally Advanced Breast Cancer

The early prediction of a patient's response to neoadjuvant chemotherapy (NAC) in breast cancer treatment is crucial for guiding therapy decisions. We aimed to develop a novel approach, named the dual-branch convolutional neural network (DBNN), based on deep learning that uses ultrasound (US) images for the early prediction of NAC response in patients with locally advanced breast cancer (LABC). This retrospective study included 114 women who were monitored with US during pretreatment (NAC pre) and after one cycle of NAC (NAC1). Pathologic complete response (pCR) was defined as no residual invasive carcinoma in the breast. For predicting pCR, the data were randomly split into a training set and test set (4:1). DBNN with US images was proposed to predict pCR early in breast cancer patients who received NAC. The connection between pretreatment data and data obtained after the first cycle of NAC was considered through the feature sharing of different branches. Moreover, the importance of data in various stages was emphasized by changing the weight of the two paths to classify those with pCR. The optimal model architecture of DBNN was determined by two ablation experiments. The diagnostic performance of DBNN for predicting pCR was compared with that of four methods from the latest research. To further validate the potential of DBNN in the early prediction of NAC response, the data from NAC pre and NAC1 were separately assessed. In the prediction of pCR, the highest diagnostic performance was obtained when combining the US image information of NAC pre and NAC1 (area under the receiver operating characteristic curve (AUC): 0.939; 95% confidence interval (CI): 0.907, 0.972; F1-score: 0.850; overall accuracy: 87.5%; sensitivity: 90.67%; and specificity: 85.67%), and the diagnostic performance with the combined data was superior to the performance when only NAC pre (AUC: 0.730; 95% CI: 0.657, 0.802; F1-score: 0.675; sensitivity: 76.00%; and specificity: 68.38%) or NAC1 (AUC: 0.739; 95% CI: 0.664, 0.813; F1-score: 0.611; sensitivity: 53.33%; and specificity: 86.32%) (p<0.01) was used. As a noninvasive prediction tool, DBNN can achieve outstanding results in the early prediction of NAC response in patients with LABC when combining the US data of NAC pre and NAC1.

Pretreatment DCE-MRI-Based Deep Learning Outperforms Radiomics Analysis in Predicting Pathologic Complete Response to Neoadjuvant Chemotherapy in Breast Cancer

PURPOSE

To compare the performances of deep learning (DL) to radiomics analysis (RA) in predicting pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) based on pretreatment dynamic contrast-enhanced MRI (DCE-MRI) in breast cancer.

MATERIALS AND METHODS

This retrospective study included 356 breast cancer patients who underwent DCE-MRI before NAC and underwent surgery after NAC. Image features and kinetic parameters of tumors were derived from DCE-MRI. Molecular information was assessed based on immunohistochemistry results. The image-based RA and DL models were constructed by adding kinetic parameters or molecular information to image-only linear discriminant analysis (LDA) and convolutional neural network (CNN) models. The predictive performances of developed models were assessed by receiver operating characteristic (ROC) curve analysis and compared with the DeLong method.

RESULTS

The overall pCR rate was 23.3% (83/356). The area under the ROC (AUROC) of the image-kinetic-molecular RA model was 0.781 [95% confidence interval (CI): 0.735, 0.828], which was higher than that of the image-kinetic RA model (0.629, 95% CI: 0.595, 0.663; P < 0.001) and comparable to that of the image-molecular RA model (0.755, 95% CI: 0.708, 0.802; P = 0.133). The AUROC of the image-kinetic-molecular DL model was 0.83 (95% CI: 0.816, 0.847), which was higher than that of the image-kinetic and image-molecular DL models (0.707, 95% CI: 0.654, 0.761; 0.79, 95% CI: 0.768, 0.812; P < 0.001) and higher than that of the image-kinetic-molecular RA model (0.778, 95% CI: 0.735, 0.828; P < 0.001).

CONCLUSIONS

The pretreatment DCE-MRI-based DL model is superior to the RA model in predicting pCR to NAC in breast cancer patients. The image-kinetic-molecular DL model has the best prediction performance.

Tomographic Ultrasound Imaging in the Diagnosis of Breast Tumors under the Guidance of Deep Learning Algorithms

This study was aimed to discuss the feasibility of distinguishing benign and malignant breast tumors under the tomographic ultrasound imaging (TUI) of deep learning algorithm. The deep learning algorithm was used to segment the images, and 120 patients with breast tumor were included in this study, all of whom underwent routine ultrasound examinations. Subsequently, TUI was used to assist in guiding the positioning, and the light scattering tomography system was used to further measure the lesions. A deep learning model was established to process the imaging results, and the pathological test results were undertaken as the gold standard for the efficiency of different imaging methods to diagnose the breast tumors. The results showed that, among 120 patients with breast tumor, 56 were benign lesions and 64 were malignant lesions. The average total amount of hemoglobin (HBT) of malignant lesions was significantly higher than that of benign lesions (P < 0.05). The sensitivity, specificity, accuracy, positive predictive value, and negative predictive value of TUI in the diagnosis of breast cancer were 90.4%, 75.6%, 81.4%, 84.7%, and 80.6%, respectively. The sensitivity, specificity, accuracy, positive predictive value, and negative predictive value of ultrasound in the diagnosis of breast cancer were 81.7%, 64.9%, 70.5%, 75.9%, and 80.6%, respectively. In addition, for suspected breast malignant lesions, the combined application of ultrasound and tomography can increase the diagnostic specificity to 82.1% and the accuracy to 83.8%. Based on the above results, it was concluded that TUI combined with ultrasound had a significant effect on benign and malignant diagnosis of breast cancer and can significantly improve the specificity and accuracy of diagnosis. It also reflected that deep learning technology had a good auxiliary role in the examination of diseases and was worth the promotion of clinical application.

Computer-aided classification of MRI for pathological complete response to neoadjuvant chemotherapy in breast cancer

Background: To determine suitable optimal classifiers and examine the general applicability of computer-aided classification to compare the differences between a computer-aided system and radiologists in predicting pathological complete response (pCR) from patients with breast cancer receiving neoadjuvant chemotherapy. Methods: We analyzed a total of 455 masses and used the U-Net network and ResNet to execute MRI segmentation and pCR classification. The diagnostic performance of radiologists, the computer-aided system and a combination of radiologists and computer-aided system were compared using receiver operating characteristic curve analysis. Results: The combination of radiologists and computer-aided system had the best performance for predicting pCR with an area under the curve (AUC) value of 0.899, significantly higher than that of radiologists alone (AUC: 0.700) and computer-aided system alone (AUC: 0.835). Conclusion: An automated classification system is feasible to predict the pCR to neoadjuvant chemotherapy in patients with breast cancer and can complement MRI.

Early prediction of neoadjuvant chemotherapy response by exploiting a transfer learning approach on breast DCE-MRIs

The dynamic contrast-enhanced MR imaging plays a crucial role in evaluating the effectiveness of neoadjuvant chemotherapy (NAC) even since its early stage through the prediction of the final pathological complete response (pCR). In this study, we proposed a transfer learning approach to predict if a patient achieved pCR (pCR) or did not (non-pCR) by exploiting, separately or in combination, pre-treatment and early-treatment exams from I-SPY1 TRIAL public database. First, low-level features, i.e., related to local structure of the image, were automatically extracted by a pre-trained convolutional neural network (CNN) overcoming manual feature extraction. Next, an optimal set of most stable features was detected and then used to design an SVM classifier. A first subset of patients, called fine-tuning dataset (30 pCR; 78 non-pCR), was used to perform the optimal choice of features. A second subset not involved in the feature selection process was employed as an independent test (7 pCR; 19 non-pCR) to validate the model. By combining the optimal features extracted from both pre-treatment and early-treatment exams with some clinical features, i.e., ER, PgR, HER2 and molecular subtype, an accuracy of 91.4% and 92.3%, and an AUC value of 0.93 and 0.90, were returned on the fine-tuning dataset and the independent test, respectively. Overall, the low-level CNN features have an important role in the early evaluation of the NAC efficacy by predicting pCR. The proposed model represents a first effort towards the development of a clinical support tool for an early prediction of pCR to NAC.

Predicting breast cancer response to neoadjuvant chemotherapy using ensemble deep transfer learning based on CT images

OBJECTIVE

To develop an ensemble a deep transfer learning model of CT images for predicting pathologic complete response (pCR) in breast cancer patients undergoing neoadjuvant chemotherapy (NAC).

METHODS

The data were obtained from the public dataset 'QIN-Breast' from The Cancer Imaging Archive (TCIA). CT images were gathered before and after the first cycle of NAC. CT images of 121 breast cancer patients were used to train and test the model. Among these patients, 58 achieved a pCR and 63 showed a non-pCR based pathology examination of surgical results after NAC. The dataset was split into training and testing subsets with a ratio of 7:3. In addition, the number of training samples in the dataset was increased from 656 to 1,968 by performing an image augmentation method. Two deep transfer learning models namely, DenseNet201 and ResNet152V2, and the ensemble model with a concatenation of two models, were trained and tested using CT images.

RESULTS

The ensemble model obtained the highest accuracy of 100% on the testing dataset. Furthermore, we received the best performance of 100% in recall, precision and f1-score value for the ensemble model. This supports the fact that the ensemble model results in better-generalized model and leads to efficient framework. Although a 0.004 and 0.003 difference were seen between the AUC of two base models (DenseNet201 and ResNet152V2) and the proposed ensemble, this increase in the model quality is critical in medical research. T-SNE revealed that in the proposed ensemble, no points were clustered into the wrong class. These results expose the strong performance of the proposed ensemble.

CONCLUSION

The study concluded that the ensemble model can increase the ability to predict breast cancer response to first-cycle NAC than two DenseNet201 and ResNet152V2 models.