Fusion deep learning approach combining diffuse optical tomography and ultrasound for improving breast cancer classification

Towards real-time diffuse optical tomography with a handheld scanning probe

Diffuse optical tomography (DOT) performed using deep-learning allows high-speed reconstruction of tissue optical properties and could thereby enable image-guided scanning, e.g., to enhance clinical breast imaging. Previously published models are geometry-specific and, therefore, require extensive data generation and training for each use case, restricting the scanning protocol at the point of use. A transformer-based architecture is proposed to overcome these obstacles that encode spatially unstructured DOT measurements, enabling a single trained model to handle arbitrary scanning pathways and measurement density. The model is demonstrated with breast tissue-emulating simulated and phantom data, yielding - for 24 mm-deep absorptions (μ a ) and reduced scattering (μ s ') images, respectively - average RMSEs of 0.0095±0.0023 cm-1 and 1.95±0.78 cm-1, Sørensen-Dice coefficients of 0.55±0.12 and 0.67±0.1, and anomaly contrast of 79±10% and 93.3±4.6% of the ground-truth contrast, with an effective imaging speed of 14 Hz. The average absolute μ a and μ s ' values of homogeneous simulated examples were within 10% of the true values.

Detection of breast cancer using machine learning on time-series diffuse optical transillumination data

Significance

Optical mammography as a promising tool for cancer diagnosis has largely fallen behind expectations. Modern machine learning (ML) methods offer ways to improve cancer detection in diffuse optical transmission data.

Aim

We aim to quantitatively evaluate the classification of cancer-positive versus cancer-negative patients using ML methods on raw transmission time series data from bilateral breast scans during subjects' rest.

Approach

We use a support vector machine (SVM) with hyperparameter optimization and cross-validation to systematically explore a range of data preprocessing and feature-generation strategies. We also apply an automated ML (AutoML) framework to validate our findings. We use receiver operating characteristics and the corresponding area under the curve (AUC) to quantify classification performance.

Results

For the sample group available ( N = 63 , 18 cancer patients), we demonstrate an AUC score of up to 93.3% for SVM classification and up to 95.0% for the AutoML classifier.

Conclusions

ML offers a viable strategy for clinically relevant breast cancer diagnosis using diffuse-optical transmission measurements. The diagnostic performance of ML on raw data can outperform traditional statistical biomarkers derived from reconstructed image time series. To achieve clinically relevant performance, our ML approach requires simultaneous bilateral scanning of the breasts with spatially dense channel coverage.

Improving diffuse optical tomography imaging quality using APU-Net: an attention-based physical U-Net model

Significance

Traditional diffuse optical tomography (DOT) reconstructions are hampered by image artifacts arising from factors such as DOT sources being closer to shallow lesions, poor optode-tissue coupling, tissue heterogeneity, and large high-contrast lesions lacking information in deeper regions (known as shadowing effect). Addressing these challenges is crucial for improving the quality of DOT images and obtaining robust lesion diagnosis.

Aim

We address the limitations of current DOT imaging reconstruction by introducing an attention-based U-Net (APU-Net) model to enhance the image quality of DOT reconstruction, ultimately improving lesion diagnostic accuracy.

Approach

We designed an APU-Net model incorporating a contextual transformer attention module to enhance DOT reconstruction. The model was trained on simulation and phantom data, focusing on challenges such as artifact-induced distortions and lesion-shadowing effects. The model was then evaluated by the clinical data.

Results

Transitioning from simulation and phantom data to clinical patients' data, our APU-Net model effectively reduced artifacts with an average artifact contrast decrease of 26.83% and improved image quality. In addition, statistical analyses revealed significant contrast improvements in depth profile with an average contrast increase of 20.28% and 45.31% for the second and third target layers, respectively. These results highlighted the efficacy of our approach in breast cancer diagnosis.

Conclusions

The APU-Net model improves the image quality of DOT reconstruction by reducing DOT image artifacts and improving the target depth profile.

CNN-based deep learning approach for classification of invasive ductal and metastasis types of breast carcinoma

OBJECTIVE

Breast cancer is one of the leading cancer causes among women worldwide. It can be classified as invasive ductal carcinoma (IDC) or metastatic cancer. Early detection of breast cancer is challenging due to the lack of early warning signs. Generally, a mammogram is recommended by specialists for screening. Existing approaches are not accurate enough for real-time diagnostic applications and thus require better and smarter cancer diagnostic approaches. This study aims to develop a customized machine-learning framework that will give more accurate predictions for IDC and metastasis cancer classification.

METHODS

This work proposes a convolutional neural network (CNN) model for classifying IDC and metastatic breast cancer. The study utilized a large-scale dataset of microscopic histopathological images to automatically perceive a hierarchical manner of learning and understanding.

RESULTS

It is evident that using machine learning techniques significantly (15%-25%) boost the effectiveness of determining cancer vulnerability, malignancy, and demise. The results demonstrate an excellent performance ensuring an average of 95% accuracy in classifying metastatic cells against benign ones and 89% accuracy was obtained in terms of detecting IDC.

CONCLUSIONS

The results suggest that the proposed model improves classification accuracy. Therefore, it could be applied effectively in classifying IDC and metastatic cancer in comparison to other state-of-the-art models.

Deep orthogonal multi-wavelength fusion for tomogram-free diagnosis in diffuse optical imaging

Novel portable diffuse optical tomography (DOT) devices for breast cancer lesions hold great promise for non-invasive, non-ionizing breast cancer screening. Critical to this capability is not just the identification of lesions but rather the complex problem of discriminating between malignant and benign lesions. To accurately reconstruct the highly heterogeneous tissue of a cancer lesion in healthy breast tissue using DOT, multiple wavelengths can be leveraged to maximize signal penetration while minimizing sensitivity to noise. However, these wavelength responses can overlap, capture common information, and correlate, potentially confounding reconstruction and downstream end tasks. We show that an orthogonal fusion loss regularizes multi-wavelength DOT leading to improved reconstruction and accuracy of end-to-end discrimination of malignant versus benign lesions. We further show that our raw-to-task model significantly reduces computational complexity without sacrificing accuracy, making it ideal for real-time throughput, desired in medical settings where handheld devices have severely restricted power budgets. Furthermore, our results indicate that image reconstruction is not necessary for unbiased classification of lesions with a balanced accuracy of 77% and 66% on the synthetic dataset and clinical dataset, respectively, using the raw-to-task model. Code is available at https://github.com/sfu-mial/FuseNet.

Real-time breast lesion classification combining diffuse optical tomography frequency domain data and BI-RADS assessment

Ultrasound (US)-guided diffuse optical tomography (DOT) has demonstrated potential for breast cancer diagnosis, in which real-time or near real-time diagnosis with high accuracy is desired. However, DOT's relatively slow data processing and image reconstruction speeds have hindered real-time diagnosis. Here, we propose a real-time classification scheme that combines US breast imaging reporting and data system (BI-RADS) readings and DOT frequency domain measurements. A convolutional neural network is trained to generate malignancy probability scores from DOT measurements. Subsequently, these scores are integrated with BI-RADS assessments using a support vector machine classifier, which then provides the final diagnostic output. An area under the receiver operating characteristic curve of 0.978 is achieved in distinguishing between benign and malignant breast lesions in patient data without image reconstruction.

Deep Learning in Breast Cancer Imaging: State of the Art and Recent Advancements in Early 2024

The rapid advancement of artificial intelligence (AI) has significantly impacted various aspects of healthcare, particularly in the medical imaging field. This review focuses on recent developments in the application of deep learning (DL) techniques to breast cancer imaging. DL models, a subset of AI algorithms inspired by human brain architecture, have demonstrated remarkable success in analyzing complex medical images, enhancing diagnostic precision, and streamlining workflows. DL models have been applied to breast cancer diagnosis via mammography, ultrasonography, and magnetic resonance imaging. Furthermore, DL-based radiomic approaches may play a role in breast cancer risk assessment, prognosis prediction, and therapeutic response monitoring. Nevertheless, several challenges have limited the widespread adoption of AI techniques in clinical practice, emphasizing the importance of rigorous validation, interpretability, and technical considerations when implementing DL solutions. By examining fundamental concepts in DL techniques applied to medical imaging and synthesizing the latest advancements and trends, this narrative review aims to provide valuable and up-to-date insights for radiologists seeking to harness the power of AI in breast cancer care.

Predicting rectal cancer tumor budding grading based on MRI and CT with multimodal deep transfer learning: A dual-center study

Objective

To investigate the effectiveness of a multimodal deep learning model in predicting tumor budding (TB) grading in rectal cancer (RC) patients.

Materials and methods

A retrospective analysis was conducted on 355 patients with rectal adenocarcinoma from two different hospitals. Among them, 289 patients from our institution were randomly divided into an internal training cohort (n = 202) and an internal validation cohort (n = 87) in a 7:3 ratio, while an additional 66 patients from another hospital constituted an external validation cohort. Various deep learning models were constructed and compared for their performance using T1CE and CT-enhanced images, and the optimal models were selected for the creation of a multimodal fusion model. Based on single and multiple factor logistic regression, clinical N staging and fecal occult blood were identified as independent risk factors and used to construct the clinical model. A decision-level fusion was employed to integrate these two models to create an ensemble model. The predictive performance of each model was evaluated using the area under the curve (AUC), DeLong's test, calibration curve, and decision curve analysis (DCA). Model visualization Gradient-weighted Class Activation Mapping (Grad-CAM) was performed for model interpretation.

Results

The multimodal fusion model demonstrated superior performance compared to single-modal models, with AUC values of 0.869 (95% CI: 0.761-0.976) for the internal validation cohort and 0.848 (95% CI: 0.721-0.975) for the external validation cohort. N-stage and fecal occult blood were identified as clinically independent risk factors through single and multivariable logistic regression analysis. The final ensemble model exhibited the best performance, with AUC values of 0.898 (95% CI: 0.820-0.975) for the internal validation cohort and 0.868 (95% CI: 0.768-0.968) for the external validation cohort.

Conclusion

Multimodal deep learning models can effectively and non-invasively provide individualized predictions for TB grading in RC patients, offering valuable guidance for treatment selection and prognosis assessment.

Enhanced model iteration algorithm with graph neural network for diffuse optical tomography

Diffuse optical tomography (DOT) employs near-infrared light to reveal the optical parameters of biological tissues. Due to the strong scattering of photons in tissues and the limited surface measurements, DOT reconstruction is severely ill-posed. The Levenberg-Marquardt (LM) is a popular iteration method for DOT, however, it is computationally expensive and its reconstruction accuracy needs improvement. In this study, we propose a neural model based iteration algorithm which combines the graph neural network with Levenberg-Marquardt (GNNLM), which utilizes a graph data structure to represent the finite element mesh. In order to verify the performance of the graph neural network, two GNN variants, namely graph convolutional neural network (GCN) and graph attention neural network (GAT) were employed in the experiments. The results showed that GCNLM performs best in the simulation experiments within the training data distribution. However, GATLM exhibits superior performance in the simulation experiments outside the training data distribution and real experiments with breast-like phantoms. It demonstrated that the GATLM trained with simulation data can generalize well to situations outside the training data distribution without transfer training. This offers the possibility to provide more accurate absorption coefficient distributions in clinical practice.

A twin convolutional neural network with hybrid binary optimizer for multimodal breast cancer digital image classification

There is a wide application of deep learning technique to unimodal medical image analysis with significant classification accuracy performance observed. However, real-world diagnosis of some chronic diseases such as breast cancer often require multimodal data streams with different modalities of visual and textual content. Mammography, magnetic resonance imaging (MRI) and image-guided breast biopsy represent a few of multimodal visual streams considered by physicians in isolating cases of breast cancer. Unfortunately, most studies applying deep learning techniques to solving classification problems in digital breast images have often narrowed their study to unimodal samples. This is understood considering the challenging nature of multimodal image abnormality classification where the fusion of high dimension heterogeneous features learned needs to be projected into a common representation space. This paper presents a novel deep learning approach combining a dual/twin convolutional neural network (TwinCNN) framework to address the challenge of breast cancer image classification from multi-modalities. First, modality-based feature learning was achieved by extracting both low and high levels features using the networks embedded with TwinCNN. Secondly, to address the notorious problem of high dimensionality associated with the extracted features, binary optimization method is adapted to effectively eliminate non-discriminant features in the search space. Furthermore, a novel method for feature fusion is applied to computationally leverage the ground-truth and predicted labels for each sample to enable multimodality classification. To evaluate the proposed method, digital mammography images and digital histopathology breast biopsy samples from benchmark datasets namely MIAS and BreakHis respectively. Experimental results obtained showed that the classification accuracy and area under the curve (AUC) for the single modalities yielded 0.755 and 0.861871 for histology, and 0.791 and 0.638 for mammography. Furthermore, the study investigated classification accuracy resulting from the fused feature method, and the result obtained showed that 0.977, 0.913, and 0.667 for histology, mammography, and multimodality respectively. The findings from the study confirmed that multimodal image classification based on combination of image features and predicted label improves performance. In addition, the contribution of the study shows that feature dimensionality reduction based on binary optimizer supports the elimination of non-discriminant features capable of bottle-necking the classifier.

Automated pipeline for breast cancer diagnosis using US assisted diffuse optical tomography

Ultrasound (US)-guided diffuse optical tomography (DOT) is a portable and non-invasive imaging modality for breast cancer diagnosis and treatment response monitoring. However, DOT data pre-processing and imaging reconstruction often require labor intensive manual processing which hampers real-time diagnosis. In this study, we aim at providing an automated US-assisted DOT pre-processing, imaging and diagnosis pipeline to achieve near real-time diagnosis. We have developed an automated DOT pre-processing method including motion detection, mismatch classification using deep-learning approach, and outlier removal. US-lesion information needed for DOT reconstruction was extracted by a semi-automated lesion segmentation approach combined with a US reading algorithm. A deep learning model was used to evaluate the quality of the reconstructed DOT images and a two-step deep-learning model developed earlier is implemented to provide final diagnosis based on US imaging features and DOT measurements and imaging results. The presented US-assisted DOT pipeline accurately processed the DOT measurements and reconstruction and reduced the procedure time to 2 to 3 minutes while maintained a comparable classification result with manually processed dataset.

Fast ICCD-based temperature modulated fluorescence tomography

Fluorescence tomography (FT) has become a powerful preclinical imaging modality with a great potential for several clinical applications. Although it has superior sensitivity and utilizes low-cost instrumentation, the highly scattering nature of bio-tissue makes FT in thick samples challenging, resulting in poor resolution and low quantitative accuracy. To overcome the limitations of FT, we previously introduced a novel method, termed temperature modulated fluorescence tomography (TMFT), which is based on two key elements: (1) temperature-sensitive fluorescent agents (ThermoDots) and (2) high-intensity focused ultrasound (HIFU). The fluorescence emission of ThermoDots increases up to hundredfold with only several degree temperature elevation. The exceptional and reversible response of these ThermoDots enables their modulation, which effectively allows their localization using the HIFU. Their localization is then used as functional a priori during the FT image reconstruction process to resolve their distribution with higher spatial resolution. The last version of the TMFT system was based on a cooled CCD camera utilizing a step-and-shoot mode, which necessitated long total imaging time only for a small selected region of interest (ROI). In this paper, we present the latest version of our TMFT technology, which uses a much faster continuous HIFU scanning mode based on an intensified CCD (ICCD) camera. This new, to the best of our knowledge, version can capture the whole field-of-view (FOV) of 50×30m m 2 at once and reduces the total imaging time down to 30 min, while preserving the same high resolution (∼1.3m m) and superior quantitative accuracy (<7% error) as the previous versions. Therefore, this new method is an important step toward utilization of TMFT for preclinical imaging.

Two-stage classification strategy for breast cancer diagnosis using ultrasound-guided diffuse optical tomography and deep learning

Significance

Ultrasound (US)-guided diffuse optical tomography (DOT) has demonstrated great potential for breast cancer diagnosis in which real-time or near real-time diagnosis with high accuracy is desired.

Aim

We aim to use US-guided DOT to achieve an automated, fast, and accurate classification of breast lesions.

Approach

We propose a two-stage classification strategy with deep learning. In the first stage, US images and histograms created from DOT perturbation measurements are combined to predict benign lesions. Then the non-benign suspicious lesions are passed through to the second stage, which combine US image features, DOT histogram features, and 3D DOT reconstructed images for final diagnosis.

Results

The first stage alone identified 73.0% of benign cases without image reconstruction. In distinguishing between benign and malignant breast lesions in patient data, the two-stage classification approach achieved an area under the receiver operating characteristic curve of 0.946, outperforming the diagnoses of all single-modality models and of a single-stage classification model that combines all US images, DOT histogram, and imaging features.

Conclusions

The proposed two-stage classification strategy achieves better classification accuracy than single-modality-only models and a single-stage classification model that combines all features. It can potentially distinguish breast cancers from benign lesions in near real-time.