Feature fusion of Raman chemical imaging and digital histopathology using machine learning for prostate cancer detection

High-Precision Intelligent Diagnosis of Pancreatic Cancer: Flowing Diffuseness from Single to Whole

Raman spectroscopy, as a label-free optical technique, provides a unique solution for tissue diagnosis. However, due to the limitation of point-by-point acquisition mode and multivariate statistical analysis methods, conventional methods pose a major bottleneck toward achieving highly time-efficient, accurate, and holistic diagnosis. Here, the authors establish line-scan Raman spectrochemical holistic analysis (LRSHA), an intelligent diagnostic method for rapid Raman data collection and holistic tissue diagnosis. The line-scan technique is first used for rapid spectral acquisition (∼15 s) in a tissue block area (0.75 × 0.5 mm2), which is 2 orders of magnitude faster than conventional methods. Then, the one-dimensional (1D) Raman spectra are converted into two-dimensional (2D) Raman encoding figures by the spectral recurrence plot transformation. The 2D deep learning models achieve 96.0% accuracy, 7% higher than that of 1D deep learning models. Moreover, the neighborhood enhancement method is applied to correct the initial deep learning results, like ripple flowing diffuseness from single to whole, which ultimately greatly improves the diagnostic accuracy to 99.7%. We also demonstrated that our method can identify tissue neoplasia at resection margins that appear nearly normal to the naked eye. Together, the LRSHA method shows valuable potential for rapid, efficient, and accurate tissue diagnosis.

Determination of Trace Organic Contaminant Concentration via Machine Classification of Surface-Enhanced Raman Spectra

Surface-enhanced Raman spectroscopy (SERS) has been well explored as a highly effective characterization technique that is capable of chemical pollutant detection and identification at very low concentrations. Machine learning has been previously used to identify compounds based on SERS spectral data. However, utilization of SERS to quantify concentrations, with or without machine learning, has been difficult due to the spectral intensity being sensitive to confounding factors such as the substrate parameters, orientation of the analyte, and sample preparation technique. Here, we demonstrate an approach for predicting the concentration of sample pollutants from SERS spectra using machine learning. Frequency domain transform methods, including the Fourier and Walsh-Hadamard transforms, are applied to spectral data sets of three analytes (rhodamine 6G, chlorpyrifos, and triclosan), which are then used to train machine learning algorithms. Using standard machine learning models, the concentration of the sample pollutants is predicted with >80% cross-validation accuracy from raw SERS data. A cross-validation accuracy of 85% was achieved using deep learning for a moderately sized data set (∼100 spectra), and 70-80% was achieved for small data sets (∼50 spectra). Performance can be maintained within this range even when combining various sample preparation techniques and environmental media interference. Additionally, as a spectral pretreatment, the Fourier and Hadamard transforms are shown to consistently improve prediction accuracy across multiple data sets. Finally, standard models were shown to accurately identify characteristic peaks of compounds via analysis of their importance scores, further verifying their predictive value.

Acceleration of high-quality Raman imaging via a locality enhanced transformer network

Raman imaging (RI) is an outstanding technique that enables molecular-level medical diagnostics and therapy assessment by providing characteristic fingerprint and morphological information about molecules. However, obtaining high-quality Raman images generally requires a long acquisition time, up to hours, which is prohibitive for RI applications of timely cytopathology and histopathology analyses. To address this issue, image super-resolution (SR) based on deep learning, including convolutional neural networks and transformers, has been widely recognized as an effective solution to reduce the time required for achieving high-quality RI. In this study, a locality enhanced transformer network (LETNet) is proposed to perform Raman image SR. Specifically, the general architecture of the transformer is adopted with the replacement of self-attention by convolution to generate high-fidelity and detailed SR images. Additionally, the convolution in the LETNet is further optimized by utilizing depth-wise convolution to improve the computational efficiency of the model. Experiments on hyperspectral Raman images of breast cancer cells and Raman images of a few channels of brain tumor tissues demonstrate that the LETNet achieves superior 2×, 4×, and 8× SR with fewer parameters compared with other SR methods. Consequently, high-quality Raman images can be obtained with a significant reduction in time, ranging from 4 to 64 times. Overall, the proposed method provides a novel, efficient, and reliable solution to expedite high-quality RI and promote its application in real-time diagnosis and therapy.

Multimodal image and spectral feature learning for efficient analysis of water-suspended particles

We have developed a method to combine morphological and chemical information for the accurate identification of different particle types using optical measurement techniques that require no sample preparation. A combined holographic imaging and Raman spectroscopy setup is used to gather data from six different types of marine particles suspended in a large volume of seawater. Unsupervised feature learning is performed on the images and the spectral data using convolutional and single-layer autoencoders. The learned features are combined, where we demonstrate that non-linear dimensional reduction of the combined multimodal features can achieve a high clustering macro F1 score of 0.88, compared to a maximum of 0.61 when only image or spectral features are used. The method can be applied to long-term monitoring of particles in the ocean without the need for sample collection. In addition, it can be applied to data from different types of sensor measurements without significant modifications.

Fusing hand-crafted and deep-learning features in a convolutional neural network model to identify prostate cancer in pathology images

Prostate cancer can be diagnosed by prostate biopsy using transectal ultrasound guidance. The high number of pathology images from biopsy tissues is a burden on pathologists, and analysis is subjective and susceptible to inter-rater variability. The use of machine learning techniques could make prostate histopathology diagnostics more precise, consistent, and efficient overall. This paper presents a new classification fusion network model that was created by fusing eight advanced image features: seven hand-crafted features and one deep-learning feature. These features are the scale-invariant feature transform (SIFT), speeded up robust feature (SURF), oriented features from accelerated segment test (FAST) and rotated binary robust independent elementary features (BRIEF) (ORB) of local features, shape and texture features of the cell nuclei, the histogram of oriented gradients (HOG) feature of the cavities, a color feature, and a convolution deep-learning feature. Matching, integrated, and fusion networks are the three essential components of the proposed deep-learning network. The integrated network consists of both a backbone and an additional network. When classifying 1100 prostate pathology images using this fusion network with different backbones (ResNet-18/50, VGG-11/16, and DenseNet-121/201), we discovered that the proposed model with the ResNet-18 backbone achieved the best performance in terms of the accuracy (95.54%), specificity (93.64%), and sensitivity (97.27%) as well as the area under the receiver operating characteristic curve (98.34%). However, each of the assessment criteria for these separate features had a value lower than 90%, which demonstrates that the suggested model combines differently derived characteristics in an effective manner. Moreover, a Grad-CAM++ heatmap was used to observe the differences between the proposed model and ResNet-18 in terms of the regions of interest. This map showed that the proposed model was better at focusing on cancerous cells than ResNet-18. Hence, the proposed classification fusion network, which combines hand-crafted features and a deep-learning feature, is useful for computer-aided diagnoses based on pathology images of prostate cancer. Because of the similarities in the feature engineering and deep learning for different types of pathology images, the proposed method could be used for other pathology images, such as those of breast, thyroid cancer.

High-Resolution Histopathological Image Classification Model Based on Fused Heterogeneous Networks with Self-Supervised Feature Representation

Applying machine learning technology to automatic image analysis and auxiliary diagnosis of whole slide image (WSI) may help to improve the efficiency, objectivity, and consistency of pathological diagnosis. Due to its extremely high resolution, it is still a great challenge to directly process WSI through deep neural networks. In this paper, we propose a novel model for the task of classification of WSIs. The model is composed of two parts. The first part is a self-supervised encoding network with a UNet-like architecture. Each patch from a WSI is encoded as a compressed latent representation. These features are placed according to their corresponding patch’s original location in WSI, forming a feature cube. The second part is a classification network fused by 4 famous network blocks with heterogeneous architectures, with feature cube as input. Our model effectively expresses the feature and preserves location information of each patch. The fused network integrates heterogeneous features generated by different networks which yields robust classification results. The model is evaluated on two public datasets with comparison to baseline models. The evaluation results show the effectiveness of the proposed model.

Label-Free Differentiation of Cancer and Non-Cancer Cells Based on Machine-Learning-Algorithm-Assisted Fast Raman Imaging

This paper proposes a rapid, label-free, and non-invasive approach for identifying murine cancer cells (B16F10 melanoma cancer cells) from non-cancer cells (C2C12 muscle cells) using machine-learning-assisted Raman spectroscopic imaging. Through quick Raman spectroscopic imaging, a hyperspectral data processing approach based on machine learning methods proved capable of presenting the cell structure and distinguishing cancer cells from non-cancer muscle cells without compromising full-spectrum information. This study discovered that biomolecular information-nucleic acids, proteins, and lipids-from cells could be retrieved efficiently from low-quality hyperspectral Raman datasets and then employed for cell line differentiation.

Raman Imaging and Fluorescence Lifetime Imaging Microscopy for Diagnosis of Cancer State and Metabolic Monitoring

Hurdles for effective tumor therapy are delayed detection and limited effectiveness of systemic drug therapies by patient-specific multidrug resistance. Non-invasive bioimaging tools such as fluorescence lifetime imaging microscopy (FLIM) and Raman-microspectroscopy have evolved over the last decade, providing the potential to be translated into clinics for early-stage disease detection, in vitro drug screening, and drug efficacy studies in personalized medicine. Accessing tissue- and cell-specific spectral signatures, Raman microspectroscopy has emerged as a diagnostic tool to identify precancerous lesions, cancer stages, or cell malignancy. In vivo Raman measurements have been enabled by recent technological advances in Raman endoscopy and signal-enhancing setups such as coherent anti-stokes Raman spectroscopy or surface-enhanced Raman spectroscopy. FLIM enables in situ investigations of metabolic processes such as glycolysis, oxidative stress, or mitochondrial activity by using the autofluorescence of co-enzymes NADH and FAD, which are associated with intrinsic proteins as a direct measure of tumor metabolism, cell death stages and drug efficacy. The combination of non-invasive and molecular-sensitive in situ techniques and advanced 3D tumor models such as patient-derived organoids or microtumors allows the recapitulation of tumor physiology and metabolism in vitro and facilitates the screening for patient-individualized drug treatment options.