Discriminating healthy from tumor tissue in breast lumpectomy specimens using deep learning-based hyperspectral imaging

Deformable multi-modal image registration for the correlation between optical measurements and histology images

Significance

The accurate correlation between optical measurements and pathology relies on precise image registration, often hindered by deformations in histology images. We investigate an automated multi-modal image registration method using deep learning to align breast specimen images with corresponding histology images.

Aim

We aim to explore the effectiveness of an automated image registration technique based on deep learning principles for aligning breast specimen images with histology images acquired through different modalities, addressing challenges posed by intensity variations and structural differences.

Approach

Unsupervised and supervised learning approaches, employing the VoxelMorph model, were examined using a dataset featuring manually registered images as ground truth.

Results

Evaluation metrics, including Dice scores and mutual information, demonstrate that the unsupervised model exceeds the supervised (and manual) approaches significantly, achieving superior image alignment. The findings highlight the efficacy of automated registration in enhancing the validation of optical technologies by reducing human errors associated with manual registration processes.

Conclusions

This automated registration technique offers promising potential to enhance the validation of optical technologies by minimizing human-induced errors and inconsistencies associated with manual image registration processes, thereby improving the accuracy of correlating optical measurements with pathology labels.

Spatial and Spectral Reconstruction of Breast Lumpectomy Hyperspectral Images

(1) Background: Hyperspectral imaging has emerged as a promising margin assessment technique for breast-conserving surgery. However, to be implicated intraoperatively, it should be both fast and capable of yielding high-quality images to provide accurate guidance and decision-making throughout the surgery. As there exists a trade-off between image quality and data acquisition time, higher resolution images come at the cost of longer acquisition times and vice versa. (2) Methods: Therefore, in this study, we introduce a deep learning spatial-spectral reconstruction framework to obtain a high-resolution hyperspectral image from a low-resolution hyperspectral image combined with a high-resolution RGB image as input. (3) Results: Using the framework, we demonstrate the ability to perform a fast data acquisition during surgery while maintaining a high image quality, even in complex scenarios where challenges arise, such as blur due to motion artifacts, dead pixels on the camera sensor, noise from the sensor's reduced sensitivity at spectral extremities, and specular reflections caused by smooth surface areas of the tissue. (4) Conclusion: This gives the opportunity to facilitate an accurate margin assessment through intraoperative hyperspectral imaging.

Point Projection Mapping System for Tracking, Registering, Labeling, and Validating Optical Tissue Measurements

The validation of newly developed optical tissue-sensing techniques for tumor detection during cancer surgery requires an accurate correlation with the histological results. Additionally, such an accurate correlation facilitates precise data labeling for developing high-performance machine learning tissue-classification models. In this paper, a newly developed Point Projection Mapping system will be introduced, which allows non-destructive tracking of the measurement locations on tissue specimens. Additionally, a framework for accurate registration, validation, and labeling with the histopathology results is proposed and validated on a case study. The proposed framework provides a more-robust and accurate method for the tracking and validation of optical tissue-sensing techniques, which saves time and resources compared to the available conventional techniques.

SpeCamX: mobile app that turns unmodified smartphones into multispectral imagers

We present the development of SpeCamX, a mobile application that enables an unmodified smartphone into a multispectral imager. Multispectral imaging provides detailed spectral information about objects or scenes, but its accessibility has been limited due to its specialized requirements for the device. SpeCamX overcomes this limitation by utilizing the RGB photographs captured by smartphones and converting them into multispectral images spanning a range of 420 to 680 nm without a need for internal modifications or external attachments. The app also includes plugin functions for extracting medical information from the resulting multispectral data cube. In a clinical study, SpeCamX was used to implement an augmented smartphone bilirubinometer, predicting blood bilirubin levels (BBL) with superior performance in accuracy, efficiency and stability compared to default smartphone cameras. This innovative technology democratizes multispectral imaging, making it accessible to a wider audience and opening new possibilities for both medical and non-medical applications.

Joint Diagnostic Method of Tumor Tissue Based on Hyperspectral Spectral-Spatial Transfer Features

In order to improve the clinical application of hyperspectral technology in the pathological diagnosis of tumor tissue, a joint diagnostic method based on spectral-spatial transfer features was established by simulating the actual clinical diagnosis process and combining micro-hyperspectral imaging with large-scale pathological data. In view of the limited sample volume of medical hyperspectral data, a multi-data transfer model pre-trained on conventional pathology datasets was applied to the classification task of micro-hyperspectral images, to explore the differences in spectral-spatial transfer features in the wavelength of 410-900 nm between tumor tissues and normal tissues. The experimental results show that the spectral-spatial transfer convolutional neural network (SST-CNN) achieved a classification accuracy of 95.46% for the gastric cancer dataset and 95.89% for the thyroid cancer dataset, thus outperforming models trained on single conventional digital pathology and single hyperspectral data. The joint diagnostic method established based on SST-CNN can complete the interpretation of a section of data in 3 min, thus providing a new technical solution for the rapid diagnosis of pathology. This study also explored problems involving the correlation between tumor tissues and typical spectral-spatial features, as well as the efficient transformation of conventional pathological and transfer spectral-spatial features, which solidified the theoretical research on hyperspectral pathological diagnosis.

Tissue Classification of Breast Cancer by Hyperspectral Unmixing

(1) Background: Assessing the resection margins during breast-conserving surgery is an important clinical need to minimize the risk of recurrent breast cancer. However, currently there is no technique that can provide real-time feedback to aid surgeons in the margin assessment. Hyperspectral imaging has the potential to overcome this problem. To classify resection margins with this technique, a tissue discrimination model should be developed, which requires a dataset with accurate ground-truth labels. However, establishing such a dataset for resection specimens is difficult. (2) Methods: In this study, we therefore propose a novel approach based on hyperspectral unmixing to determine which pixels within hyperspectral images should be assigned to the ground-truth labels from histopathology. Subsequently, we use this hyperspectral-unmixing-based approach to develop a tissue discrimination model on the presence of tumor tissue within the resection margins of ex vivo breast lumpectomy specimens. (3) Results: In total, 372 measured locations were included on the lumpectomy resection surface of 189 patients. We achieved a sensitivity of 0.94, specificity of 0.85, accuracy of 0.87, Matthew's correlation coefficient of 0.71, and area under the curve of 0.92. (4) Conclusion: Using this hyperspectral-unmixing-based approach, we demonstrated that the measured locations with hyperspectral imaging on the resection surface of lumpectomy specimens could be classified with excellent performance.

Discrimination between normal and necrotic small intestinal tissue using hyperspectral imaging and unsupervised classification

Objective and automatic clinical discrimination of normal and necrotic sites of small intestinal tissue remains challenging. In this study, hyperspectral imaging (HSI) and unsupervised classification techniques were used to distinguish normal and necrotic sites of small intestinal tissues. Small intestinal tissue hyperspectral images of eight Japanese large-eared white rabbits were acquired using a visible near-infrared hyperspectral camera, and K-means and density peaks (DP) clustering algorithms were used to differentiate between normal and necrotic tissue. The three cases in this study showed that the average clustering purity of the DP clustering algorithm reached 92.07% when the two band combinations of 500-622 and 700-858 nm were selected. The results of this study suggest that HSI and DP clustering can assist physicians in distinguishing between normal and necrotic sites in the small intestine in vivo.

Emerging Technology for Intraoperative Margin and Assisting in Post-Surgery tissue diagnostic for Future Breast-Conserving

INTRODUCTION

Tissue-preserving surgery is utilized progressively in cancer therapy, where a clear surgical margin is critical to avoid cancer recurrence, specifically in breast cancer (BC) surgery. The Intraoperative pathologic approaches that rely on tissue segmenting and staining have been recognized as the ground truth for BC diagnosis. Nevertheless, these methods are constrained by its complication and timewasting for tissue preparation.

OBJECTIVE

We present a non-invasive optical imaging system incorporating a hyperspectral (HS) camera to discriminate between cancerous and non-cancerous tissues in ex-vivo breast specimens, which could be an intraoperative diagnostic technique to aid surgeons during surgery and later a valuable tool to assist pathologists.

METHODS

We have established a hyperspectral Imaging (HSI) system comprising a push-broom HS camera at wavelength 380∼1050 nm with source light 390∼980 nm. We have measured the investigated samples' diffuse reflectance (R_d), fixed on slides from 30 distinct patients incorporating mutually normal and ductal carcinoma tissue. The samples were divided into two groups, stained tissues during the surgery (control group) and unstained samples (test group), both captured with the HSI system in the visible and near-infrared (VIS-NIR) range. Then, to address the problem of the spectral nonuniformity of the illumination device and the influence of the dark current, the radiance data were normalized to yield the radiance of the specimen and neutralize the intensity effect to focus on the spectral reflectance shift for each tissue. The selection of the threshold window from the measured R_d is carried out by exploiting the statistical analysis by calculating each region's mean and standard deviation. Afterward, we selected the optimum spectral images from the HS data cube to apply a custom K-means algorithm and contour delineation to identify the regular districts from the BC regions.

RESULTS

We noticed that the measured spectral R_d for the malignant tissues of the investigated case studies versus the reference source light varies regarding the cancer stage, as sometimes the R_d is higher for the tumor or vice versa for the normal tissue. Later, from the analysis of the whole samples, we found that the most appropriate wavelength for the BC tissues was 447 nm, which was highly reflected versus the normal tissue. However, the most convenient one for the normal tissue was at 545 nm with high reflection versus the BC tissue. Finally, we implement a moving average filter for noise reduction and a custom K-means clustering algorithm on the selected two spectral images (447, 551 nm) to identify the various regions and effectively-identified spectral tissue variations with a sensitivity of 98.95%, and specificity of 98.44%. A pathologist later confirmed these outcomes as the ground truth for the tissue sample investigations.

CONCLUSIONS

The proposed system could help the surgeon and the pathologist identify the cancerous tissue margins from the non-cancerous tissue with a non-invasive, rapid, and minimum time method achieving high sensitivity up to 98.95%.

Estimating quantitative physiological and morphological tissue parameters of murine tumor models using hyperspectral imaging and optical profilometry

Understanding tumors and their microenvironment are essential for successful and accurate disease diagnosis. Tissue physiology and morphology are altered in tumors compared to healthy tissues, and there is a need to monitor tumors and their surrounding tissues, including blood vessels, non-invasively. This preliminary study utilizes a multimodal optical imaging system combining hyperspectral imaging (HSI) and three-dimensional (3D) optical profilometry (OP) to capture hyperspectral images and surface shapes of subcutaneously grown murine tumor models. Hyperspectral images are corrected with 3D OP data and analyzed using the inverse-adding doubling (IAD) method to extract tissue properties such as melanin volume fraction and oxygenation. Blood vessels are segmented using the B-COSFIRE algorithm from oxygenation maps. From 3D OP data, tumor volumes are calculated and compared to manual measurements using a vernier caliper. Results show that tumors can be distinguished from healthy tissue based on most extracted tissue parameters ( p < 0.05 ). Furthermore, blood oxygenation is 50% higher within the blood vessels than in the surrounding tissue, and tumor volumes calculated using 3D OP agree within 26% with manual measurements using a vernier caliper. Results suggest that combining HSI and OP could provide relevant quantitative information about tumors and improve the disease diagnosis.