Manifold Embedding and Semantic Segmentation for Intraoperative Guidance With Hyperspectral Brain Imaging

Toward real-time margin assessment in breast-conserving surgery with hyperspectral imaging

Margin assessment in breast-conserving surgery (BSC) remains a critical challenge, with 20-25% of cases resulting in inadequate tumor resection, increasing the risk of local recurrence and the need for additional treatment. In this study, we evaluate the diagnostic performance of hyperspectral imaging (HSI) as a non-invasive technique for assessing resection margins in ex vivo lumpectomy specimens. A dataset of over 200 lumpectomy specimens was collected using two hyperspectral cameras, and a classification algorithm was developed to distinguish between healthy and tumor tissue within margins of 0 and 2 mm. The proposed approach achieved its highest diagnostic performance at a 0 mm margin, with a sensitivity of 92%, specificity of 78%, accuracy of 83%, Matthews correlation coefficient of 68%, and an area under the curve of 89%. The entire resection surface could be imaged and evaluated within 10 minutes, providing a rapid and non-invasive alternative to conventional margin assessment techniques. These findings represent a significant advancement toward real-time intraoperative margin assessment, highlighting the potential of HSI to enhance surgical precision and reduce re-excision rates in BCS.

R2D2-GAN: Robust Dual Discriminator Generative Adversarial Network for Microscopy Hyperspectral Image Super-Resolution

High-resolution microscopy hyperspectral (HS) images can provide highly detailed spatial and spectral information, enabling the identification and analysis of biological tissues at a microscale level. Recently, significant efforts have been devoted to enhancing the resolution of HS images by leveraging high spatial resolution multispectral (MS) images. However, the inherent hardware constraints lead to a significant distribution gap between HS and MS images, posing challenges for image super-resolution within biomedical domains. This discrepancy may arise from various factors, including variations in camera imaging principles (e.g., snapshot and push-broom imaging), shooting positions, and the presence of noise interference. To address these challenges, we introduced a unique unsupervised super-resolution framework named R2D2-GAN. This framework utilizes a generative adversarial network (GAN) to efficiently merge the two data modalities and improve the resolution of microscopy HS images. Traditionally, supervised approaches have relied on intuitive and sensitive loss functions, such as mean squared error (MSE). Our method, trained in a real-world unsupervised setting, benefits from exploiting consistent information across the two modalities. It employs a game-theoretic strategy and dynamic adversarial loss, rather than relying solely on fixed training strategies for reconstruction loss. Furthermore, we have augmented our proposed model with a central consistency regularization (CCR) module, aiming to further enhance the robustness of the R2D2-GAN. Our experimental results show that the proposed method is accurate and robust for super-resolution images. We specifically tested our proposed method on both a real and a synthetic dataset, obtaining promising results in comparison to other state-of-the-art methods.

Technological Frontiers in Brain Cancer: A Systematic Review and Meta-Analysis of Hyperspectral Imaging in Computer-Aided Diagnosis Systems

Brain cancer is a substantial factor in the mortality associated with cancer, presenting difficulties in the timely identification of the disease. The precision of diagnoses is significantly dependent on the proficiency of radiologists and neurologists. Although there is potential for early detection with computer-aided diagnosis (CAD) algorithms, the majority of current research is hindered by its modest sample sizes. This meta-analysis aims to comprehensively assess the diagnostic test accuracy (DTA) of computer-aided design (CAD) models specifically designed for the detection of brain cancer utilizing hyperspectral (HSI) technology. We employ Quadas-2 criteria to choose seven papers and classify the proposed methodologies according to the artificial intelligence method, cancer type, and publication year. In order to evaluate heterogeneity and diagnostic performance, we utilize Deeks' funnel plot, the forest plot, and accuracy charts. The results of our research suggest that there is no notable variation among the investigations. The CAD techniques that have been examined exhibit a notable level of precision in the automated detection of brain cancer. However, the absence of external validation hinders their potential implementation in real-time clinical settings. This highlights the necessity for additional studies in order to authenticate the CAD models for wider clinical applicability.

Handheld hyperspectral imaging as a tool for the post-mortem interval estimation of human skeletal remains

In forensic medicine, estimating human skeletal remains' post-mortem interval (PMI) can be challenging. Following death, bones undergo a series of chemical and physical transformations due to their interactions with the surrounding environment. Post-mortem changes have been assessed using various methods, but estimating the PMI of skeletal remains could still be improved. We propose a new methodology with handheld hyperspectral imaging (HSI) system based on the first results from 104 human skeletal remains with PMIs ranging between 1 day and 2000 years. To differentiate between forensic and archaeological bone material, the Convolutional Neural Network analyzed 65.000 distinct diagnostic spectra: the classification accuracy was 0.58, 0.62, 0.73, 0.81, and 0.98 for PMIs of 0 week-2 weeks, 2 weeks-6 months, 6 months-1 year, 1 year-10 years, and >100 years, respectively. In conclusion, HSI can be used in forensic medicine to distinguish bone materials >100 years old from those <10 years old with an accuracy of 98%. The model has adequate predictive performance, and handheld HSI could serve as a novel approach to objectively and accurately determine the PMI of human skeletal remains.

Density clustering-based automatic anatomical section recognition in colonoscopy video using deep learning

Recognizing anatomical sections during colonoscopy is crucial for diagnosing colonic diseases and generating accurate reports. While recent studies have endeavored to identify anatomical regions of the colon using deep learning, the deformable anatomical characteristics of the colon pose challenges for establishing a reliable localization system. This study presents a system utilizing 100 colonoscopy videos, combining density clustering and deep learning. Cascaded CNN models are employed to estimate the appendix orifice (AO), flexures, and "outside of the body," sequentially. Subsequently, DBSCAN algorithm is applied to identify anatomical sections. Clustering-based analysis integrates clinical knowledge and context based on the anatomical section within the model. We address challenges posed by colonoscopy images through non-informative removal preprocessing. The image data is labeled by clinicians, and the system deduces section correspondence stochastically. The model categorizes the colon into three sections: right (cecum and ascending colon), middle (transverse colon), and left (descending colon, sigmoid colon, rectum). We estimated the appearance time of anatomical boundaries with an average error of 6.31 s for AO, 9.79 s for HF, 27.69 s for SF, and 3.26 s for outside of the body. The proposed method can facilitate future advancements towards AI-based automatic reporting, offering time-saving efficacy and standardization.

Unsupervised band selection of medical hyperspectral images guided by data gravitation and weak correlation

BACKGROUND AND OBJECTIVE

Medical hyperspectral images (MHSIs) are used for a contact-free examination of patients without harmful radiation. However, high-dimensionality images contain large amounts of data that are sparsely distributed in a high-dimensional space, which leads to the "curse of dimensionality" (called Hughes' phenomenon) and increases the complexity and cost of data processing and storage. Hence, there is a need for spectral dimensionality reduction before the clinical application of MHSIs. Some dimensionality-reducing strategies have been proposed; however, they distort the data within MHSIs.

METHODS

To compress dimensionality without destroying the original data structure, we propose a method that involves data gravitation and weak correlation-based ranking (DGWCR) for removing bands of noise from MHSIs while clustering signal-containing bands. Band clustering is done by using the connection centre evolution (CCE) algorithm and selecting the most representative bands in each cluster based on the composite force. The bands within the clusters are ranked using the new entropy-containing matrix, and a global ranking of bands is obtained by applying an S-shaped strategy. The source code is available at https://www.github.com/zhangchenglong1116/DGWCR.

RESULTS

Upon feeding the reduced-dimensional images into various classifiers, the experimental results demonstrated that the small number of bands selected by the proposed DGWCR consistently achieved higher classification accuracy than the original data. Unlike other reference methods (e.g. the latest deep-learning-based strategies), DGWCR chooses the spectral bands with the least redundancy and greatest discrimination.

CONCLUSION

In this study, we present a method for efficient band selection for MHSIs that alleviates the "curse of dimensionality". Experiments were validated with three MHSIs in the human brain, and they outperformed several other band selection methods, demonstrating the clinical potential of DGWCR.

Unsupervised dimensionality reduction of medical hyperspectral imagery in tensor space

BACKGROUND AND OBJECTIVES

Compared with traditional RGB images, medical hyperspectral imagery (HSI) has numerous continuous narrow spectral bands, which can provide rich information for cancer diagnosis. However, the abundant spectral bands also contain a large amount of redundancy information and increase computational complexity. Thus, dimensionality reduction (DR) is essential in HSI analysis. All vector-based DR methods ignore the cubic nature of HSI resulting from vectorization. To overcome the disadvantage of vector-based DR methods, tensor-based techniques have been developed by employing multi-linear algebra.

METHODS

To fully exploit the structure features of medical HSI and enhance computational efficiency, a novel method called unsupervised dimensionality reduction via tensor-based low-rank collaborative graph embedding (TLCGE) is proposed. TLCGE introduces entropy rate superpixel (ERS) segmentation algorithm to generate superpixels. Then, a low-rank collaborative graph weight matrix is constructed on each superpixel, greatly improving the efficiency and robustness of the proposed method. After that, TLCGE reduces dimensions in tensor space to well preserve intrinsic structure of HSI.

RESULTS

The proposed TLCGE is tested on cholangiocarcinoma microscopic hyperspectral data sets. To further demonstrate the effectiveness of the proposed algorithm, other machine learning DR methods are used for comparison. Experimental results on cholangiocarcinoma microscopic hyperspectral data sets validate the effectiveness of the proposed TLCGE.

CONCLUSIONS

The proposed TLCGE is a tensor-based DR method, which can maintain the intrinsic 3-D data structure of medical HSI. By imposing the low-rank and sparse constraints on the objective function, the proposed TLCGE can fully explore the local and global structures within each superpixel. The computational efficiency of the proposed TLCGE is better than other tensor-based DR methods, which can be used as a preprocessing step in real medical HSI classification or segmentation.

Hyperspectral Imaging in Brain Tumor Surgery-Evidence of Machine Learning-Based Performance

BACKGROUND

Hyperspectral imaging (HSI) has the potential to enhance surgical tissue detection and diagnostics. Definite utilization of intraoperative HSI guidance demands validated machine learning and public datasets that currently do not exist. Moreover, current imaging conventions are dispersed, and evidence-based paradigms for neurosurgical HSI have not been declared.

METHODS

We presented the rationale and a detailed clinical paradigm for establishing microneurosurgical HSI guidance. In addition, a systematic literature review was conducted to summarize the current indications and performance of neurosurgical HSI systems, with an emphasis on machine learning-based methods.

RESULTS

The published data comprised a few case series or case reports aiming to classify tissues during glioma operations. For a multitissue classification problem, the highest overall accuracy of 80% was obtained using deep learning. Our HSI system was capable of intraoperative data acquisition and visualization with minimal disturbance to glioma surgery.

CONCLUSIONS

In a limited number of publications, neurosurgical HSI has demonstrated unique capabilities in contrast to the established imaging techniques. Multidisciplinary work is required to establish communicable HSI standards and clinical impact. Our HSI paradigm endorses systematic intraoperative HSI data collection, which aims to facilitate the related standards, medical device regulations, and value-based medical imaging systems.

Review on the Application of Hyperspectral Imaging Technology of the Exposed Cortex in Cerebral Surgery

The study of brain science is vital to human health. The application of hyperspectral imaging in biomedical fields has grown dramatically in recent years due to their unique optical imaging method and multidimensional information acquisition. Hyperspectral imaging technology can acquire two-dimensional spatial information and one-dimensional spectral information of biological samples simultaneously, covering the ultraviolet, visible and infrared spectral ranges with high spectral resolution, which can provide diagnostic information about the physiological, morphological and biochemical components of tissues and organs. This technology also presents finer spectral features for brain imaging studies, and further provides more auxiliary information for cerebral disease research. This paper reviews the recent advance of hyperspectral imaging in cerebral diagnosis. Firstly, the experimental setup, image acquisition and pre-processing, and analysis methods of hyperspectral technology were introduced. Secondly, the latest research progress and applications of hyperspectral imaging in brain tissue metabolism, hemodynamics, and brain cancer diagnosis in recent years were summarized briefly. Finally, the limitations of the application of hyperspectral imaging in cerebral disease diagnosis field were analyzed, and the future development direction was proposed.

Unsupervised Foggy Scene Understanding via Self Spatial-Temporal Label Diffusion

Understanding foggy image sequence in driving scene is critical for autonomous driving, but it remains a challenging task due to the difficulty in collecting and annotating real-world images of adverse weather. Recently, self-training strategy has been considered as a powerful solution for unsupervised domain adaptation, which iteratively adapts the model from the source domain to the target domain by generating target pseudo labels and re-training the model. However, the selection of confident pseudo labels inevitably suffers from the conflict between sparsity and accuracy, both of which will lead to suboptimal models. To tackle this problem, we exploit the characteristics of the foggy image sequence of driving scenes to densify the confident pseudo labels. Specifically, based on the two discoveries of local spatial similarity and adjacent temporal correspondence of the sequential image data, we propose a novel Target-Domain driven pseudo label Diffusion (TDo-Dif) scheme. It employs superpixels and optical flows to identify the spatial similarity and temporal correspondence, respectively, and then diffuses the confident but sparse pseudo labels within a superpixel or a temporal corresponding pair linked by the flow. Moreover, to ensure the feature similarity of the diffused pixels, we introduce local spatial similarity loss and temporal contrastive loss in the model re-training stage. Experimental results show that our TDo-Dif scheme helps the adaptive model achieve 51.92% and 53.84% mean intersection-over-union (mIoU) on two publicly available natural foggy datasets (Foggy Zurich and Foggy Driving), which exceeds the state-of-the-art unsupervised domain adaptive semantic segmentation methods. The proposed method can also be applied to non-sequential images in the target domain by considering only spatial similarity.

MLAF-CapsNet: Multi-lane atrous feature fusion capsule network with contrast limited adaptive histogram equalization for brain tumor classification from MRI images

Convolutional neural networks (CNNs) for automatic classification and medical image diagnosis have recently displayed a remarkable performance. However, the CNNs fail to recognize original images rotated and oriented differently, limiting their performance. This paper presents a new capsule network (CapsNet) based framework known as the multi-lane atrous feature fusion capsule network (MLAF-CapsNet) for brain tumor type classification. The MLAF-CapsNet consists of atrous and CLAHE, where the atrous increases receptive fields and maintains spatial representation, whereas the CLAHE is used as a base layer that uses an improved adaptive histogram equalization (AHE) to enhance the input images. The proposed method is evaluated using whole-brain tumor and segmented tumor datasets. The efficiency performance of the two datasets is explored and compared. The experimental results of the MLAF-CapsNet show better accuracies (93.40% and 96.60%) and precisions (94.21% and 96.55%) in feature extraction based on the original images from the two datasets than the traditional CapsNet (78.93% and 97.30%). Based on the two datasets’ augmentation, the proposed method achieved the best accuracy (98.48% and 98.82%) and precisions (98.88% and 98.58%) in extracting features compared to the traditional CapsNet. Our results indicate that the proposed method can successfully improve brain tumor classification problems and support radiologists in medical diagnostics.

Spatial-Spectral Density Peaks-Based Discriminant Analysis for Membranous Nephropathy Classification Using Microscopic Hyperspectral Images

The traditional differential diagnosis of membranous nephropathy (MN) mainly relies on clinical symptoms, serological examination and optical renal biopsy. However, there is a probability of false positives in the optical inspection results, and it is unable to detect the change of biochemical components, which poses an obstacle to pathogenic mechanism analysis. Microscopic hyperspectral imaging can reveal detailed component information of immune complexes, but the high dimensionality of microscopic hyperspectral image brings difficulties and challenges to image processing and disease diagnosis. In this paper, a novel classification framework, including spatial-spectral density peaks-based discriminant analysis (SSDP), is proposed for intelligent diagnosis of MN using a microscopic hyperspectral pathological dataset. SSDP constructs a set of graphs describing intrinsic structure of MHSI in both spatial and spectral domains by employing density peak clustering. In the process of graph embedding, low-dimensional features with important diagnostic information in the immune complex are obtained by compacting the spatial-spectral local intra-class pixels while separating the spectral inter-class pixels. For the MN recognition task, a support vector machine (SVM) is used to classify pixels in the low-dimensional space. Experimental validation data employ two types of MN that are difficult to distinguish with optical microscope, including primary MN and hepatitis B virus-associated MN. Experimental results show that the proposed SSDP achieves a sensitivity of 99.36%, which has potential clinical value for automatic diagnosis of MN.

Tongue Tumor Detection in Hyperspectral Images Using Deep Learning Semantic Segmentation

OBJECTIVE

The utilization of hyperspectral imaging (HSI) in real-time tumor segmentation during a surgery have recently received much attention, but it remains a very challenging task.

METHODS

In this work, we propose semantic segmentation methods, and compare them with other relevant deep learning algorithms for tongue tumor segmentation. To the best of our knowledge, this is the first work using deep learning semantic segmentation for tumor detection in HSI data using channel selection, and accounting for more spatial tissue context, and global comparison between the prediction map, and the annotation per sample. Results, and Conclusion: On a clinical data set with tongue squamous cell carcinoma, our best method obtains very strong results of average dice coefficient, and area under the ROC-curve of [Formula: see text], and [Formula: see text], respectively on the original spatial image size. The results show that a very good performance can be achieved even with a limited amount of data. We demonstrate that important information regarding tumor decision is encoded in various channels, but some channel selection, and filtering is beneficial over the full spectra. Moreover, we use both visual (VIS), and near-infrared (NIR) spectrum, rather than commonly used only VIS spectrum; although VIS spectrum is generally of higher significance, we demonstrate NIR spectrum is crucial for tumor capturing in some cases.

SIGNIFICANCE

The HSI technology augmented with accurate deep learning algorithms has a huge potential to be a promising alternative to digital pathology or a doctors' supportive tool in real-time surgeries.

Unsupervised Clustering of Hyperspectral Paper Data Using t-SNE

For a suspected forgery that involves the falsification of a document or its contents, the investigator will primarily analyze the document’s paper and ink in order to establish the authenticity of the subject under investigation. As a non-destructive and contactless technique, Hyperspectral Imaging (HSI) is gaining popularity in the field of forensic document analysis. HSI returns more information compared to conventional three channel imaging systems due to the vast number of narrowband images recorded across the electromagnetic spectrum. As a result, HSI can provide better classification results. In this publication, we present results of an approach known as the t-Distributed Stochastic Neighbor Embedding (t-SNE) algorithm, which we have applied to HSI paper data analysis. Even though t-SNE has been widely accepted as a method for dimensionality reduction and visualization of high dimensional data, its usefulness has not yet been evaluated for the classification of paper data. In this research, we present a hyperspectral dataset of paper samples, and evaluate the clustering quality of the proposed method both visually and quantitatively. The t-SNE algorithm shows exceptional discrimination power when compared to traditional PCA with k-means clustering, in both visual and quantitative evaluations.

CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions

Data-driven computational approaches have evolved to enable extraction of information from medical images with a reliability, accuracy and speed which is already transforming their interpretation and exploitation in clinical practice. While similar benefits are longed for in the field of interventional imaging, this ambition is challenged by a much higher heterogeneity. Clinical workflows within interventional suites and operating theatres are extremely complex and typically rely on poorly integrated intra-operative devices, sensors, and support infrastructures. Taking stock of some of the most exciting developments in machine learning and artificial intelligence for computer assisted interventions, we highlight the crucial need to take context and human factors into account in order to address these challenges. Contextual artificial intelligence for computer assisted intervention, or CAI4CAI, arises as an emerging opportunity feeding into the broader field of surgical data science. Central challenges being addressed in CAI4CAI include how to integrate the ensemble of prior knowledge and instantaneous sensory information from experts, sensors and actuators; how to create and communicate a faithful and actionable shared representation of the surgery among a mixed human-AI actor team; how to design interventional systems and associated cognitive shared control schemes for online uncertainty-aware collaborative decision making ultimately producing more precise and reliable interventions.

Adaptive deep learning for head and neck cancer detection using hyperspectral imaging

It can be challenging to detect tumor margins during surgery for complete resection. The purpose of this work is to develop a novel learning method that learns the difference between the tumor and benign tissue adaptively for cancer detection on hyperspectral images in an animal model. Specifically, an auto-encoder network is trained based on the wavelength bands on hyperspectral images to extract the deep information to create a pixel-wise prediction of cancerous and benign pixel. According to the output hypothesis of each pixel, the misclassified pixels would be reclassified in the right prediction direction based on their adaptive weights. The auto-encoder network is again trained based on these updated pixels. The learner can adaptively improve the ability to identify the cancer and benign tissue by focusing on the misclassified pixels, and thus can improve the detection performance. The adaptive deep learning method highlighting the tumor region proved to be accurate in detecting the tumor boundary on hyperspectral images and achieved a sensitivity of 92.32% and a specificity of 91.31% in our animal experiments. This adaptive learning method on hyperspectral imaging has the potential to provide a noninvasive tool for tumor detection, especially, for the tumor whose margin is indistinct and irregular.

Hyperspectral imaging for head and neck cancer detection: specular glare and variance of the tumor margin in surgical specimens

Head and neck squamous cell carcinoma (SCC) is primarily managed by surgical cancer resection. Recurrence rates after surgery can be as high as 55%, if residual cancer is present. Hyperspectral imaging (HSI) is evaluated for detection of SCC in ex-vivo surgical specimens. Several machine learning methods are investigated, including convolutional neural networks (CNNs) and a spectral-spatial classification framework based on support vector machines. Quantitative results demonstrate that additional data preprocessing and unsupervised segmentation can improve CNN results to achieve optimal performance. The methods are trained in two paradigms, with and without specular glare. Classifying regions that include specular glare degrade the overall results, but the combination of the CNN probability maps and unsupervised segmentation using a majority voting method produces an area under the curve value of 0.81 [0.80, 0.83]. As the wavelengths of light used in HSI can penetrate different depths into biological tissue, cancer margins may change with depth and create uncertainty in the ground truth. Through serial histological sectioning, the variance in the cancer margin with depth is investigated and paired with qualitative classification heat maps using the methods proposed for the testing group of SCC patients. The results determined that the validity of the top section alone as the ground truth may be limited to 1 to 2 mm. The study of specular glare and margin variation provided better understanding of the potential of HSI for the use in the operating room.

Intraoperative multispectral and hyperspectral label-free imaging: A systematic review of in vivo clinical studies

Multispectral and hyperspectral imaging (HSI) are emerging optical imaging techniques with the potential to transform the way surgery is performed but it is not clear whether current systems are capable of delivering real-time tissue characterization and surgical guidance. We conducted a systematic review of surgical in vivo label-free multispectral and HSI systems that have been assessed intraoperatively in adult patients, published over a 10-year period to May 2018. We analysed 14 studies including 8 different HSI systems. Current in-vivo HSI systems generate an intraoperative tissue oxygenation map or enable tumour detection. Intraoperative tissue oxygenation measurements may help to predict those patients at risk of postoperative complications and in-vivo intraoperative tissue characterization may be performed with high specificity and sensitivity. All systems utilized a line-scanning or wavelength-scanning method but the spectral range and number of spectral bands employed varied significantly between studies and according to the system's clinical aim. The time to acquire a hyperspectral cube dataset ranged between 5 and 30 seconds. No safety concerns were reported in any studies. A small number of studies have demonstrated the capabilities of intraoperative in-vivo label-free HSI but further work is needed to fully integrate it into the current surgical workflow.

Tissue classification of oncologic esophageal resectates based on hyperspectral data

PURPOSE

Esophageal carcinoma is the eighth most common cancer worldwide. Esophageal resection with gastric pull-up is a potentially curative therapeutic option. After this procedure, the specimen is examined by the pathologist to confirm complete removal of the cancer. An intraoperative analysis of the resectate would be less time-consuming and therefore improve patient safety.

METHODS

Hyperspectral imaging (HSI) is a relatively new modality, which has shown promising results for the detection of tumors. Automatic approaches could support the surgeon in the visualization of tumor margins. Therefore, we evaluated four supervised classification algorithms: random forest, support vector machines (SVM), multilayer perceptron, and k-nearest neighbors to differentiate malignant from healthy tissue based on HSI recordings of esophago-gastric resectates in 11 patients.

RESULTS

The best performances were obtained with a cancerous tissue detection of 63% sensitivity and 69% specificity with the SVM. In a leave-one patient-out cross-validation, the classification showed larger performance differences according to the patient data used. In less than 1 s, data classification and visualization was shown.

CONCLUSION

In this work, we successfully tested several classification algorithms for the automatic detection of esophageal carcinoma in resected tissue. A larger data set and a combination of several methods would probably increase the performance. Moreover, the implementation of software tools for intraoperative tumor boundary visualization will further support the surgeon during oncologic operations.

[Hyperspectral imaging of gastrointestinal anastomoses]

INTRODUCTION

Anastomotic insufficiency (AI) remains the most feared surgical complication in gastrointestinal surgery, which is closely associated with a prolonged inpatient hospital stay and significant postoperative mortality. Hyperspectral imaging (HSI) is a relatively new medical imaging procedure which has proven to be promising in tissue identification as well as in the analysis of tissue oxygenation and water content. Until now, no data exist on the in vivo HSI analysis of gastrointestinal anastomoses.

METHODS

Intraoperative images were obtained using the TIVITA™ tissue system HSI camera from Diaspective Vision GmbH (Pepelow, Germany). In 47 patients who underwent gastrointestinal surgery with esophageal, gastric, pancreatic, small bowel or colorectal anastomoses, 97 assessable recordings were generated. Parameters obtained at the sites of the anastomoses included tissue oxygenation (StO₂), the tissue hemoglobin index (THI), near-infrared (NIR) perfusion index, and tissue water index (TWI).

RESULTS

Obtaining and analyzing the intraoperative images with this non-invasive imaging system proved practicable and delivered good results on a consistent basis. A NIR gradient along and across the anastomosis was observed and, furthermore, analysis of the tissue water and oxygenation content showed specific changes at the site of anastomosis.

CONCLUSION

The HSI method provides a non-contact, non-invasive, intraoperative imaging procedure without the use of a contrast medium, which enables a real-time analysis of physiological anastomotic parameters, which may contribute to determine the "ideal" anastomotic region. In light of this, the establishment of this methodology in the field of visceral surgery, enabling the generation of normal or cut off values for different gastrointestinal anastomotic types, is an obvious necessity.

In-Vivo and Ex-Vivo Tissue Analysis through Hyperspectral Imaging Techniques: Revealing the Invisible Features of Cancer

In contrast to conventional optical imaging modalities, hyperspectral imaging (HSI) is able to capture much more information from a certain scene, both within and beyond the visual spectral range (from 400 to 700 nm). This imaging modality is based on the principle that each material provides different responses to light reflection, absorption, and scattering across the electromagnetic spectrum. Due to these properties, it is possible to differentiate and identify the different materials/substances presented in a certain scene by their spectral signature. Over the last two decades, HSI has demonstrated potential to become a powerful tool to study and identify several diseases in the medical field, being a non-contact, non-ionizing, and a label-free imaging modality. In this review, the use of HSI as an imaging tool for the analysis and detection of cancer is presented. The basic concepts related to this technology are detailed. The most relevant, state-of-the-art studies that can be found in the literature using HSI for cancer analysis are presented and summarized, both in-vivo and ex-vivo. Lastly, we discuss the current limitations of this technology in the field of cancer detection, together with some insights into possible future steps in the improvement of this technology.

Optical biopsy of head and neck cancer using hyperspectral imaging and convolutional neural networks

For patients undergoing surgical cancer resection of squamous cell carcinoma (SCCa), cancer-free surgical margins are essential for good prognosis. We developed a method to use hyperspectral imaging (HSI), a noncontact optical imaging modality, and convolutional neural networks (CNNs) to perform an optical biopsy of ex-vivo, surgical gross-tissue specimens, collected from 21 patients undergoing surgical cancer resection. Using a cross-validation paradigm with data from different patients, the CNN can distinguish SCCa from normal aerodigestive tract tissues with an area under the receiver operator curve (AUC) of 0.82. Additionally, normal tissue from the upper aerodigestive tract can be subclassified into squamous epithelium, muscle, and gland with an average AUC of 0.94. After separately training on thyroid tissue, the CNN can differentiate between thyroid carcinoma and normal thyroid with an AUC of 0.95, 92% accuracy, 92% sensitivity, and 92% specificity. Moreover, the CNN can discriminate medullary thyroid carcinoma from benign multinodular goiter (MNG) with an AUC of 0.93. Classical-type papillary thyroid carcinoma is differentiated from MNG with an AUC of 0.91. Our preliminary results demonstrate that an HSI-based optical biopsy method using CNNs can provide multicategory diagnostic information for normal and cancerous head-and-neck tissue, and more patient data are needed to fully investigate the potential and reliability of the proposed technique.

Adversarial training with cycle consistency for unsupervised super-resolution in endomicroscopy

•

We propose an unsupervised SR framework based on an adversarial deep neural network with a physically-inspired cycle consistency.

•

Our framework can exploit HR images, regardless of the domain where they are coming from, to transfer the quality of the HR images to the initial LR images.

•

Our system can be particularly useful in all situations where pairs of LR/HR are not available during the training of a super-resolution algorithm.

•

Our quantitative analysis, validated using a database of 238 endomicroscopy video sequences from 143 patients, shows the ability of the pipeline to produce convincing super-resolved images.

•

A Mean Opinion Score (MOS) study also confirms this quantitative image quality assessment.

In recent years, endomicroscopy has become increasingly used for diagnostic purposes and interventional guidance. It can provide intraoperative aids for real-time tissue characterization and can help to perform visual investigations aimed for example to discover epithelial cancers. Due to physical constraints on the acquisition process, endomicroscopy images, still today have a low number of informative pixels which hampers their quality. Post-processing techniques, such as Super-Resolution (SR), are a potential solution to increase the quality of these images. SR techniques are often supervised, requiring aligned pairs of low-resolution (LR) and high-resolution (HR) images patches to train a model. However, in our domain, the lack of HR images hinders the collection of such pairs and makes supervised training unsuitable. For this reason, we propose an unsupervised SR framework based on an adversarial deep neural network with a physically-inspired cycle consistency, designed to impose some acquisition properties on the super-resolved images. Our framework can exploit HR images, regardless of the domain where they are coming from, to transfer the quality of the HR images to the initial LR images. This property can be particularly useful in all situations where pairs of LR/HR are not available during the training. Our quantitative analysis, validated using a database of 238 endomicroscopy video sequences from 143 patients, shows the ability of the pipeline to produce convincing super-resolved images. A Mean Opinion Score (MOS) study also confirms this quantitative image quality assessment.

Cancer Detection Using Hyperspectral Imaging and Evaluation of the Superficial Tumor Margin Variance with Depth

Head and neck squamous cell carcinoma (SCCa) is primarily managed by surgical resection. Recurrence rates after surgery can be as high as 55% if residual cancer is present. In this study, hyperspectral imaging (HSI) is evaluated for detection of SCCa in ex-vivo surgical specimens. Several methods are investigated, including convolutional neural networks (CNNs) and a spectral-spatial variant of support vector machines. Quantitative results demonstrate that additional processing and unsupervised filtering can improve CNN results to achieve optimal performance. Classifying regions that include specular glare, the average AUC is increased from 0.73 [0.71, 0.75 (95% confidence interval)] to 0.81 [0.80, 0.83] through an unsupervised filtering and majority voting method described. The wavelengths of light used in HSI can penetrate different depths into biological tissue, while the cancer margin may change with depth and create uncertainty in the ground-truth. Through serial histological sectioning, the variance in cancer-margin with depth is also investigated and paired with qualitative classification heat maps using the methods proposed for the testing group SCC patients.

Bioluminescence imaging and two-photon microscopy guided laser ablation of GBM decreases tumor burden

Brain tumor delineation and treatment are the main concerns of neurosurgeons in neurosurgical operations. Bridging the gap between imaging/diagnosis and treatment will provide great convenience for neurosurgeons. Here, we developed an optical theranostics platform that helps to delineate the boundary and quantitatively analyze glioblastoma multiforms (GBMs) with bioluminescence imaging (BLI) to guide laser ablation, and we imaged the GBM cells with two-photon microscopy (TPM) to visualize the laser ablation zone in vivo.

Methods: Laser ablation, using the method of coupled ablated path planning with the guidance of BLI, was implemented in vivo for mouse brain tumors. The mapping relationship between semi-quantitative BLI and the laser ablation path was built through the quantitative tumor burden. The mapping was reflected through coupled ablated path planning. The BLI quantitatively and qualitatively evaluated treatment using laser ablation with the appropriate laser parameters and laser-tissue parameters. These parameters were measured after treatment. Furthermore, histopathological analysis of the brain tissue was conducted to compare the TPM images before and after laser ablation and to evaluate the results of in vivo laser ablation. The local recurrences were measured with three separate cohorts. The weights of all of the mice were measured during the experiment.

Results: Our in vivo BLI data show that the tumor cell numbers were significantly attenuated after treatment with the optical theranostics platform, and the delineation of GBM margins had clear views to guide the laser resection; the fluorescence intensity in vivo of GBMs quantitatively analyzed the rapid progression of GBMs. The laser-tissue parameters under guidance of multimodality imaging ranged between 1.0 mm and 0.1 mm. The accuracy of the laser ablation reached a submillimeter level, and the resection ratio reached more than 99% under the guidance of BLI. The histopathological sections were compared to TPM images, and the results demonstrated that these images highly coincided. The weight index and local recurrence results demonstrated that the therapeutic effect of the optical theranostics platform was significant.

Conclusion: We propose an optical multimodality imaging-guided laser ablation theranostics platform for the treatment of GBMs in an intravital mouse model. The experimental results demonstrated that the integration of multimodality imaging can precisely guide laser ablation for the treatment of GBMs. This preclinical research provides a possibility for the precision treatment of GBMs. The study also provides some theoretical support for clinical research.

Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations

Surgery for brain cancer is a major problem in neurosurgery. The diffuse infiltration into the surrounding normal brain by these tumors makes their accurate identification by the naked eye difficult. Since surgery is the common treatment for brain cancer, an accurate radical resection of the tumor leads to improved survival rates for patients. However, the identification of the tumor boundaries during surgery is challenging. Hyperspectral imaging is a non-contact, non-ionizing and non-invasive technique suitable for medical diagnosis. This study presents the development of a novel classification method taking into account the spatial and spectral characteristics of the hyperspectral images to help neurosurgeons to accurately determine the tumor boundaries in surgical-time during the resection, avoiding excessive excision of normal tissue or unintentionally leaving residual tumor. The algorithm proposed in this study to approach an efficient solution consists of a hybrid framework that combines both supervised and unsupervised machine learning methods. Firstly, a supervised pixel-wise classification using a Support Vector Machine classifier is performed. The generated classification map is spatially homogenized using a one-band representation of the HS cube, employing the Fixed Reference t-Stochastic Neighbors Embedding dimensional reduction algorithm, and performing a K-Nearest Neighbors filtering. The information generated by the supervised stage is combined with a segmentation map obtained via unsupervised clustering employing a Hierarchical K-Means algorithm. The fusion is performed using a majority voting approach that associates each cluster with a certain class. To evaluate the proposed approach, five hyperspectral images of surface of the brain affected by glioblastoma tumor in vivo from five different patients have been used. The final classification maps obtained have been analyzed and validated by specialists. These preliminary results are promising, obtaining an accurate delineation of the tumor area.

Wide-field spectrally resolved quantitative fluorescence imaging system: toward neurosurgical guidance in glioma resection

In high-grade glioma surgery, tumor resection is often guided by intraoperative fluorescence imaging. 5-aminolevulinic acid-induced protoporphyrin IX (PpIX) provides fluorescent contrast between normal brain tissue and glioma tissue, thus achieving improved tumor delineation and prolonged patient survival compared with conventional white-light-guided resection. However, commercially available fluorescence imaging systems rely solely on visual assessment of fluorescence patterns by the surgeon, which makes the resection more subjective than necessary. We developed a wide-field spectrally resolved fluorescence imaging system utilizing a Generation II scientific CMOS camera and an improved computational model for the precise reconstruction of the PpIX concentration map. In our model, the tissue's optical properties and illumination geometry, which distort the fluorescent emission spectra, are considered. We demonstrate that the CMOS-based system can detect low PpIX concentration at short camera exposure times, while providing high-pixel resolution wide-field images. We show that total variation regularization improves the contrast-to-noise ratio of the reconstructed quantitative concentration map by approximately twofold. Quantitative comparison between the estimated PpIX concentration and tumor histopathology was also investigated to further evaluate the system.