Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery

RGB-Image-Based Real-Time Hemodynamic Monitoring of Intraperitoneal Organs in Rats Using a Standard Laparoscopic Imaging System

The aim of this study is to validate an approach to monitor the spatial and temporal hemodynamics of intraperitoneal organs using a commercially available laparoscopic system. The approach to create a spatial map of tissue oxygen saturation (StO2) and total hemoglobin concentration (CHbT) is based on a multiple regression model using Monte Carlo simulation of light transport in tissues to specify relationships between RGB values, oxygenated hemoglobin concentration, and deoxygenated hemoglobin concentration. Experiments with an optical phantom are performed to confirm the ability of the approach to detect changes in StO2 and CHbT under different working distances of the endoscope that may occur during actual surgery. In vivo experiments in rats confirm that the proposed approach can quantitatively monitor changes in StO2 and CHbT induced in the small intestine, liver, and cecum. The proposed approach has the potential as a tool for monitoring intraperitoneal organs in real time during laparoscopy.

Toward real-time margin assessment in breast-conserving surgery with hyperspectral imaging

Margin assessment in breast-conserving surgery (BSC) remains a critical challenge, with 20-25% of cases resulting in inadequate tumor resection, increasing the risk of local recurrence and the need for additional treatment. In this study, we evaluate the diagnostic performance of hyperspectral imaging (HSI) as a non-invasive technique for assessing resection margins in ex vivo lumpectomy specimens. A dataset of over 200 lumpectomy specimens was collected using two hyperspectral cameras, and a classification algorithm was developed to distinguish between healthy and tumor tissue within margins of 0 and 2 mm. The proposed approach achieved its highest diagnostic performance at a 0 mm margin, with a sensitivity of 92%, specificity of 78%, accuracy of 83%, Matthews correlation coefficient of 68%, and an area under the curve of 89%. The entire resection surface could be imaged and evaluated within 10 minutes, providing a rapid and non-invasive alternative to conventional margin assessment techniques. These findings represent a significant advancement toward real-time intraoperative margin assessment, highlighting the potential of HSI to enhance surgical precision and reduce re-excision rates in BCS.

Evaluating Normalization Methods for Robust Spectral Performance Assessments of Hyperspectral Imaging Cameras

Hyperspectral imaging (HSI) technology, which offers both spatial and spectral information, holds significant potential for enhancing diagnostic performance during endoscopy and other medical procedures. However, quantitative evaluation of HSI cameras is challenging due to various influencing factors (e.g., light sources, working distance, and illumination angle) that can alter the reflectance spectra of the same target as these factors vary. Towards robust, universal test methods, we evaluated several data normalization methods aimed at minimizing the impact of these factors. Using a high-resolution HSI camera, we measured the reflectance spectra of diffuse reflectance targets illuminated by two different light sources. These spectra, along with the reference spectra from the target manufacturer, were normalized with nine different methods (e.g., area under the curve, standard normal variate, and centering power methods), followed by a uniform scaling step. We then compared the measured spectra to the reference to evaluate the capability of each normalization method in ensuring a consistent, standardized performance evaluation. Our results demonstrate that normalization can mitigate the impact of some factors during HSI camera evaluation, with performance varying across methods. Generally, noisy spectra pose challenges for normalization methods that rely on limited reflectance values, while methods based on reflectance values across the entire spectrum (such as standard normal variate) perform better. The findings also suggest that absolute reflectance spectral measurements may be less effective for clinical diagnostics, whereas normalized spectral measurements are likely more appropriate. These findings provide a foundation for standardized performance testing of HSI-based medical devices, promoting the adoption of high-quality HSI technology for critical applications such as early cancer detection.

Speckle-illumination spatial frequency domain imaging with a stereo laparoscope for profile-corrected optical property mapping

Significance

Laparoscopic surgery presents challenges in localizing oncological margins due to poor contrast between healthy and malignant tissues. Optical properties can uniquely identify various tissue types and disease states with high sensitivity and specificity, making it a promising tool for surgical guidance. Although spatial frequency domain imaging (SFDI) effectively measures quantitative optical properties, its deployment in laparoscopy is challenging due to the constrained imaging environment. Thus, there is a need for compact structured illumination techniques to enable accurate, quantitative endogenous contrast in minimally invasive surgery.

Aim

We introduce a compact, two-camera laparoscope that incorporates both active stereo depth estimation and speckle-illumination SFDI (si-SFDI) to map profile-corrected, pixel-level absorption (μ a ), and reduced scattering (μ s ' ) optical properties in images of tissues with complex geometries.

Approach

We used a multimode fiber-coupled 639-nm laser illumination to generate high-contrast speckle patterns on the object. These patterns were imaged through a modified commercial stereo laparoscope for optical property estimation via si-SFDI. Compared with the original si-SFDI work, which required ≥ 10 images of randomized speckle patterns for accurate optical property estimations, our approach approximates the DC response using a laser speckle reducer (LSR) and consequently requires only two images. In addition, we demonstrate 3D profilometry using active stereo from low-coherence RGB laser flood illumination. Sample topography was then used to correct for measured intensity variations caused by object height and surface angle differences with respect to a calibration phantom. The low-contrast RGB speckle pattern was blurred using an LSR to approximate incoherent white light illumination. We validated profile-corrected si-SFDI against conventional SFDI in phantoms with simple and complex geometries, as well as in a human finger in vivo time-series constriction study.

Results

Laparoscopic si-SFDI optical property measurements agreed with conventional SFDI measurements when measuring flat tissue phantoms, exhibiting an error of 6.4% for absorption and 5.8% for reduced scattering. Profile-correction improved the accuracy for measurements of phantoms with complex geometries, particularly for absorption, where it reduced the error by 23.7%. An in vivo finger constriction study further validated laparoscopic si-SFDI, demonstrating an error of 8.2% for absorption and 5.8% for reduced scattering compared with conventional SFDI. Moreover, the observed trends in optical properties due to physiological changes were consistent with previous studies.

Conclusions

Our stereo-laparoscopic implementation of si-SFDI provides a simple method to obtain accurate optical property maps through a laparoscope for flat and complex geometries. This has the potential to provide quantitative endogenous contrast for minimally invasive surgical guidance.

[Status Quo of Surgical Navigation]

Surgical navigation, also referred to as computer-assisted or image-guided surgery, is a technique that employs a variety of methods - such as 3D imaging, tracking systems, specialised software, and robotics to support surgeons during surgical interventions. These emerging technologies aim not only to enhance the accuracy and precision of surgical procedures, but also to enable less invasive approaches, with the objective of reducing complications and improving operative outcomes for patients. By harnessing the integration of emerging digital technologies, surgical navigation holds the promise of assisting complex procedures across various medical disciplines. In recent years, the field of surgical navigation has witnessed significant advances. Abdominal surgical navigation, particularly endoscopy, laparoscopic, and robot-assisted surgery, is currently undergoing a phase of rapid evolution. Emphases include image-guided navigation, instrument tracking, and the potential integration of augmented and mixed reality (AR, MR). This article will comprehensively delve into the latest developments in surgical navigation, spanning state-of-the-art intraoperative technologies like hyperspectral and fluorescent imaging, to the integration of preoperative radiological imaging within the intraoperative setting.

Precision at scale: Machine learning revolutionizing laparoscopic surgery

In their recent study published in the World Journal of Clinical Cases, the article found that minimally invasive laparoscopic surgery under general anesthesia demonstrates superior efficacy and safety compared to traditional open surgery for early ovarian cancer patients. This editorial discusses the integration of machine learning in laparoscopic surgery, emphasizing its transformative potential in improving patient outcomes and surgical precision. Machine learning algorithms analyze extensive datasets to optimize procedural techniques, enhance decision-making, and personalize treatment plans. Advanced imaging modalities like augmented reality and real-time tissue classification, alongside robotic surgical systems and virtual reality simulations driven by machine learning, enhance imaging and training techniques, offering surgeons clearer visualization and precise tissue manipulation. Despite promising advancements, challenges such as data privacy, algorithm bias, and regulatory hurdles need addressing for the responsible deployment of machine learning technologies. Interdisciplinary collaborations and ongoing technological innovations promise further enhancement in laparoscopic surgery, fostering a future where personalized medicine and precision surgery redefine patient care.

Separating Surface Reflectance from Volume Reflectance in Medical Hyperspectral Imaging

Hyperspectral imaging has shown great promise for diagnostic applications, particularly in cancer surgery. However, non-bulk tissue-related spectral variations complicate the data analysis. Common techniques, such as standard normal variate normalization, often lead to a loss of amplitude and scattering information. This study investigates a novel approach to address these spectral variations in hyperspectral images of optical phantoms and excised human breast tissue. Our method separates surface and volume reflectance, hypothesizing that spectral variability arises from significant variations in surface reflectance across pixels. An illumination setup was developed to measure samples with a hyperspectral camera from different axial positions but with identical zenith angles. This configuration, combined with a novel data analysis approach, allows for the estimation and separation of surface reflectance for each direction and volume reflectance across all directions. Validated with optical phantoms, our method achieved an 83% reduction in spectral variability. Its functionality was further demonstrated in excised human breast tissue. Our method effectively addresses variations caused by surface reflectance or glare while conserving surface reflectance information, which may enhance sample analysis and evaluation. It benefits samples with unknown refractive index spectra and can be easily adapted and applied across a wide range of fields where hyperspectral imaging is used.

Handheld hyperspectral imaging as a tool for the post-mortem interval estimation of human skeletal remains

In forensic medicine, estimating human skeletal remains' post-mortem interval (PMI) can be challenging. Following death, bones undergo a series of chemical and physical transformations due to their interactions with the surrounding environment. Post-mortem changes have been assessed using various methods, but estimating the PMI of skeletal remains could still be improved. We propose a new methodology with handheld hyperspectral imaging (HSI) system based on the first results from 104 human skeletal remains with PMIs ranging between 1 day and 2000 years. To differentiate between forensic and archaeological bone material, the Convolutional Neural Network analyzed 65.000 distinct diagnostic spectra: the classification accuracy was 0.58, 0.62, 0.73, 0.81, and 0.98 for PMIs of 0 week-2 weeks, 2 weeks-6 months, 6 months-1 year, 1 year-10 years, and >100 years, respectively. In conclusion, HSI can be used in forensic medicine to distinguish bone materials >100 years old from those <10 years old with an accuracy of 98%. The model has adequate predictive performance, and handheld HSI could serve as a novel approach to objectively and accurately determine the PMI of human skeletal remains.

A dual-camera hyperspectral laparoscopic imaging system

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. However, there are still prevalent issues surrounding intracorporeal surgery, such as iatrogenic injury, anastomotic leakage, or the presence of positive tumor margins after resection. Current approaches to address these issues and advance laparoscopic imaging techniques often involve fluorescence imaging agents, such as indocyanine green (ICG), to improve visualization, but these have drawbacks. Hyperspectral imaging (HSI) is an emerging optical imaging modality that takes advantage of spectral characteristics of different tissues. Various applications include tissue classification and digital pathology. In this study, we developed a dual-camera system for high-speed hyperspectral imaging. This includes the development of a custom application interface and corresponding hardware setup. Characterization of the system was performed, including spectral accuracy and spatial resolution, showing little sacrifice in speed for the approximate doubling of the covered spectral range, with our system acquiring 29 spectral images from 460-850 nm. Reference color tiles with various reflectance profiles were imaged and a RMSE of 3.56 ± 1.36% was achieved. Sub-millimeter resolution was shown at 7 cm working distance for both hyperspectral cameras. Finally, we image ex vivo tissues, including porcine stomach, liver, intestine, and kidney with our system and use a high-resolution, radiometrically calibrated spectrometer for comparison and evaluation of spectral fidelity. The dual-camera hyperspectral laparoscopic imaging system can have immediate applications in various surgeries.

Diffuse reflectance spectroscopy for colorectal cancer surgical guidance: towards real-time tissue characterization and new biomarkers

Colorectal cancer (CRC) is the third most common and second most deadly type of cancer worldwide, representing 11.3% of the diagnosed cancer cases and resulting in 10.2% (0.88 million) of the cancer related deaths in 2020. CRCs are typically detected at the late stage, which leads to high mortality and morbidity. Mortality and poor prognosis are partially caused by cancer recurrence and postoperative complications. Patient survival could be increased by improving precision in surgical resection using accurate surgical guidance tools based on diffuse reflectance spectroscopy (DRS). DRS enables real-time tissue identification for potential cancer margin delineation through determination of the circumferential resection margin (CRM), while also supporting non-invasive and label-free approaches for laparoscopic surgery to avoid short-term complications of open surgery as suitable. In this study, we have estimated the scattering properties and chromophore concentrations based on 2949 DRS measurements of freshly excised ex vivo specimens of 47 patients, and used this estimation to classify normal colorectal wall (CW), fat and tumor tissues. DRS measurements were performed with fiber-optic probes of 630 μm source-detector distance (SDD; probe 1) and 2500 μm SDD (probe 2) to measure tissue layers ∼0.5-1 mm and ∼0.5-2 mm deep, respectively. By using the 5-fold cross-validation of machine learning models generated with the classification and regression tree (CART) algorithm, we achieved 95.9 ± 0.7% sensitivity, 98.9 ± 0.3% specificity, 90.2 ± 0.4% accuracy, and 95.5 ± 0.3% AUC for probe 1. Similarly, we achieved 96.9 ± 0.8% sensitivity, 98.9 ± 0.2% specificity, 94.0 ± 0.4% accuracy, and 96.7 ± 0.4% AUC for probe 2.

HeiPorSPECTRAL - the Heidelberg Porcine HyperSPECTRAL Imaging Dataset of 20 Physiological Organs

Hyperspectral Imaging (HSI) is a relatively new medical imaging modality that exploits an area of diagnostic potential formerly untouched. Although exploratory translational and clinical studies exist, no surgical HSI datasets are openly accessible to the general scientific community. To address this bottleneck, this publication releases HeiPorSPECTRAL ( https://www.heiporspectral.org ; https://doi.org/10.5281/zenodo.7737674 ), the first annotated high-quality standardized surgical HSI dataset. It comprises 5,758 spectral images acquired with the TIVITA® Tissue and annotated with 20 physiological porcine organs from 8 pigs per organ distributed over a total number of 11 pigs. Each HSI image features a resolution of 480 × 640 pixels acquired over the 500-1000 nm wavelength range. The acquisition protocol has been designed such that the variability of organ spectra as a function of several parameters including the camera angle and the individual can be assessed. A comprehensive technical validation confirmed both the quality of the raw data and the annotations. We envision potential reuse within this dataset, but also its reuse as baseline data for future research questions outside this dataset. Measurement(s) Spectral Reflectance Technology Type(s) Hyperspectral Imaging Sample Characteristic - Organism Sus scrofa.

Tissue Classification of Breast Cancer by Hyperspectral Unmixing

(1) Background: Assessing the resection margins during breast-conserving surgery is an important clinical need to minimize the risk of recurrent breast cancer. However, currently there is no technique that can provide real-time feedback to aid surgeons in the margin assessment. Hyperspectral imaging has the potential to overcome this problem. To classify resection margins with this technique, a tissue discrimination model should be developed, which requires a dataset with accurate ground-truth labels. However, establishing such a dataset for resection specimens is difficult. (2) Methods: In this study, we therefore propose a novel approach based on hyperspectral unmixing to determine which pixels within hyperspectral images should be assigned to the ground-truth labels from histopathology. Subsequently, we use this hyperspectral-unmixing-based approach to develop a tissue discrimination model on the presence of tumor tissue within the resection margins of ex vivo breast lumpectomy specimens. (3) Results: In total, 372 measured locations were included on the lumpectomy resection surface of 189 patients. We achieved a sensitivity of 0.94, specificity of 0.85, accuracy of 0.87, Matthew's correlation coefficient of 0.71, and area under the curve of 0.92. (4) Conclusion: Using this hyperspectral-unmixing-based approach, we demonstrated that the measured locations with hyperspectral imaging on the resection surface of lumpectomy specimens could be classified with excellent performance.

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging

Significance

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging.

Aim

With this review paper, researchers will (1) understand the technological trends of upcoming spectral cameras, (2) understand new specific applications that portable spectral imaging unlocked, and (3) evaluate proper spectral imaging systems for their specific applications.

Approach

We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition.

Results

There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings.

Conclusions

Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Spectral imaging enables contrast agent-free real-time ischemia monitoring in laparoscopic surgery

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent-free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning-based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

A High-Speed Hyperspectral Laparoscopic Imaging System

Minimally invasive surgery (MIS) has expanded broadly in the field of abdominal and pelvic surgery. Laparoscopic and robotic surgery has improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery across various abdominal procedures. The goal of this study is to establish the feasibility of implementing high-speed hyperspectral imaging into a standard laparoscopic setup and exploring its benefit to common intracorporeal procedures. A hyperspectral laparoscopic imaging system was constructed using a customized hyperspectral camera alongside a standard rigid laparoscope and was validated for both spectral and spatial accuracy. Demosaicing methods were investigated for improved full-resolution visualization. Hyperspectral cameras with different spectral ranges were considered and compared with one another alongside two different light sources to determine the most effective configuration. Finally, different porcine tissues were imaged ex-vivo to test the capabilities of the system and spectral footprints of the various tissues were extracted. The tissue was also imaged in a phantom to simulate the system's use in MIS. The results demonstrated a hyperspectral laparoscopic imaging system that could provide quantitative, diagnostic information while not disrupting normal workflow nor adding excessive weight to the laparoscopic setup. The high-speed hyperspectral laparoscopic imaging system can have immediate applications in image-guided surgery.

Visible- and near-infrared hyperspectral imaging for the quantitative analysis of PD-L1+ cells in human lymphomas: Comparison with fluorescent multiplex immunohistochemistry

INTRODUCTION

We analyzed the expression of PD-L1 in human lymphomas using hyperspectral imaging (HSI) compared to visual assessment (VA) and conventional digital image analysis (DIA) to strengthen further the value of HSI as a tool for the evaluation of brightfield-based immunohistochemistry (IHC). In addition, fluorescent multiplex immunohistochemistry (mIHC) was used as a second detection method to analyze the impact of a different detection method.

MATERIAL AND METHODS

18 cases (6 follicular lymphomas and 12 diffuse large B-cell lymphomas) were stained for PD-L1 by IHC and for PD-L1, CD3, and CD8 by fluorescent mIHC. The percentage of positively stained cells was evaluated with VA, HSI, and DIA for IHC and VA and DIA for mIHC. Results were compared between the different methods of detection and analysis.

RESULTS

An overall high concordance was found between VA, HSI, and DIA in IHC (Cohens Kappa = 0.810^VA/HSI, 0.710 ^VA/DIA, and 0.516 ^HSI/DIA) and for VA^mIHCversus DIA^mIHC (Cohens Kappa = 0.894). Comparing IHC and mIHC general agreement differed depending on the methods compared but reached at most a moderate agreement (Coheńs Kappa between 0.250 and 0.483). This is reflected by the significantly higher percentage of PD-L1+ cells found with mIHC (p^Friedman = 0.014).

CONCLUSION

Our study shows a good concordance for the different analysis methods. Compared to VA and DIA, HSI proved to be a reliable tool for assessing IHC. Understanding the regulation of PD-L1 expression will further enlighten the role of PD-L1 as a biomarker. Therefore it is necessary to develop an instrument, such as HSI, which can offer a reliable and objective evaluation of PD-L1 expression.

Estimating quantitative physiological and morphological tissue parameters of murine tumor models using hyperspectral imaging and optical profilometry

Understanding tumors and their microenvironment are essential for successful and accurate disease diagnosis. Tissue physiology and morphology are altered in tumors compared to healthy tissues, and there is a need to monitor tumors and their surrounding tissues, including blood vessels, non-invasively. This preliminary study utilizes a multimodal optical imaging system combining hyperspectral imaging (HSI) and three-dimensional (3D) optical profilometry (OP) to capture hyperspectral images and surface shapes of subcutaneously grown murine tumor models. Hyperspectral images are corrected with 3D OP data and analyzed using the inverse-adding doubling (IAD) method to extract tissue properties such as melanin volume fraction and oxygenation. Blood vessels are segmented using the B-COSFIRE algorithm from oxygenation maps. From 3D OP data, tumor volumes are calculated and compared to manual measurements using a vernier caliper. Results show that tumors can be distinguished from healthy tissue based on most extracted tissue parameters ( p < 0.05 ). Furthermore, blood oxygenation is 50% higher within the blood vessels than in the surrounding tissue, and tumor volumes calculated using 3D OP agree within 26% with manual measurements using a vernier caliper. Results suggest that combining HSI and OP could provide relevant quantitative information about tumors and improve the disease diagnosis.

Signal to noise ratio quantifies the contribution of spectral channels to classification of human head and neck tissues ex vivo using deep learning and multispectral imaging

SIGNIFICANCE

Accurate identification of tissues is critical for performing safe surgery. Combining multispectral imaging (MSI) with deep learning is a promising approach to increasing tissue discrimination and classification. Evaluating the contributions of spectral channels to tissue discrimination is important for improving MSI systems.

AIM

Develop a metric to quantify the contributions of individual spectral channels to tissue classification in MSI.

APPROACH

MSI was integrated into a digital operating microscope with three sensors and seven illuminants. Two convolutional neural network (CNN) models were trained to classify 11 head and neck tissue types using white light (RGB) or MSI images. The signal to noise ratio (SNR) of spectral channels was compared with the impact of channels on tissue classification performance as determined using CNN visualization methods.

RESULTS

Overall tissue classification accuracy was higher with use of MSI images compared with RGB images, both for classification of all 11 tissue types and binary classification of nerve and parotid ( p < 0.001 ). Removing spectral channels with SNR > 20 reduced tissue classification accuracy.

CONCLUSIONS

The spectral channel SNR is a useful metric for both understanding CNN tissue classification and quantifying the contributions of different spectral channels in an MSI system.

Insights into Biochemical Sources and Diffuse Reflectance Spectral Features for Colorectal Cancer Detection and Localization

Colorectal cancer (CRC) is the third most common and second most deadly type of cancer worldwide. Early detection not only reduces mortality but also improves patient prognosis by allowing the use of minimally invasive techniques to remove cancer while avoiding major surgery. Expanding the use of microsurgical techniques requires accurate diagnosis and delineation of the tumor margins in order to allow complete excision of cancer. We have used diffuse reflectance spectroscopy (DRS) to identify the main optical CRC biomarkers and to optimize parameters for the integration of such technologies into medical devices. A total number of 2889 diffuse reflectance spectra were collected in ex vivo specimens from 47 patients. Short source-detector distance (SDD) and long-SDD fiber-optic probes were employed to measure tissue layers from 0.5 to 1 mm and from 0.5 to 1.9 mm deep, respectively. The most important biomolecules contributing to differentiating DRS between tissue types were oxy- and deoxy-hemoglobin (Hb and HbO₂), followed by water and lipid. Accurate tissue classification and potential DRS device miniaturization using Hb, HbO₂, lipid and water data were achieved particularly well within the wavelength ranges 350-590 nm and 600-1230 nm for the short-SDD probe, and 380-400 nm, 420-610 nm, and 650-950 nm for the long-SDD probe.

Automatic optical biopsy for colorectal cancer using hyperspectral imaging and artificial neural networks

BACKGROUND

Intraoperative identification of cancerous tissue is fundamental during oncological surgical or endoscopic procedures. This relies on visual assessment supported by histopathological evaluation, implying a longer operative time. Hyperspectral imaging (HSI), a contrast-free and contactless imaging technology, provides spatially resolved spectroscopic analysis, with the potential to differentiate tissue at a cellular level. However, HSI produces "big data", which is impossible to directly interpret by clinicians. We hypothesize that advanced machine learning algorithms (convolutional neural networks-CNNs) can accurately detect colorectal cancer in HSI data.

METHODS

In 34 patients undergoing colorectal resections for cancer, immediately after extraction, the specimen was opened, the tumor-bearing section was exposed and imaged using HSI. Cancer and normal mucosa were categorized from histopathology. A state-of-the-art CNN was developed to automatically detect regions of colorectal cancer in a hyperspectral image. Accuracy was validated with three levels of cross-validation (twofold, fivefold, and 15-fold).

RESULTS

32 patients had colorectal adenocarcinomas confirmed by histopathology (9 left, 11 right, 4 transverse colon, and 9 rectum). 6 patients had a local initial stage (T1-2) and 26 had a local advanced stage (T3-4). The cancer detection performance of the CNN using 15-fold cross-validation showed high sensitivity and specificity (87% and 90%, respectively) and a ROC-AUC score of 0.95 (considered outstanding). In the T1-2 group, the sensitivity and specificity were 89% and 90%, respectively, and in the T3-4 group, the sensitivity and specificity were 81% and 93%, respectively.

CONCLUSIONS

Automatic colorectal cancer detection on fresh specimens using HSI, using a properly trained CNN is feasible and accurate, even with small datasets, regardless of the local tumor extension. In the near future, this approach may become a useful intraoperative tool during oncological endoscopic and surgical procedures, and may result in precise and non-destructive optical biopsies to support objective and consistent tumor-free resection margins.

Towards Automated Classification of Zooplankton Using Combination of Laser Spectral Techniques and Advanced Chemometrics

Zooplankton identification has been the subject of many studies. They are mainly based on the analysis of photographs (computer vision). However, spectroscopic techniques can be a good alternative due to the valuable additional information that they provide. We tested the performance of several chemometric techniques (principal component analysis (PCA), non-negative matrix factorisation (NMF), and common dimensions and specific weights analysis (CCSWA of ComDim)) for the unsupervised classification of zooplankton species based on their spectra. The spectra were obtained using laser-induced breakdown spectroscopy (LIBS) and Raman spectroscopy. It was convenient to assess the discriminative power in terms of silhouette metrics (Sil). The LIBS data were substantially more useful for the task than the Raman spectra, although the best results were achieved for the combined LIBS + Raman dataset (best Sil = 0.67). Although NMF (Sil = 0.63) and ComDim (Sil = 0.39) gave interesting information in the loadings, PCA was generally enough for the discrimination based on the score graphs. The distinguishing between Calanoida and Euphausiacea crustaceans and Limacina helicina sea snails has proved possible, probably because of their different mineral compositions. Conversely, arrow worms (Parasagitta elegans) usually fell into the same class with Calanoida despite the differences in their Raman spectra.

Comparison of preprocessing techniques to reduce nontissue-related variations in hyperspectral reflectance imaging

SIGNIFICANCE

Hyperspectral reflectance imaging can be used in medicine to identify tissue types, such as tumor tissue. Tissue classification algorithms are developed based on, e.g., machine learning or principle component analysis. For the development of these algorithms, data are generally preprocessed to remove variability in data not related to the tissue itself since this will improve the performance of the classification algorithm. In hyperspectral imaging, the measured spectra are also influenced by reflections from the surface (glare) and height variations within and between tissue samples.

AIM

To compare the ability of different preprocessing algorithms to decrease variations in spectra induced by glare and height differences while maintaining contrast based on differences in optical properties between tissue types.

APPROACH

We compare eight preprocessing algorithms commonly used in medical hyperspectral imaging: standard normal variate, multiplicative scatter correction, min-max normalization, mean centering, area under the curve normalization, single wavelength normalization, first derivative, and second derivative. We investigate conservation of contrast stemming from differences in: blood volume fraction, presence of different absorbers, scatter amplitude, and scatter slope-while correcting for glare and height variations. We use a similarity metric, the overlap coefficient, to quantify contrast between spectra. We also investigate the algorithms for clinical datasets from the colon and breast.

CONCLUSIONS

Preprocessing reduces the overlap due to glare and distance variations. In general, the algorithms standard normal variate, min-max, area under the curve, and single wavelength normalization are the most suitable to preprocess data used to develop a classification algorithm for tissue classification. The type of contrast between tissue types determines which of these four algorithms is most suitable.

Comparison of image registration methods for combining laparoscopic video and spectral image data

Laparoscopic procedures can be assisted by intraoperative modalities, such as quantitative perfusion imaging based on fluorescence or hyperspectral data. If these modalities are not available at video frame rate, fast image registration is needed for the visualization in augmented reality. Three feature-based algorithms and one pre-trained deep homography neural network (DH-NN) were tested for single and multi-homography estimation. Fine-tuning was used to bridge the domain gap of the DH-NN for non-rigid registration of laparoscopic images. The methods were validated on two datasets: an open-source record of 750 manually annotated laparoscopic images, presented in this work, and in-vivo data from a novel laparoscopic hyperspectral imaging system. All feature-based single homography methods outperformed the fine-tuned DH-NN in terms of reprojection error, Structural Similarity Index Measure, and processing time. The feature detector and descriptor ORB1000 enabled video-rate registration of laparoscopic images on standard hardware with submillimeter accuracy.

Model-based characterization platform of fiber optic extended-wavelength diffuse reflectance spectroscopy for identification of neurovascular bundles

SIGNIFICANCE

Fiber-optic extended-wavelength diffuse reflectance spectroscopy (EWDRS) using both visible/near-infrared and shortwave-infrared detectors enables improved detection of spectral absorbances arising from lipids, water, and collagen and has demonstrated promise in a variety of applications, including detection of nerves and neurovascular bundles (NVB). Development of future applications of EWDRS for nerve detection could benefit from the use of model-based analyses including Monte Carlo (MC) simulations and evaluation of agreement between model systems and empirical measurements.

AIM

The aim of this work is to characterize agreement between EWDRS measurements and simulations and inform future applications of model-based studies of nerve-detecting applications.

APPROACH

A model-based platform consisting of an ex vivo microsurgical nerve dissection model, unique two-layer optical phantoms, and MC model simulations of fiber-optic EWDRS spectroscopic measurements were used to characterize EWDRS and compare agreement across models. In addition, MC simulations of an EWDRS measurement scenario are performed to provide a representative example of future analyses.

RESULTS

EWDRS studies performed in the common chicken thigh femoral nerve microsurgical dissection model indicate similar spectral features for classification of NVB versus adjacent tissues as reported in porcine models and human subjects. A comparison of measurements from unique EWDRS issue mimicking optical phantoms and MC simulations indicates high agreement between the two in homogeneous and two-layer optical phantoms, as well as in dissected tissues. Finally, MC simulations of measurement over a simulated NVB indicate the potential of future applications for measurement of nerve plexus.

CONCLUSIONS

Characterization of agreement between fiber-optic EWDRS measurements and MC simulations demonstrates strong agreement across a variety of tissues and optical phantoms, offering promise for further use to guide the continued development of EWDRS for translational applications.

Microscopy is better in color: development of a streamlined spectral light path for real-time multiplex fluorescence microscopy

Spectroscopic image data has provided molecular discrimination for numerous fields including: remote sensing, food safety and biomedical imaging. Despite the various technologies for acquiring spectral data, there remains a trade-off when acquiring data. Typically, spectral imaging either requires long acquisition times to collect an image stack with high spectral specificity or acquisition times are shortened at the expense of fewer spectral bands or reduced spatial sampling. Hence, new spectral imaging microscope platforms are needed to help mitigate these limitations. Fluorescence excitation-scanning spectral imaging is one such new technology, which allows more of the emitted signal to be detected than comparable emission-scanning spectral imaging systems. Here, we have developed a new optical geometry that provides spectral illumination for use in excitation-scanning spectral imaging microscope systems. This was accomplished using a wavelength-specific LED array to acquire spectral image data. Feasibility of the LED-based spectral illuminator was evaluated through simulation and benchtop testing and assessment of imaging performance when integrated with a widefield fluorescence microscope. Ray tracing simulations (TracePro) were used to determine optimal optical component selection and geometry. Spectral imaging feasibility was evaluated using a series of 6-label fluorescent slides. The LED-based system response was compared to a previously tested thin-film tunable filter (TFTF)-based system. Spectral unmixing successfully discriminated all fluorescent components in spectral image data acquired from both the LED and TFTF systems. Therefore, the LED-based spectral illuminator provided spectral image data sets with comparable information content so as to allow identification of each fluorescent component. These results provide proof-of-principle demonstration of the ability to combine output from many discrete wavelength LED sources using a double-mirror (Cassegrain style) optical configuration that can be further modified to allow for high speed, video-rate spectral image acquisition. Real-time spectral fluorescence microscopy would allow monitoring of rapid cell signaling processes (i.e., Ca2+ and other second messenger signaling) and has potential to be translated to clinical imaging platforms.

Review on the Application of Hyperspectral Imaging Technology of the Exposed Cortex in Cerebral Surgery

The study of brain science is vital to human health. The application of hyperspectral imaging in biomedical fields has grown dramatically in recent years due to their unique optical imaging method and multidimensional information acquisition. Hyperspectral imaging technology can acquire two-dimensional spatial information and one-dimensional spectral information of biological samples simultaneously, covering the ultraviolet, visible and infrared spectral ranges with high spectral resolution, which can provide diagnostic information about the physiological, morphological and biochemical components of tissues and organs. This technology also presents finer spectral features for brain imaging studies, and further provides more auxiliary information for cerebral disease research. This paper reviews the recent advance of hyperspectral imaging in cerebral diagnosis. Firstly, the experimental setup, image acquisition and pre-processing, and analysis methods of hyperspectral technology were introduced. Secondly, the latest research progress and applications of hyperspectral imaging in brain tissue metabolism, hemodynamics, and brain cancer diagnosis in recent years were summarized briefly. Finally, the limitations of the application of hyperspectral imaging in cerebral disease diagnosis field were analyzed, and the future development direction was proposed.

In vivo multi spectral colonoscopy in mice

Multi- and hyperspectral endoscopy are possibilities to improve the endoscopic detection of neoplastic lesions in the colon and rectum during colonoscopy. However, most studies in this context are performed on histological samples/biopsies or ex vivo. This leads to the question if previous results can be transferred to an in vivo setting. Therefore, the current study evaluated the usefulness of multispectral endoscopy in identifying neoplastic lesions in the colon. The data set consists of 25 mice with colonic neoplastic lesions and the data analysis is performed by machine learning. Another question addressed was whether adding additional spatial features based on Gauss-Laguerre polynomials leads to an improved detection rate. As a result, detection of neoplastic lesions was achieved with an MCC of 0.47. Therefore, the classification accuracy of multispectral colonoscopy is comparable with hyperspectral colonoscopy in the same spectral range when additional spatial features are used. Moreover, this paper strongly supports the current path towards the application of multi/hyperspectral endoscopy in clinical settings and shows that the challenges from transferring results from ex vivo to in vivo endoscopy can be solved.

Video: Clinical evaluation of a laparoscopic hyperspectral imaging system

Background

Hyperspectral imaging (HSI) during surgical procedures is a new method for perfusion quantification and tissue discrimination. Its use has been limited to open surgery due to large camera sizes, missing color video, or long acquisition times. A hand-held, laparoscopic hyperspectral camera has been developed now to overcome those disadvantages and evaluated clinically for the first time.

Methods

In a clinical evaluation study, gastrointestinal resectates of ten cancer patients were investigated using the laparoscopic hyperspectral camera. Reference data from corresponding anatomical regions were acquired with a clinically approved HSI system. An image registration process was executed that allowed for pixel-wise comparisons of spectral data and parameter images (StO₂: oxygen saturation of tissue, NIR PI: near-infrared perfusion index, OHI: organ hemoglobin index, TWI: tissue water index) provided by both camera systems. The mean absolute error (MAE) and root mean square error (RMSE) served for the quantitative evaluations. Spearman’s rank correlation between factors related to the study design like the time of spectral white balancing and MAE, respectively RMSE, was calculated.

Results

The obtained mean MAEs between the TIVITA® Tissue and the laparoscopic hyperspectral system resulted in StO₂: 11% ± 7%, NIR PI: 14±3, OHI: 14± 5, and TWI: 10 ± 2. The mean RMSE between both systems was 0.1±0.03 from 500 to 750 nm and 0.15 ±0.06 from 750 to 1000 nm. Spearman’s rank correlation coefficients showed no significant correlation between MAE or RMSE and influencing factors related to the study design.

Conclusion

Qualitatively, parameter images of the laparoscopic system corresponded to those of the system for open surgery. Quantitative deviations were attributed to technical differences rather than the study design. Limitations of the presented study are addressed in current large-scale in vivo trials.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00464-022-09282-y.

Intraoperative imaging in pathology-assisted surgery

The pathological assessment of surgical specimens during surgery can reduce the incidence of positive resection margins, which otherwise can result in additional surgeries or aggressive therapeutic regimens. To improve patient outcomes, intraoperative spectroscopic, fluorescence-based, structural, optoacoustic and radiological imaging techniques are being tested on freshly excised tissue. The specific clinical setting and tumour type largely determine whether endogenous or exogenous contrast is to be detected and whether the tumour specificity of the detected biomarker, image resolution, image-acquisition times or penetration depth are to be prioritized. In this Perspective, we describe current clinical standards for intraoperative tissue analysis and discuss how intraoperative imaging is being implemented. We also discuss potential implementations of intraoperative pathology-assisted surgery for clinical decision-making.

Hyperspectral Imaging for Tissue Classification after Advanced Stage Ovarian Cancer Surgery-A Pilot Study

The most important prognostic factor for the survival of advanced-stage epithelial ovarian cancer (EOC) is the completeness of cytoreductive surgery (CRS). Therefore, an intraoperative technique to detect microscopic tumors would be of great value. The aim of this pilot study is to assess the feasibility of near-infrared hyperspectral imaging (HSI) for EOC detection in ex vivo tissue samples. Images were collected during CRS in 11 patients in the wavelength range of 665−975 nm, and processed by calibration, normalization, and noise filtering. A linear support vector machine (SVM) was employed to classify healthy and tumorous tissue (defined as >50% tumor cells). Classifier performance was evaluated using leave-one-out cross-validation. Images of 26 tissue samples from 10 patients were included, containing 26,446 data points that were matched to histopathology. Tumorous tissue could be classified with an area under the curve of 0.83, a sensitivity of 0.81, a specificity of 0.70, and Matthew’s correlation coefficient of 0.41. This study paves the way to in vivo and intraoperative use of HSI during CRS. Hyperspectral imaging can scan a whole tissue surface in a fast and non-contact way. Our pilot study demonstrates that HSI and SVM learning can be used to discriminate EOC from surrounding tissue.

Hyperspectral Imaging for Clinical Applications

Measuring morphological and biochemical features of tissue is crucial for disease diagnosis and surgical guidance, providing clinically significant information related to pathophysiology. Hyperspectral imaging (HSI) techniques obtain both spatial and spectral features of tissue without labeling molecules such as fluorescent dyes, which provides rich information for improved disease diagnosis and treatment. Recent advances in HSI systems have demonstrated its potential for clinical applications, especially in disease diagnosis and image-guided surgery. This review summarizes the basic principle of HSI and optical systems, deep-learning-based image analysis, and clinical applications of HSI to provide insight into this rapidly growing field of research. In addition, the challenges facing the clinical implementation of HSI techniques are discussed.

Comparison of Whiskbroom and Pushbroom darkfield elastic light scattering spectroscopic imaging for head and neck cancer identification in a mouse model

The early detection of head and neck cancer is a prolonged challenging task. It requires a precise and accurate identification of tissue alterations as well as a distinct discrimination of cancerous from healthy tissue areas. A novel approach for this purpose uses microspectroscopic techniques with special focus on hyperspectral imaging (HSI) methods. Our proof-of-principle study presents the implementation and application of darkfield elastic light scattering spectroscopy (DF ELSS) as a non-destructive, high-resolution, and fast imaging modality to distinguish lingual healthy from altered tissue regions in a mouse model. The main aspect of our study deals with the comparison of two varying HSI detection principles, which are a point-by-point and line scanning imaging, and whether one might be more appropriate in differentiating several tissue types. Statistical models are formed by deploying a principal component analysis (PCA) with the Bayesian discriminant analysis (DA) on the elastic light scattering (ELS) spectra. Overall accuracy, sensitivity, and precision values of 98% are achieved for both models whereas the overall specificity results in 99%. An additional classification of model-unknown ELS spectra is performed. The predictions are verified with histopathological evaluations of identical HE-stained tissue areas to prove the model's capability of tissue distinction. In the context of our proof-of-principle study, we assess the Pushbroom PCA-DA model to be more suitable for tissue type differentiations and thus tissue classification. In addition to the HE-examination in head and neck cancer diagnosis, the usage of HSI-based statistical models might be conceivable in a daily clinical routine.

Using physiological parameters measured by hyperspectral imaging to detect colorectal cancer

The accurate detection of malignant tissue during colorectal surgery impacts operation outcome. The non-invasive spectral imaging combined with machine learning (ML) methods showed to be promising for tumor identification. However, large spectral range implies large computing time. To reduce the number of features, ML methods (e.g. logistic regression and convolutional neuronal network CNN) were evaluated based on four physiological tissue parameters to automatically classify cancer and healthy mucosa in resected colon tissue. A ROC AUC of 0.81 was achieved with the CNN. This study shows that the use of only specific wavelengths bands can detect cancer.Clinical Relevance- These outcomes support the possibility to automatically classify colon tumor based on physiological parameters calculated using only specific wavelength bands. Hence, future image-guided colorectal surgeries can be performed with real-time multispectral imaging.

Extended Blind End-member and Abundance Estimation with Spatial Total Variation for Hyperspectral Imaging

Blind linear unmixing (BLU) methods allow the separation of multi and hyperspectral data into end-members and abundance maps in an unsupervised fashion. However, due to incident noise, the abundance maps can exhibit high presence of granularity. To address this problem, in this paper, we present a novel proposal for BLU that considers spatial coherence in the abundance estimations, through a total spatial variation component. The proposed BLU formulation is based on the blind end-member and abundance extraction perspective with total spatial variation (EBEAE-STV). In EBEAE-STV, internal abundances are added to incorporate the spatial coherence in the cost function, which is solved by a coordinates descent algorithm. The results with synthetic data show that the proposed algorithm can significantly decrease the granularity in the estimated abundances, and the estimation errors and computational times are lower compared to state of the art methodologies.Clinical relevance- The proper and robust estimation of end-members and their respective contributions (abundances) in multi-spectral and hyper-spectral images from the proposed EBEAE-STV methodology might provide useful information in several biomedical applications, such as chemometric analysis on different biological samples, tumor identification and brain tissue classification for hyper-spectral imaging, among others.

Automatic Recognition of Colon and Esophagogastric Cancer with Machine Learning and Hyperspectral Imaging

There are approximately 1.8 million diagnoses of colorectal cancer, 1 million diagnoses of stomach cancer, and 0.6 million diagnoses of esophageal cancer each year globally. An automatic computer-assisted diagnostic (CAD) tool to rapidly detect colorectal and esophagogastric cancer tissue in optical images would be hugely valuable to a surgeon during an intervention. Based on a colon dataset with 12 patients and an esophagogastric dataset of 10 patients, several state-of-the-art machine learning methods have been trained to detect cancer tissue using hyperspectral imaging (HSI), including Support Vector Machines (SVM) with radial basis function kernels, Multi-Layer Perceptrons (MLP) and 3D Convolutional Neural Networks (3DCNN). A leave-one-patient-out cross-validation (LOPOCV) with and without combining these sets was performed. The ROC-AUC score of the 3DCNN was slightly higher than the MLP and SVM with a difference of 0.04 AUC. The best performance was achieved with the 3DCNN for colon cancer and esophagogastric cancer detection with a high ROC-AUC of 0.93. The 3DCNN also achieved the best DICE scores of 0.49 and 0.41 on the colon and esophagogastric datasets, respectively. These scores were significantly improved using a patient-specific decision threshold to 0.58 and 0.51, respectively. This indicates that, in practical use, an HSI-based CAD system using an interactive decision threshold is likely to be valuable. Experiments were also performed to measure the benefits of combining the colorectal and esophagogastric datasets (22 patients), and this yielded significantly better results with the MLP and SVM models.

Deep Learning Analysis of In Vivo Hyperspectral Images for Automated Intraoperative Nerve Detection

Nerves are critical structures that may be difficult to recognize during surgery. Inadvertent nerve injuries can have catastrophic consequences for the patient and lead to life-long pain and a reduced quality of life. Hyperspectral imaging (HSI) is a non-invasive technique combining photography with spectroscopy, allowing non-invasive intraoperative biological tissue property quantification. We show, for the first time, that HSI combined with deep learning allows nerves and other tissue types to be automatically recognized in in vivo hyperspectral images. An animal model was used, and eight anesthetized pigs underwent neck midline incisions, exposing several structures (nerve, artery, vein, muscle, fat, skin). State-of-the-art machine learning models were trained to recognize these tissue types in HSI data. The best model was a convolutional neural network (CNN), achieving an overall average sensitivity of 0.91 and a specificity of 1.0, validated with leave-one-patient-out cross-validation. For the nerve, the CNN achieved an average sensitivity of 0.76 and a specificity of 0.99. In conclusion, HSI combined with a CNN model is suitable for in vivo nerve recognition.

[New intraoperative fluorescence-based and spectroscopic imaging techniques in visceral medicine - precision surgery in the "high tech"-operating room]

INTRODUCTION

Fluorescence angiography (FA) with indocyanine green (ICG) and hyperspectral imaging (HSI) are novel intraoperative visualization techniques in abdominal, vascular and transplant surgery. With the purpose of precision surgery, and in order to increase patient's safety, these new tools aim at reducing postoperative morbidity and mortality. This review discusses and highlights recent developments and the future potential of real-time imaging modalities.

METHODS

The underlying mechanisms of the novel imaging methods and their clinical impact are displayed in the context of avoiding anastomotic leaks, the most momentous complications in gastrointestinal surgery after oncologic resections.

RESULTS

While FA is associated with the admission of a fluorescence agent, HSI is contact-free and non-invasive. Both methods are able to record physiological tissue properties in real-time. Additionally, FA also measures dynamic phenomena. The techniques take a few seconds only and do not hamper the operative workflow considerably. With regard to a potential change of the surgical strategy, FA and HSI have an equal significance. Our own advancements reflect, in particular, the topics of data visualization and automated data analyses together with the implementation of artificial intelligence (AI) and minimalization of the current devices to install them into endoscopes, minimal-invasive and robot-guided surgery.

CONCLUSION

There are a limited number of studies in the field of intraoperative imaging techniques. Whether precision surgery in the "high-tech" OR together with FA, HSI and robotics will result in more secure operative procedures to minimize the postoperative morbidity and mortality will have to be evaluated in future multicenter trials.

Hyperspectral autofluorescence characterization of drusen and sub-RPE deposits in age-related macular degeneration

Background:

Soft drusen and basal linear deposit (BLinD) are two forms of the same extracellular lipid rich material that together make up an Oil Spill on Bruch’s membrane (BrM). Drusen are focal and can be recognized clinically. In contrast BLinD is thin and diffusely distributed, and invisible clinically, even on highest resolution OCT, but has been detected on en face hyperspectral autofluorescence (AF) imaging ex vivo. We sought to optimize histologic hyperspectral AF imaging and image analysis for recognition of drusen and sub-RPE deposits (including BLinD and basal laminar deposit), for potential clinical application.

Methods:

Twenty locations specifically with drusen and 12 additional locations specifically from fovea, perifovea and mid-periphery from RPE/BrM flatmounts from 4 AMD donors underwent hyperspectral AF imaging with 4 excitation wavelengths (λ_ex 436, 450, 480 and 505 nm), and the resulting image cubes were simultaneously decomposed with our published non-negative matrix factorization (NMF). Rank 4 recovery of 4 emission spectra was chosen for each excitation wavelength.

Results:

A composite emission spectrum, sensitive and specific for drusen and presumed sub-RPE deposits (the SDr spectrum) was recovered with peak at 510–520 nm in all tissues with drusen, with greatest amplitudes at excitations λ_ex 436, 450 and 480 nm. The RPE spectra of combined sources Lipofuscin (LF)/Melanolipofuscin (MLF) were of comparable amplitude and consistently recapitulated the spectra S1, S2 and S3 previously reported from all tissues: tissues with drusen, foveal and extra-foveal locations.

Conclusions:

A clinical hyperspectral AF camera, with properly chosen excitation wavelengths in the blue range and a hyperspectral AF detector, should be capable of detecting and quantifying drusen and sub-RPE deposits, the earliest known lesions of AMD, before any other currently available imaging modality.

Feedforward Artificial Neural Network-Based Colorectal Cancer Detection Using Hyperspectral Imaging: A Step towards Automatic Optical Biopsy

Simple Summary

Detection of colorectal carcinoma is performed visually by investigators and is confirmed pathologically. With hyperspectral imaging, an expanded spectral range of optical information is now available for analysis. The acquired recordings were analyzed with a neural network, and it was possible to differentiate tumor from healthy mucosa in colorectal carcinoma by automatic classification with high reliability. Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. This is a step towards optical biopsy.

Abstract

Currently, colorectal cancer (CRC) is mainly identified via a visual assessment during colonoscopy, increasingly used artificial intelligence algorithms, or surgery. Subsequently, CRC is confirmed through a histopathological examination by a pathologist. Hyperspectral imaging (HSI), a non-invasive optical imaging technology, has shown promising results in the medical field. In the current study, we combined HSI with several artificial intelligence algorithms to discriminate CRC. Between July 2019 and May 2020, 54 consecutive patients undergoing colorectal resections for CRC were included. The tumor was imaged from the mucosal side with a hyperspectral camera. The image annotations were classified into three groups (cancer, CA; adenomatous margin around the central tumor, AD; and healthy mucosa, HM). Classification and visualization were performed based on a four-layer perceptron neural network. Based on a neural network, the classification of CA or AD resulted in a sensitivity of 86% and a specificity of 95%, by means of leave-one-patient-out cross-validation. Additionally, significant differences in terms of perfusion parameters (e.g., oxygen saturation) related to tumor staging and neoadjuvant therapy were observed. Hyperspectral imaging combined with automatic classification can be used to differentiate between CRC and healthy mucosa. Additionally, the biological changes induced by chemotherapy to the tissue are detectable with HSI.

Computer Vision in the Operating Room: Opportunities and Caveats

Effectiveness of computer vision techniques has been demonstrated through a number of applications, both within and outside healthcare. The operating room environment specifically is a setting with rich data sources compatible with computational approaches and high potential for direct patient benefit. The aim of this review is to summarize major topics in computer vision for surgical domains. The major capabilities of computer vision are described as an aid to surgical teams to improve performance and contribute to enhanced patient safety. Literature was identified through leading experts in the fields of surgery, computational analysis and modeling in medicine, and computer vision in healthcare. The literature supports the application of computer vision principles to surgery. Potential applications within surgery include operating room vigilance, endoscopic vigilance, and individual and team-wide behavioral analysis. To advance the field, we recommend collecting and publishing carefully annotated datasets. Doing so will enable the surgery community to collectively define well-specified common objectives for automated systems, spur academic research, mobilize industry, and provide benchmarks with which we can track progress. Leveraging computer vision approaches through interdisciplinary collaboration and advanced approaches to data acquisition, modeling, interpretation, and integration promises a powerful impact on patient safety, public health, and financial costs.

Evaluation of wavelength ranges and tissue depth probed by diffuse reflectance spectroscopy for colorectal cancer detection

Colorectal cancer (CRC) is the third most common type of cancer worldwide and the second most deadly. Recent research efforts have focused on developing non-invasive techniques for CRC detection. In this study, we evaluated the diagnostic capabilities of diffuse reflectance spectroscopy (DRS) for CRC detection by building 6 classification models based on support vector machines (SVMs). Our dataset consists of 2889 diffuse reflectance spectra collected from freshly excised ex vivo tissues of 47 patients over wavelengths ranging from 350 and 1919 nm with source-detector distances of 630-µm and 2500-µm to probe different depths. Quadratic SVMs were used and performance was evaluated using twofold cross-validation on 10 iterations of randomized training and test sets. We achieved (93.5 ± 2.4)% sensitivity, (94.0 ± 1.7)% specificity AUC by probing the superficial colorectal tissue and (96.1 ± 1.8)% sensitivity, (95.7 ± 0.6)% specificity AUC by sampling deeper tissue layers. To the best of our knowledge, this is the first DRS study to investigate the potential of probing deeper tissue layers using larger SDD probes for CRC detection in the luminal wall. The data analysis showed that using a broader spectrum and longer near-infrared wavelengths can improve the diagnostic accuracy of CRC as well as probing deeper tissue layers.

Excitation-scanning hyperspectral video endoscopy: enhancing the light at the end of the tunnel

Colorectal cancer is the 3^rd leading cancer for incidence and mortality rates. Positive treatment outcomes have been associated with early detection; however, early stage lesions have limited contrast to surrounding mucosa. A potential technology to enhance early stagise detection is hyperspectral imaging (HSI). While HSI technologies have been previously utilized to detect colorectal cancer ex vivo or post-operation, they have been difficult to employ in real-time endoscopy scenarios. Here, we describe an LED-based multifurcated light guide and spectral light source that can provide illumination for spectral imaging at frame rates necessary for video-rate endoscopy. We also present an updated light source optical ray-tracing model that resulted in further optimization and provided a ∼10X light transmission increase compared to the initial prototype. Future work will iterate simulation and benchtop testing of the hyperspectral endoscopic system to achieve the goal of video-rate spectral endoscopy.

Hyperspectral imaging for colon cancer classification in surgical specimens: towards optical biopsy during image-guided surgery

The main curative treatment for localized colon cancer is surgical resection. However when tumor residuals are left positive margins are found during the histological examinations and additional treatment is needed to inhibit recurrence. Hyperspectral imaging (HSI) can offer non-invasive surgical guidance with the potential of optimizing the surgical effectiveness. In this paper we investigate the capability of HSI for automated colon cancer detection in six ex-vivo specimens employing a spectral-spatial patch-based classification approach. The results demonstrate the feasibility in assessing the benign and malignant boundaries of the lesion with a sensitivity of 0.88 and specificity of 0.78. The results are compared with the state-of-the-art deep learning based approaches. The method with a new hybrid CNN outperforms the state-of the-art approaches (0.74 vs. 0.82 AUC). This study paves the way for further investigation towards improving surgical outcomes with HSI.

Hyperspectral characterization of re-epithelialization in an in vitro wound model

In vitro wound models are useful for research on wound re-epithelialization. Hyperspectral imaging represents a non-destructive alternative to histology analysis for detection of re-epithelialization. This study aims to characterize the main optical behavior of a wound model in order to enable development of detection algorithms. K-Means clustering and agglomerative analysis were used to group spatial regions based on the spectral behavior, and an inverse photon transport model was used to explain differences in optical properties. Six samples of the wound model were prepared from human tissue and followed over 22 days. Re-epithelialization occurred at a mean rate of 0.24 mm² /day after day 8 to 10. Suppression of wound spectral features was the main feature characterizing re-epithelialized and intact tissue. Modeling the photon transport through a diffuse layer placed on top of wound tissue properties reproduced the spectral behavior. The missing top layer represented by wounds is thus optically detectable using hyperspectral imaging.

Selecting Image-Guided Surgical Technologies in Oncology: A Surgeon's Perspective

BACKGROUND

To improve surgical performance, image-guided (IG) technologies are increasingly introduced. Yet, it is unknown which oncological procedures yield most value from these technologies. This study aimed to select the most promising IG technology per oncologic indication.

METHODS

An Analytic Hierarchical Process was used to evaluate three IG technologies: navigation, optical imaging, and augmented reality, in five oncologic indications compared with usual care. Sixteen decision criteria were selected. The relative importance of the criteria and the expected performance of the technologies were evaluated among surgeons. The combination of these scores gives the expected value per technology.

RESULTS

On criteria level, sparing critical tissue (9%-18%) and reducing the risk of local recurrence (11%-27%) were most important. Navigation was preferred in three indications-removal of lymph nodes (42%), liver (47%), and rectal tumors (33%). In removing rectal tumors, optical imaging was equally preferred (34%). In removing breast and tongue tumors, no technology was clearly preferred.

CONCLUSIONS

In selecting IG technologies, especially optical and navigation technologies are expected to add value in addition to usual care. Further development of those technologies for the preferred indications seems valuable. Multi-attribute analysis showed to be useful in prioritization of conducting clinical studies and steer research and development initiatives.

Exploiting scale-invariance: a top layer targeted inverse model for hyperspectral images of wounds

Detection of re-epithelialization in wound healing is important, but challenging. Hyperspectral imaging can be used for non-destructive characterization, but efficient techniques are needed to extract and interpret the information. An inverse photon transport model suitable for characterization of re-epithelialization is validated and explored in this study. It exploits scale-invariance to enable fitting of the epidermal skin layer only. Monte Carlo simulations indicate that the fitted layer transmittance and reflectance spectra are unique, and that there exists an infinite number of coupled parameter solutions. The method is used to explain the optical behavior of and detect re-epithelialization in an in vitro wound model.

Multispectral Imaging for Automated Tissue Identification of Normal Human Surgical Specimens

OBJECTIVE

Safe surgery requires the accurate discrimination of tissue intraoperatively. We assess the feasibility of using multispectral imaging and deep learning to enhance surgical vision by automated identification of normal human head and neck tissues.

STUDY DESIGN

Construction and feasibility testing of novel multispectral imaging system for surgery.

SETTING

Academic university hospital.

SUBJECTS AND METHODS

Multispectral images of fresh-preserved human cadaveric tissues were captured with our adapted digital operating microscope. Eleven tissue types were sampled, each sequentially exposed to 6 lighting conditions. Two convolutional neural network machine learning models were developed to classify tissues based on multispectral and white-light color images (ARRInet-M and ARRInet-W, respectively). Blinded otolaryngology residents were asked to identify tissue specimens from white-light color images, and their performance was compared with that of the ARRInet models.

RESULTS

A novel multispectral imaging system was developed with minimal adaptation to an existing digital operating microscope. With 81.8% accuracy in tissue identification of full-size images, the multispectral ARRInet-M classifier outperformed the white-light-only ARRInet-W model (45.5%) and surgical residents (69.7%). Challenges with discrimination occurred with parotid vs fat and blood vessels vs nerve.

CONCLUSIONS

A deep learning model using multispectral imaging outperformed a similar model and surgical residents using traditional white-light imaging at the task of classifying normal human head and neck tissue ex vivo. These results suggest that multispectral imaging can enhance surgical vision and augment surgeons' ability to identify tissues during a procedure.

Laparoscopic system for simultaneous high-resolution video and rapid hyperspectral imaging in the visible and near-infrared spectral range

SIGNIFICANCE

Hyperspectral imaging (HSI) can support intraoperative perfusion assessment, the identification of tissue structures, and the detection of cancerous lesions. The practical use of HSI for minimal-invasive surgery is currently limited, for example, due to long acquisition times, missing video, or large set-ups.

AIM

An HSI laparoscope is described and evaluated to address the requirements for clinical use and high-resolution spectral imaging.

APPROACH

Reflectance measurements with reference objects and resected human tissue from 500 to 1000 nm are performed to show the consistency with an approved medical HSI device for open surgery. Varying object distances are investigated, and the signal-to-noise ratio (SNR) is determined for different light sources.

RESULTS

The handheld design enables real-time processing and visualization of HSI data during acquisition within 4.6 s. A color video is provided simultaneously and can be augmented with spectral information from push-broom imaging. The reflectance data from the HSI system for open surgery at 50 cm and the HSI laparoscope are consistent for object distances up to 10 cm. A standard rigid laparoscope in combination with a customized LED light source resulted in a mean SNR of 30 to 43 dB (500 to 950 nm).

CONCLUSIONS

Compact and rapid HSI with a high spatial- and spectral-resolution is feasible in clinical practice. Our work may support future studies on minimally invasive HSI to reduce intra- and postoperative complications.