Hyperspectral and multispectral imaging in digital and computational pathology: a systematic review [Invited]

Enhancing surgical precision in squamous cell carcinoma of the head and neck: Hyperspectral imaging and artificial intelligence for improved margin assessment in an ex vivo setting

BACKGROUND

Head and neck cancers, constituting 3-5% of all cancer cases, often require surgical resection for optimal outcomes. Achieving complete resection (R0) is crucial, but current methods, relying on white light endoscopy and microscopy, have limitations. Hyperspectral imaging (HSI) offers potential benefits by capturing detailed spectral information beyond human vision.

MATERIAL AND METHODS

This study enrolled 32 patients with head and neck squamous cell carcinoma (HNSCC). Following surgical resection specimens underwent ex vivo HSI imaging. Annotated regions were utilized to train a Convolutional Neural Network (CNN) and Graph Neural Network (GNN). Imaging parameters were carefully optimized for efficiency.

RESULTS

Our HSI imaging setup required around 12 min per measurement and demonstrated feasibility with promising accuracy. The combination of HSI and artificial intelligence (AI) achieved an 86% accuracy in predicting tumor tissue. Challenges included data volume and extended capture times.

CONCLUSION

Hyperspectral imaging, complemented by AI, shows promise in enhancing tissue differentiation for HNSCC. The study envisions real-time integration of HSI into surgery for margin assessment. Challenges such as data volume and capture times warrant further exploration, emphasizing the need for ongoing investigations to refine clinical applications.

Stain Normalization of Histopathological Images Based on Deep Learning: A Review

Histopathological images stained with hematoxylin and eosin (H&E) are crucial for cancer diagnosis and prognosis. However, color variations caused by differences in tissue preparation and scanning devices can lead to data distribution discrepancies, adversely affecting the performance of downstream algorithms in tasks like classification, segmentation, and detection. To address these issues, stain normalization methods have been developed to standardize color distributions across images from various sources. Recent advancements in deep learning-based stain normalization methods have shown significant promise due to their minimal preprocessing requirements, independence from reference templates, and robustness. This review examines 115 publications to explore the latest developments in this field. We first outline the evaluation metrics and publicly available datasets used for assessing stain normalization methods. Next, we systematically review deep learning-based approaches, including supervised, unsupervised, and self-supervised methods, categorizing them by core technologies and analyzing their contributions and limitations. Finally, we discuss current challenges and future directions, aiming to provide researchers with a comprehensive understanding of the field, promote further development, and accelerate the progress of intelligent cancer diagnosis.

Deep learning-based hyperspectral technique identifies metastatic lymph nodes in oral squamous cell carcinoma-A pilot study

AIMS

To establish a system based on hyperspectral imaging and deep learning for the detection of cancer cells in metastatic lymph nodes.

MAIN METHODS

The continuous sections of metastatic lymph nodes from 45 oral squamous cell carcinoma (OSCC) patients were collected. An improved ResUNet algorithm was established for deep learning to analyze the spectral curve differences between cancer cells and lymphocytes, and that between tumor tissue and normal tissue.

KEY FINDINGS

It was found that cancer cells, lymphocytes, and erythrocytes in the metastatic lymph nodes could be distinguished basing hyperspectral image, with overall accuracy (OA) as 87.30% and average accuracy (AA) as 85.46%. Cancerous area could be recognized by hyperspectral image and deep learning, and the average intersection over union (IOU) and accuracy were 0.6253 and 0.7692, respectively.

SIGNIFICANCE

This study indicated that deep learning-based hyperspectral techniques can identify tumor tissue in OSCC metastatic lymph nodes, achieving high accuracy of pathological diagnosis, high work efficiency, and reducing work burden. But these are preliminary results limited to a small sample.

Hyperspectral analysis to assess gametocytogenesis stage progression in malaria-infected human erythrocytes

Significance

Developments of anti-gametocyte drugs have been delayed due to insufficient understanding of gametocyte biology. We report a systematic workflow of data processing algorithms to quantify changes in the absorption spectrum and cell morphology of single malaria-infected erythrocytes. These changes may serve as biomarkers instrumental for the future development of antimalarial strategies, especially for anti-gametocyte drug design and testing. Image-based biomarkers may also be useful for nondestructive, label-free malaria detection and drug efficacy evaluation in resource-limited communities.

Aim

We extend the application of hyperspectral microscopy to provide detailed insights into gametocyte stage progression through the quantitative analysis of absorbance spectra and cell morphology in malaria-infected erythrocytes.

Approach

Malaria-infected erythrocytes at asexual and different gametocytogenesis stages were imaged through hyperspectral confocal microscopy. The preprocessing of the hyperspectral data cubes to transform them to color images and spectral angle mapper (SAM) analysis were first used to segment hemoglobin (Hb)- and hemozoin (Hz)-abundant areas within the host erythrocytes. Correlations between changes in cell morphology and increasing Hz-abundant areas of the infected erythrocytes were then examined to test their potential as optical biomarkers to determine the progression of infection, involving transitions from asexual to various gametocytogenesis stages.

Results

Following successful segmentation of Hb- and Hz-abundant areas in malaria-infected erythrocytes through SAM analysis, a modest correlation between the segmented Hz-abundant area and cell shape changes over time was observed. A significant increase in both the areal fraction of Hz and the ellipticity of the cell confirms that the Hz fraction change correlates with the progression of gametocytogenesis.

Conclusions

Our workflow enables the quantification of changes in host cell morphology and the relative contents of Hb and Hz at various parasite growth stages. The quantified results exhibit a trend that both the segmented areal fraction of intracellular Hz and the ellipticity of the host cell increase as gametocytogenesis progresses, suggesting that these two metrics may serve as useful biomarkers to determine the stage of gametocytogenesis.

Sine oculis homeobox homolog family function in gastrointestinal cancer: Progression and comprehensive analysis

The sine oculis homeobox homolog (SIX) family, a group of transcription factors characterized by a conserved DNA-binding homology domain, plays a critical role in orchestrating embryonic development and organogenesis across various organisms, including humans. Comprising six distinct members, from SIX1 to SIX6, each member contributes uniquely to the development and differentiation of diverse tissues and organs, underscoring the versatility of the SIX family. Dysregulation or mutations in SIX genes have been implicated in a spectrum of developmental disorders, as well as in tumor initiation and progression, highlighting their pivotal role in maintaining normal developmental trajectories and cellular functions. Efforts to target the transcriptional complex of the SIX gene family have emerged as a promising strategy to inhibit tumor development. While the development of inhibitors targeting this gene family is still in its early stages, the significant potential of such interventions holds promise for future therapeutic advances. Therefore, this review aimed to comprehensively explore the advancements in understanding the SIX family within gastrointestinal cancers, focusing on its critical role in normal organ development and its implications in gastrointestinal cancers, including gastric, pancreatic, colorectal cancer, and hepatocellular carcinomas. In conclusion, this review deepened the understanding of the functional roles of the SIX family and explored the potential of utilizing this gene family for the diagnosis, prognosis, and treatment of gastrointestinal cancers.

Advancing hyperspectral imaging and machine learning tools toward clinical adoption in tissue diagnostics: A comprehensive review

Hyperspectral imaging (HSI) has become an evident transformative apparatus in medical diagnostics. The review aims to appraise the present advancement and challenges in HSI for medical applications. It features a variety of medical applications namely diagnosing diabetic retinopathy, neurodegenerative diseases like Parkinson's and Alzheimer's, which illustrates its effectiveness in early diagnosis, early caries detection in periodontal disease, and dermatology by detecting skin cancer. Regardless of these advances, the challenges exist within every aspect that limits its broader clinical adoption. It has various constraints including difficulties with technology related to the complexity of the HSI system and needing specialist training, which may act as a drawback to its clinical settings. This article pertains to potential challenges expressed in medical applications and probable solutions to overcome these constraints. Successful companies that perform advanced solutions with HSI in terms of medical applications are being emphasized in this study to signal the high level of interest in medical diagnosis for systems to incorporate machine learning ML and artificial intelligence AI to foster precision diagnosis and standardized clinical workflow. This advancement signifies progressive possibilities of HSI in real-time clinical assessments. In conclusion despite HSI has been presented as a significant advanced medical imaging tool, addressing its limitations and probable solutions is for broader clinical adoption.

Advancing hyperspectral imaging and machine learning tools toward clinical adoption in tissue diagnostics: A comprehensive review

Hyperspectral imaging (HSI) has become an evident transformative apparatus in medical diagnostics. The review aims to appraise the present advancement and challenges in HSI for medical applications. It features a variety of medical applications namely diagnosing diabetic retinopathy, neurodegenerative diseases like Parkinson's and Alzheimer's, which illustrates its effectiveness in early diagnosis, early caries detection in periodontal disease, and dermatology by detecting skin cancer. Regardless of these advances, the challenges exist within every aspect that limits its broader clinical adoption. It has various constraints including difficulties with technology related to the complexity of the HSI system and needing specialist training, which may act as a drawback to its clinical settings. This article pertains to potential challenges expressed in medical applications and probable solutions to overcome these constraints. Successful companies that perform advanced solutions with HSI in terms of medical applications are being emphasized in this study to signal the high level of interest in medical diagnosis for systems to incorporate machine learning ML and artificial intelligence AI to foster precision diagnosis and standardized clinical workflow. This advancement signifies progressive possibilities of HSI in real-time clinical assessments. In conclusion despite HSI has been presented as a significant advanced medical imaging tool, addressing its limitations and probable solutions is for broader clinical adoption.

Engineered feature embeddings meet deep learning: A novel strategy to improve bone marrow cell classification and model transparency

Cytomorphology evaluation of bone marrow cell is the initial step to diagnose different hematological diseases. This assessment is still manually performed by trained specialists, who may be a bottleneck within the clinical process. Deep learning algorithms are a promising approach to automate this bone marrow cell evaluation. These artificial intelligence models have focused on limited cell subtypes, mainly associated to a particular disease, and are frequently presented as black boxes. The herein introduced strategy presents an engineered feature representation, the region-attention embedding, which improves the deep learning classification performance of a cytomorphology with 21 bone marrow cell subtypes. This embedding is built upon a specific organization of cytology features within a squared matrix by distributing them after pre-segmented cell regions, i.e., cytoplasm, nucleus, and whole-cell. This novel cell image representation, aimed to preserve spatial/regional relations, is used as input of the network. Combination of region-attention embedding and deep learning networks (Xception and ResNet50) provides local relevance associated to image regions, adding up interpretable information to the prediction. Additionally, this approach is evaluated in a public database with the largest number of cell subtypes (21) by a thorough evaluation scheme with three iterations of a 3-fold cross-validation, performed in 80% of the images (n = 89,484), and a testing process in an unseen set of images composed by the remaining 20% of the images (n = 22,371). This evaluation process demonstrates the introduced strategy outperforms previously published approaches in an equivalent validation set, with a f1-score of 0.82, and presented competitive results on the unseen data partition with a f1-score of 0.56.

Towards reliable hyperspectral imaging biomarkers of CT26 murine tumor model

The non-invasive monitoring of tumor growth can offer invaluable diagnostic insights and enhance our understanding of tumors and their microenvironment. Integrating hyperspectral imaging (HSI) with three-dimensional optical profilometry (3D OP) makes contactless and non-invasive tumor diagnosis possible by utilizing the inherent tissue contrast provided by visible (VIS) and near-infrared (NIR) light. Consequently, valuable information regarding tumors and healthy tissues can be extracted from the acquired hyperspectral images. Until now, very few methods have been used to monitor tumor models in vivo daily and non-invasively. In this research, we conducted a 14-day study monitoring BALB/c mice with subcutaneously grown CT26 murine colon carcinomas in vivo, commencing on the day of tumor cell injection. We extracted physiological properties such as total hemoglobin (THB) and tissue oxygenation (StO 2 ) using the inverse adding-doubling (IAD) algorithm and manually segmented the tissues. We then selected the ten most relevant features describing tumors using the Max-Relevance Min-Redundancy (MRMR) algorithm and utilized 30 classic and advanced machine learning (ML) algorithms to discriminate tumors from healthy tissues. Finally, we tested the robustness of feature selection and model performance by smoothing tissue parameter maps extracted by IAD with a variable kernel and omitting selected training data. We could discriminate CT26 tumor models from surrounding healthy tissues with an area under the curve (AUC) of up to 1 for models based on the gradient boosting method, linear discriminant analysis, and random forests. Our findings help pave the way for precise and robust imaging biomarkers that could aid tumor diagnosis and advance clinical practice.

R2D2-GAN: Robust Dual Discriminator Generative Adversarial Network for Microscopy Hyperspectral Image Super-Resolution

High-resolution microscopy hyperspectral (HS) images can provide highly detailed spatial and spectral information, enabling the identification and analysis of biological tissues at a microscale level. Recently, significant efforts have been devoted to enhancing the resolution of HS images by leveraging high spatial resolution multispectral (MS) images. However, the inherent hardware constraints lead to a significant distribution gap between HS and MS images, posing challenges for image super-resolution within biomedical domains. This discrepancy may arise from various factors, including variations in camera imaging principles (e.g., snapshot and push-broom imaging), shooting positions, and the presence of noise interference. To address these challenges, we introduced a unique unsupervised super-resolution framework named R2D2-GAN. This framework utilizes a generative adversarial network (GAN) to efficiently merge the two data modalities and improve the resolution of microscopy HS images. Traditionally, supervised approaches have relied on intuitive and sensitive loss functions, such as mean squared error (MSE). Our method, trained in a real-world unsupervised setting, benefits from exploiting consistent information across the two modalities. It employs a game-theoretic strategy and dynamic adversarial loss, rather than relying solely on fixed training strategies for reconstruction loss. Furthermore, we have augmented our proposed model with a central consistency regularization (CCR) module, aiming to further enhance the robustness of the R2D2-GAN. Our experimental results show that the proposed method is accurate and robust for super-resolution images. We specifically tested our proposed method on both a real and a synthetic dataset, obtaining promising results in comparison to other state-of-the-art methods.

Colorectal cancer histopathology image analysis: A comparative study of prognostic values of automatically extracted morphometric nuclear features in multispectral and red-blue-green imagery

OBJECTIVES

Multispectral imaging (MSI) has been utilized to predict the prognosis of colorectal cancer (CRC) patients, however, our understanding of the prognostic value of nuclear morphological parameters of bright-field MSI in CRC is still limited. This study was designed to compare the efficiency of MSI and standard red-green-blue (RGB) images in predicting the prognosis of CRC.

METHODS

We compared the efficiency of MS and conventional RGB images on the quantitative assessment of hematoxylin-eosin (HE) stained histopathology images. A pipeline was developed using a pixel-wise support vector machine (SVM) classifier for gland-stroma segmentation, and a marker-controlled watershed algorithm was used for nuclei segmentation. The correlation between extracted morphological parameters and the five-year disease-free survival (5-DFS) was analyzed.

RESULTS

Forty-seven nuclear morphological parameters were extracted in total. Based on Kaplan-Meier analysis, eight features derived from MS images and seven featured derived from RGB images were significantly associated with 5-DFS, respectively. Compared with RGB images, MSI showed higher accuracy, precision, and Dice index in nuclei segmentation. Multivariate analysis indicated that both integrated parameters 1 (factors negatively correlated with CRC prognosis including nuclear number, circularity, eccentricity, major axis length) and 2 (factors positively correlated with CRC prognosis including nuclear average area, area perimeter, total area/total perimeter ratio, average area/perimeter ratio) in MS images were independent prognostic factors of 5-DFS, in contrast with only integrated parameter 1 (P<0.001) in RGB images. More importantly, the quantification of HE-stained MS images displayed higher accuracy in predicting 5-DFS compared with RGB images (76.9% vs 70.9%).

CONCLUSIONS

Quantitative evaluation of HE-stained MS images could yield more information and better predictive performance for CRC prognosis than conventional RGB images, thereby contributing to precision oncology.

Label-Free Assessment of Key Biological Autofluorophores: Material Characteristics and Opportunities for Clinical Applications

Autofluorophores are endogenous fluorescent compounds that naturally occur in the intra and extracellular spaces of all tissues and organs. Most have vital biological functions - like the metabolic cofactors NAD(P)H and FAD+, as well as the structural protein collagen. Others are considered to be waste products - like lipofuscin and advanced glycation end products - which accumulate with age and are associated with cellular dysfunction. Due to their natural fluorescence, these materials have great utility for enabling non-invasive, label-free assays with direct ties to biological function. Numerous technologies, with different advantages and drawbacks, are applied to their assessment, including fluorescence lifetime imaging microscopy, hyperspectral microscopy, and flow cytometry. Here, the applications of label-free autofluorophore assessment are reviewed for clinical and health-research applications, with specific attention to biomaterials, disease detection, surgical guidance, treatment monitoring, and tissue assessment - fields that greatly benefit from non-invasive methodologies capable of continuous, in vivo characterization.

Hyperspectral imaging of 4T1 mammary carcinomas grown in dorsal skinfold window chambers: spectral renormalization and fluorescence modeling

Significance

Hyperspectral imaging (HSI) of murine tumor models grown in dorsal skinfold window chambers (DSWCs) offers invaluable insight into the tumor microenvironment. However, light loss in a glass coverslip is often overlooked, and particular tissue characteristics are improperly modeled, leading to errors in tissue properties extracted from hyperspectral images.

Aim

We highlight the significance of spectral renormalization in HSI of DSWC models and demonstrate the benefit of incorporating enhanced green fluorescent protein (EGFP) excitation and emission in the skin tissue model for tumors expressing genes to produce EGFP.

Approach

We employed an HSI system for intravital imaging of mice with 4T1 mammary carcinoma in a DSWC over 14 days. We performed spectral renormalization of hyperspectral images based on the measured reflectance spectra of glass coverslips and utilized an inverse adding-doubling (IAD) algorithm with a two-layer murine skin model, to extract tissue parameters, such as total hemoglobin concentration and tissue oxygenation (StO 2 ). The model was upgraded to consider EGFP fluorescence excitation and emission. Moreover, we conducted additional experiments involving tissue phantoms, human forearm skin imaging, and numerical simulations.

Results

Hyperspectral image renormalization and the addition of EGFP fluorescence in the murine skin model reduced the mean absolute percentage errors (MAPEs) of fitted and measured spectra by up to 10% in tissue phantoms, 0.55% to 1.5% in the human forearm experiment and numerical simulations, and up to 0.7% in 4T1 tumors. Similarly, the MAPEs for tissue parameters extracted by IAD were reduced by up to 3% in human forearms and numerical simulations. For some parameters, statistically significant differences ( p < 0.05 ) were observed in 4T1 tumors. Ultimately, we have shown that fluorescence emission could be helpful for 4T1 tumor segmentation.

Conclusions

The results contribute to improving intravital monitoring of DWSC models using HSI and pave the way for more accurate and precise quantitative imaging.

Detection and margin assessment of thyroid carcinoma with microscopic hyperspectral imaging using transformer networks

Significance

Hyperspectral imaging (HSI) is an emerging imaging modality for oncological applications and can improve cancer detection with digital pathology.

Aim

The study aims to highlight the increased accuracy and sensitivity of detecting the margin of thyroid carcinoma in hematoxylin and eosin (H&E)-stained histological slides using HSI and data augmentation methods.

Approach

Using an automated microscopic imaging system, we captured 2599 hyperspectral images from 65 H&E-stained human thyroid slides. Images were then preprocessed into 153,906 image patches of dimension 250 × 250 × 84   pixels . We modified the TimeSformer network architecture, which used alternating spectral attention and spatial attention layers. We implemented several data augmentation methods for HSI based on the RandAugment algorithm. We compared the performances of TimeSformer on HSI against the performances of pretrained ConvNext and pretrained vision transformers (ViT) networks on red, green, and blue (RGB) images. Finally, we applied attention unrolling techniques on the trained TimeSformer network to identify the biological features to which the network paid attention.

Results

In the testing dataset, TimeSformer achieved an accuracy of 90.87%, a weightedF 1 score of 89.79%, a sensitivity of 91.50%, and an area under the receiving operator characteristic curve (AU-ROC) score of 97.04%. Additionally, TimeSformer produced thyroid carcinoma tumor margins with an average Jaccard score of 0.76 mm. Without data augmentation, TimeSformer achieved an accuracy of 88.23%, a weightedF 1 score of 86.46%, a sensitivity of 85.53%, and an AU-ROC score of 94.94%. In comparison, the ViT network achieved an 89.98% accuracy, an 88.14% weightedF 1 score, an 84.77% sensitivity, and a 96.17% AU-ROC. Our visualization results showed that the network paid attention to biological features.

Conclusions

The TimeSformer model trained with hyperspectral histological data consistently outperformed conventional RGB-based models, highlighting the superiority of HSI in this context. Our proposed augmentation methods improved the accuracy, theF 1 score, and the sensitivity score.

An Analytical Study on the Utility of RGB and Multispectral Imagery with Band Selection for Automated Tumor Grading

The implementation of tumor grading tasks with image processing and machine learning techniques has progressed immensely over the past several years. Multispectral imaging enabled us to capture the sample as a set of image bands corresponding to different wavelengths in the visible and infrared spectrums. The higher dimensional image data can be well exploited to deliver a range of discriminative features to support the tumor grading application. This paper compares the classification accuracy of RGB and multispectral images, using a case study on colorectal tumor grading with the QU-Al Ahli Dataset (dataset I). Rotation-invariant local phase quantization (LPQ) features with an SVM classifier resulted in 80% accuracy for the RGB images compared to 86% accuracy with the multispectral images in dataset I. However, the higher dimensionality elevates the processing time. We propose a band-selection strategy using mutual information between image bands. This process eliminates redundant bands and increases classification accuracy. The results show that our band-selection method provides better results than normal RGB and multispectral methods. The band-selection algorithm was also tested on another colorectal tumor dataset, the Texas University Dataset (dataset II), to further validate the results. The proposed method demonstrates an accuracy of more than 94% with 10 bands, compared to using the whole set of 16 multispectral bands. Our research emphasizes the advantages of multispectral imaging over the RGB imaging approach and proposes a band-selection method to address the higher computational demands of multispectral imaging.

Colorimetric Analyses of the Optic Nerve Head and Retina Indicate Increased Blood Flow After Vitrectomy

Purpose

The purpose of this study was to evaluate the impact of vitrectomy and posterior hyaloid (PH) peeling on color alteration of optic nerve head (ONH) and retina as a surrogate biomarker of induced perfusion changes.

Methods

Masked morphometric and colorimetric analyses were conducted on preoperative (<1 month) and postoperative (<18 months) color fundus photographs of 54 patients undergoing vitrectomy, either with (44) or without (10) PH peeling and 31 years of age and gender-matched control eyes. Images were calibrated according to the hue and saturation values of the parapapillary venous blood column. Chromatic spectra of the retinal pigment epithelium and choroid were subtracted to avoid color aberrations. Red, green, and blue (RGB) bit values over the ONH and retina were plotted within the constructed RGB color space to analyze vitrectomy-induced color shift. Vitrectomy-induced parapapillary vein caliber changes were also computed morphometrically.

Results

A significant post-vitrectomy red hue shift was noted on the ONH (37.1 degrees ± 10.9 degrees vs. 4.1 degrees ± 17.7 degrees, P < 0.001), which indicates a 2.8-fold increase in blood perfusion compared to control (2.6 ± 1.9 vs. 0.9 ± 1.8, P < 0.001). A significant post-vitrectomy increase in the retinal vein diameter was also noticed (6.8 ± 6.4% vs. 0.1 ± 0.3%, P < 0.001), which was more pronounced with PH peeling (7.9 ± 6.6% vs. 3.1 ± 4.2%, P = 0.002).

Conclusions

Vitrectomy and PH peeling increase ONH and retinal blood flow. Colorimetric and morphometric analyses offer valuable insights for future artificial intelligence and deep learning applications in this field.

Translational Relevance

The methodology described herein can easily be applied in different clinical settings and may enlighten the beneficial effects of vitrectomy in several retinal vascular diseases.

Histological Hyperspectral Glioblastoma Dataset (HistologyHSI-GB)

Hyperspectral (HS) imaging (HSI) technology combines the main features of two existing technologies: imaging and spectroscopy. This allows to analyse simultaneously the morphological and chemical attributes of the objects captured by a HS camera. In recent years, the use of HSI provides valuable insights into the interaction between light and biological tissues, and makes it possible to detect patterns, cells, or biomarkers, thus, being able to identify diseases. This work presents the HistologyHSI-GB dataset, which contains 469 HS images from 13 patients diagnosed with brain tumours, specifically glioblastoma. The slides were stained with haematoxylin and eosin (H&E) and captured using a microscope at 20× power magnification. Skilled histopathologists diagnosed the slides and provided image-level annotations. The dataset was acquired using custom HSI instrumentation, consisting of a microscope equipped with an HS camera covering the spectral range from 400 to 1000 nm.

Investigation of the usefulness of a bile duct biopsy and bile cytology using a hyperspectral camera and machine learning

To improve the efficiency of pathological diagnoses, the development of automatic pathological diagnostic systems using artificial intelligence (AI) is progressing; however, problems include the low interpretability of AI technology and the need for large amounts of data. We herein report the usefulness of a general-purpose method that combines a hyperspectral camera with machine learning. As a result of analyzing bile duct biopsy and bile cytology specimens, which are especially difficult to determine as benign or malignant, using multiple machine learning models, both were able to identify benign or malignant cells with an accuracy rate of more than 80% (93.3% for bile duct biopsy specimens and 83.2% for bile cytology specimens). This method has the potential to contribute to the diagnosis and treatment of bile duct cancer and is expected to be widely applied and utilized in general pathological diagnoses.

SAHIS-Net: a spectral attention and feature enhancement network for microscopic hyperspectral cholangiocarcinoma image segmentation

Cholangiocarcinoma (CCA) poses a significant clinical challenge due to its aggressive nature and poor prognosis. While traditional diagnosis relies on color-based histopathology, hyperspectral imaging (HSI) offers rich, high-dimensional data holding potential for more accurate diagnosis. However, extracting meaningful insights from this data remains challenging. This work investigates the application of deep learning for CCA segmentation in microscopic HSI images, and introduces two novel neural networks: (1) Histogram Matching U-Net (HM-UNet) for efficient image pre-processing, and (2) Spectral Attention based Hyperspectral Image Segmentation Net (SAHIS-Net) for CCA segmentation. SAHIS-Net integrates a novel Spectral Attention (SA) module for adaptively weighing spectral information, an improved attention-aware feature enhancement (AFE) mechanism for better providing the model with more discriminative features, and a multi-loss training strategy for effective early stage feature extraction. We compare SAHIS-Net against several general and CCA-specific models, demonstrating its superior performance in segmenting CCA regions. These results highlight the potential of our approach for segmenting medical HSI images.

Current Landscape of Advanced Imaging Tools for Pathology Diagnostics

Histopathology relies on century-old workflows of formalin fixation, paraffin embedding, sectioning, and staining tissue specimens on glass slides. Despite being robust, this conventional process is slow, labor-intensive, and limited to providing two-dimensional views. Emerging technologies promise to enhance and accelerate histopathology. Slide-free microscopy allows rapid imaging of fresh, unsectioned specimens, overcoming slide preparation delays. Methods such as fluorescence confocal microscopy, multiphoton microscopy, along with more recent innovations including microscopy with UV surface excitation and fluorescence-imitating brightfield imaging can generate images resembling conventional histology directly from the surface of tissue specimens. Slide-free microscopy enable applications such as rapid intraoperative margin assessment and, with appropriate technology, three-dimensional histopathology. Multiomics profiling techniques, including imaging mass spectrometry and Raman spectroscopy, provide highly multiplexed molecular maps of tissues, although clinical translation remains challenging. Artificial intelligence is aiding the adoption of new imaging modalities via virtual staining, which converts methods such as slide-free microscopy into synthetic brightfield-like or even molecularly informed images. Although not yet commonplace, these emerging technologies collectively demonstrate the potential to modernize histopathology. Artificial intelligence-assisted workflows will ease the transition to new imaging modalities. With further validation, these advances may transform the century-old conventional histopathology pipeline to better serve 21st-century medicine. This review provides an overview of these enabling technology platforms and discusses their potential impact.

Hyperspectral imaging and evaluation of surgical margins: where do we stand?

PURPOSE OF REVIEW

To highlight the recent literature on the use of hyperspectral imaging (HSI) for cancer margin evaluation ex vivo, for head and neck cancer pathology and in vivo during head and neck cancer surgery.

RECENT FINDINGS

HSI can be used ex vivo on unstained and stained tissue sections to analyze head and neck tissue and tumor cells in combination with machine learning approaches to analyze head and neck cancer cell characteristics and to discriminate the tumor border from normal tissue. Data on in vivo applications during head and neck cancer surgery are preliminary and limited. Even now an accuracy of 80% for tumor versus nonneoplastic tissue classification can be achieved for certain tasks, within the current in vivo settings.

SUMMARY

Significant progress has been made to introduce HSI for ex vivo head and neck cancer pathology evaluation and for an intraoperative use to define the tumor margins. To optimize the accuracy for in vivo use, larger HSI databases with annotations for head and neck cancer are needed.

Mapping cutaneous field carcinogenesis of nonmelanoma skin cancer using mesoscopic imaging of pro-inflammation cues

Nonmelanoma skin cancers remain the most widely diagnosed types of cancers globally. Thus, for optimal patient management, it has become imperative that we focus our efforts on the detection and monitoring of cutaneous field carcinogenesis. The concept of field cancerization (or field carcinogenesis), introduced by Slaughter in 1953 in the context of oral cancer, suggests that invasive cancer may emerge from a molecularly and genetically altered field affecting a substantial area of underlying tissue including the skin. A carcinogenic field alteration, present in precancerous tissue over a relatively large area, is not easily detected by routine visualization. Conventional dermoscopy and microscopy imaging are often limited in assessing the entire carcinogenic landscape. Recent efforts have suggested the use of noninvasive mesoscopic (between microscopic and macroscopic) optical imaging methods that can detect chronic inflammatory features to identify pre-cancerous and cancerous angiogenic changes in tissue microenvironments. This concise review covers major types of mesoscopic optical imaging modalities capable of assessing pro-inflammatory cues by quantifying blood haemoglobin parameters and hemodynamics. Importantly, these imaging modalities demonstrate the ability to detect angiogenesis and inflammation associated with actinically damaged skin. Representative experimental preclinical and human clinical studies using these imaging methods provide biological and clinical relevance to cutaneous field carcinogenesis in altered tissue microenvironments in the apparently normal epidermis and dermis. Overall, mesoscopic optical imaging modalities assessing chronic inflammatory hyperemia can enhance the understanding of cutaneous field carcinogenesis, offer a window of intervention and monitoring for actinic keratoses and nonmelanoma skin cancers and maximise currently available treatment options.

Review of the application of the most current sophisticated image processing methods for the skin cancer diagnostics purposes

This paper presents the most current and innovative solutions applying modern digital image processing methods for the purpose of skin cancer diagnostics. Skin cancer is one of the most common types of cancers. It is said that in the USA only, one in five people will develop skin cancer and this trend is constantly increasing. Implementation of new, non-invasive methods plays a crucial role in both identification and prevention of skin cancer occurrence. Early diagnosis and treatment are needed in order to decrease the number of deaths due to this disease. This paper also contains some information regarding the most common skin cancer types, mortality and epidemiological data for Poland, Europe, Canada and the USA. It also covers the most efficient and modern image recognition methods based on the artificial intelligence applied currently for diagnostics purposes. In this work, both professional, sophisticated as well as inexpensive solutions were presented. This paper is a review paper and covers the period of 2017 and 2022 when it comes to solutions and statistics. The authors decided to focus on the latest data, mostly due to the rapid technology development and increased number of new methods, which positively affects diagnosis and prognosis.

Handheld hyperspectral imaging as a tool for the post-mortem interval estimation of human skeletal remains

In forensic medicine, estimating human skeletal remains' post-mortem interval (PMI) can be challenging. Following death, bones undergo a series of chemical and physical transformations due to their interactions with the surrounding environment. Post-mortem changes have been assessed using various methods, but estimating the PMI of skeletal remains could still be improved. We propose a new methodology with handheld hyperspectral imaging (HSI) system based on the first results from 104 human skeletal remains with PMIs ranging between 1 day and 2000 years. To differentiate between forensic and archaeological bone material, the Convolutional Neural Network analyzed 65.000 distinct diagnostic spectra: the classification accuracy was 0.58, 0.62, 0.73, 0.81, and 0.98 for PMIs of 0 week-2 weeks, 2 weeks-6 months, 6 months-1 year, 1 year-10 years, and >100 years, respectively. In conclusion, HSI can be used in forensic medicine to distinguish bone materials >100 years old from those <10 years old with an accuracy of 98%. The model has adequate predictive performance, and handheld HSI could serve as a novel approach to objectively and accurately determine the PMI of human skeletal remains.

An Ensemble Learning Method for Detection of Head and Neck Squamous Cell Carcinoma Using Polarized Hyperspectral Microscopic Imaging

Head and neck squamous cell carcinoma (HNSCC) has a high mortality rate. In this study, we developed a Stokes-vector-derived polarized hyperspectral imaging (PHSI) system for H&E-stained pathological slides with HNSCC and built a dataset to develop a deep learning classification method based on convolutional neural networks (CNN). We use our polarized hyperspectral microscope to collect the four Stokes parameter hypercubes (S0, S1, S2, and S3) from 56 patients and synthesize pseudo-RGB images using a transformation function that approximates the human eye's spectral response to visual stimuli. Each image is divided into patches. Data augmentation is applied using rotations and flipping. We create a four-branch model architecture where each branch is trained on one Stokes parameter individually, then we freeze the branches and fine-tune the top layers of our model to generate final predictions. Our results show high accuracy, sensitivity, and specificity, indicating that our model performed well on our dataset. Future works can improve upon these results by training on more varied data, classifying tumors based on their grade, and introducing more recent architectural techniques.

Polarized hyperspectral microscopic imaging system for enhancing the visualization of collagen fibers and head and neck squamous cell carcinoma

Significance

Polarized hyperspectral microscopes with the capability of full Stokes vector imaging have potential for many biological and medical applications.

Aim

The aim of this study is to investigate polarized hyperspectral imaging (PHSI) for improving the visualization of collagen fibers, which is an important biomarker related to tumor development, and improving the differentiation of normal and tumor cells on pathologic slides.

Approach

We customized a polarized hyperspectral microscopic imaging system comprising an upright microscope with a motorized stage, two linear polarizers, two liquid crystal variable retarders (LCVRs), and a compact SnapScan hyperspectral camera. The polarizers and LCVRs worked in tandem with the hyperspectral camera to acquire polarized hyperspectral images, which were further used to calculate four Stokes vectors: S 0 , S 1 , S 2 , and S 3 . Synthetic RGB images of the Stokes vectors were generated for the visualization of cellular components in PHSI images. Regions of interest of collagen, normal cells, and tumor cells in the synthetic RGB images were selected, and spectral signatures of the selected components were extracted for comparison. Specifically, we qualitatively and quantitatively investigated the enhanced visualization and spectral characteristics of dense fibers and sparse fibers in normal stroma tissue, fibers accumulated within tumors, and fibers accumulated around tumors.

Results

By employing our customized polarized hyperspectral microscope, we extract the spectral signatures of Stokes vector parameters of collagen as well as of tumor and normal cells. The measurement of Stokes vector parameters increased the image contrast of collagen fibers and cells in the slides.

Conclusions

With the spatial and spectral information from the Stokes vector data cubes (S 0 , S 1 , S 2 , and S 3 ), our PHSI microscope system enhances the visualization of tumor cells and tumor microenvironment components, thus being beneficial for pathology and oncology.

Hyperspectral imaging benchmark based on machine learning for intraoperative brain tumour detection

Brain surgery is one of the most common and effective treatments for brain tumour. However, neurosurgeons face the challenge of determining the boundaries of the tumour to achieve maximum resection, while avoiding damage to normal tissue that may cause neurological sequelae to patients. Hyperspectral (HS) imaging (HSI) has shown remarkable results as a diagnostic tool for tumour detection in different medical applications. In this work, we demonstrate, with a robust k-fold cross-validation approach, that HSI combined with the proposed processing framework is a promising intraoperative tool for in-vivo identification and delineation of brain tumours, including both primary (high-grade and low-grade) and secondary tumours. Analysis of the in-vivo brain database, consisting of 61 HS images from 34 different patients, achieve a highest median macro F1-Score result of 70.2 ± 7.9% on the test set using both spectral and spatial information. Here, we provide a benchmark based on machine learning for further developments in the field of in-vivo brain tumour detection and delineation using hyperspectral imaging to be used as a real-time decision support tool during neurosurgical workflows.

Alternative intraoperative optical imaging modalities for fluorescence angiography in gastrointestinal surgery: spectral imaging and imaging photoplethysmography

INTRODUCTION

Intraoperative near-infrared fluorescence angiography with indocyanine green (ICG-FA) is a well-established modality in gastrointestinal surgery. Its main drawback is the application of a fluorescent agent with possible side effects for patients. The goal of this review paper is the presentation of alternative, non-invasive optical imaging methods and their comparison with ICG-FA.

MATERIAL AND METHODS

The principles of ICG-FA, spectral imaging, imaging photoplethysmography (iPPG), and their applications in gastrointestinal surgery are described based on selected published works.

RESULTS

The main applications of the three modalities are the evaluation of tissue perfusion, the identification of risk structures, and tissue segmentation or classification. While the ICG-FA images are mainly evaluated visually, leading to subjective interpretations, quantitative physiological parameters and tissue segmentation are provided in spectral imaging and iPPG. The combination of ICG-FA and spectral imaging is a promising method.

CONCLUSIONS

Non-invasive spectral imaging and iPPG have shown promising results in gastrointestinal surgery. They can overcome the main drawbacks of ICG-FA, i.e. the use of contrast agents, the lack of quantitative analysis, repeatability, and a difficult standardization of the acquisition. Further technical improvements and clinical evaluations are necessary to establish them in daily clinical routine.

InSPECtor: an end-to-end design framework for compressive pixelated hyperspectral instruments

Classic designs of hyperspectral instrumentation densely sample the spatial and spectral information of the scene of interest. Data may be compressed after the acquisition. In this paper, we introduce a framework for the design of an optimized, micropatterned snapshot hyperspectral imager that acquires an optimized subset of the spatial and spectral information in the scene. The data is thereby already compressed at the sensor level but can be restored to the full hyperspectral data cube by the jointly optimized reconstructor. This framework is implemented with TensorFlow and makes use of its automatic differentiation for the joint optimization of the layout of the micropatterned filter array as well as the reconstructor. We explore the achievable compression ratio for different numbers of filter passbands, number of scanning frames, and filter layouts using data collected by the Hyperscout instrument. We show resulting instrument designs that take snapshot measurements without losing significant information while reducing the data volume, acquisition time, or detector space by a factor of 40 as compared to classic, dense sampling. The joint optimization of a compressive hyperspectral imager design and the accompanying reconstructor provides an avenue to substantially reduce the data volume from hyperspectral imagers.

Unsupervised design for broadband multispectral and polarization filter array patterns

Imaging multiple wavelength and polarization components is problematic due to the complexity of equipment and the increase in the number of imaging shots, so imaging using filter arrays with various patterns has been widely reported from elemental research to practical applications. Most of them use bandpass filters with different center wavelengths for each pixel. Recently, however, filter arrays with multimodal transmission characteristics have been proposed using photonic crystals or Fabry-Perot filters. In any of these methods, the design of the filter array arrangement pattern is important to improve the quality of the captured image, as well as the improvement of the demosaicking algorithm. One way to design a filter array pattern is to minimize the mean squared error (MSE) between the ideal image and the demosaicked image. However, the more multidimensional the imaging components, the more difficult it becomes to collect training data. In such cases, it is necessary to empirically determine candidate transmission characteristics and patterns of filter arrays. In this study, we propose a method for evaluating filter array patterns without using any training data in the design of filter arrays for multispectral and polarization imaging. The proposed method estimates the MSE by approximating the autocorrelation matrix without using image data by expressing the imaging model as a linear forward problem and the demosaicking as a linear inverse problem. Since this method can be applied not only to ideal bandpass filter arrangements, but also to multispectral filter arrays with multimodal spectral transmission characteristics and even multispectral polarization filter arrays with different extinction ratios at different wavelengths, we will show that image quality can be improved over empirical arrangements by evaluating these patterns and by testing examples of optimal designs using genetic algorithms.

Analysis of optical absorption of photoaged human skin using a high-frequency illumination microscopy analysis system

Skin is composed of different layers, including the stratum corneum, epidermal living layer and papillary and reticular dermis. Each has specific optical properties due to differences in their biological components. Alterations in the skin's cutaneous biological components resulting from photoaging caused by chronic exposure to UV light affect the deterioration of appearance associated with the skin's optical properties. Various methods for analysing cutaneous optical properties have been previously proposed, including mathematical models and computer simulations. However, these were insufficient to elucidate changes in each skin layer and comprehensively understand the skin's integrated optical properties. We focused on UV-induced yellowing of the facial skin. We evaluated site-specific optical absorption of human skin tissue sections to investigate the yellowish discoloration, which is suggested to be related to the photodamage process. The method includes our original technique of separating the transmitted and scattered light using high-frequency illumination microscopy, leading to microscopic analysis of the tissue's optical absorption in the regions of interest. In analysing the sun-exposed facial skin tissue sections, we successfully showed that dermal regions of aged skin have increased absorption at 450 nm, where yellowish colours are complemented. Furthermore, we confirmed that elastic fibres with observable histological disorder resulting from photodamage are a prominent source of high optical absorption. We detected changes in the skin's optical absorption associated with dermal degeneration resulting from photodamage using a novel optical microscopy technique. The results provide a base for the evaluation of optical property changes for both yellowing discoloration and other tissue disorders.

Multispectral Imaging Method for Rapid Identification and Analysis of Paraffin-Embedded Pathological Tissues

The study of the interaction between light and biological tissue is of great help in the identification of diseases as well as structural alterations in tissues. In the present study, we have developed a tissue diagnostic technique by using multispectral imaging in the visible spectrum combined with principal component analysis (PCA). We used information from the propagation of light through paraffin-embedded tissues to assess differences in the eye tissues of control mouse embryos compared to mouse embryos whose mothers were deprived of folic acid (FA), a crucial vitamin necessary for the growth and development of the fetus. After acquiring the endmembers from the multispectral images, spectral unmixing was used to identify the abundances of those endmembers in each pixel. For each acquired image, the final analysis was performed by performing a pixel-by-pixel and wavelength-by-wavelength absorbance calculation. Non-negative least squares (NNLS) were used in this research. The abundance maps obtained for the first endmember revealed vascular alterations (vitreous and choroid) in the embryos with maternal FA deficiency. However, the abundance maps obtained for the third endmember showed alterations in the texture of some tissues such as the lens and retina. Results indicated that multispectral imaging applied to paraffin-embedded tissues enhanced tissue visualization. Using this method, first, it can be seen tissue damage location and then decide what kind of biological techniques to apply.

Area-Selective Atomic Layer Deposition of ZnO on Si\SiO2 Modified with Tris(dimethylamino)methylsilane

Delayed atomic layer deposition (ALD) of ZnO, i.e., area selective (AS)-ALD, was successfully achieved on silicon wafers (Si\SiO₂) terminated with tris(dimethylamino)methylsilane (TDMAMS). This resist molecule was deposited in a home-built, near-atmospheric pressure, flow-through, gas-phase reactor. TDMAMS had previously been shown to react with Si\SiO₂ in a single cycle/reaction and to drastically reduce the number of silanols that remain at the surface. ZnO was deposited in a commercial ALD system using dimethylzinc (DMZ) as the zinc precursor and H₂O as the coreactant. Deposition of TDMAMS was confirmed by spectroscopic ellipsometry (SE), X-ray photoelectron spectroscopy (XPS), and wetting. ALD of ZnO, including its selectivity on TDMAMS-terminated Si\SiO₂ (Si\SiO₂\TDMAMS), was confirmed by in situ multi-wavelength ellipsometry, ex situ SE, XPS, and/or high-sensitivity/low-energy ion scattering (HS-LEIS). The thermal stability of the TDMAMS resist layer, which is an important parameter for AS-ALD, was investigated by heating Si\SiO₂\TDMAMS in air and nitrogen at 330 °C. ALD of ZnO takes place more readily on Si\SiO₂\TDMAMS heated in the air than in N₂, suggesting greater damage to the surface heated in the air. To better understand the in situ ALD of ZnO on Si\SiO₂\TDMAMS and modified (thermally stressed) forms of it, the ellipsometry results were plotted as the normalized growth per cycle. Even one short pulse of TDMAMS effectively passivates Si\SiO₂. TDMAMS can be a useful, small-molecule inhibitor of ALD of ZnO on Si\SiO₂ surfaces.

Spectrally-shaped illumination for improved optical inspection of lateral III-V-semiconductor oxidation

We report an hyperspectral imaging microscopy system based on a spectrally-shaped illumination and its use to offer an enhanced in-situ inspection of a technological process that is critical in Vertical-Cavity Surface-Emitting Laser (VCSEL) manufacturing, the lateral III-V-semiconductor oxidation (AlOx). The implemented illumination source exploits a digital micromirror device (DMD) to arbitrarily tailor its emission spectrum. When combined to an imager, this source is shown to provide an additional ability to detect minute surface reflectance contrasts on any VCSEL or AlOx-based photonic structure and, in turn, to offer improved in-situ inspection of the oxide aperture shapes and dimensions down to the best-achievable optical resolution. The demonstrated technique is very versatile and could be readily extended to the real-time monitoring of oxidation or other semiconductor technological processes as soon as they rely on a real-time yet accurate measurement of spatio-spectral (reflectance) maps.

Blur-specific image quality assessment of microscopic hyperspectral images

Hyperspectral (HS) imaging (HSI) expands the number of channels captured within the electromagnetic spectrum with respect to regular imaging. Thus, microscopic HSI can improve cancer diagnosis by automatic classification of cells. However, homogeneous focus is difficult to achieve in such images, being the aim of this work to automatically quantify their focus for further image correction. A HS image database for focus assessment was captured. Subjective scores of image focus were obtained from 24 subjects and then correlated to state-of-the-art methods. Maximum Local Variation, Fast Image Sharpness block-based Method and Local Phase Coherence algorithms provided the best correlation results. With respect to execution time, LPC was the fastest.

Evaluation of the Reduction of Skin Hyperpigmentation Changes under the Influence of a Preparation Containing Kojic Acid Using Hyperspectral Imaging—Preliminary Study

Aim: The aim of this study was to demonstrate the effects of using a preparation containing kojic acid on skin hyperpigmentation using hyperspectral imaging, which enables a quantitative assessment of the effect of the preparation used on the reduction of skin discoloration. Materials and methods: Preliminary studies were carried out on 12 patients with post-acne skin. A hyperspectral camera with a spectral range of 400–1000 nm was used to image skin hyperpigmentation before and after the application of 3% kojic acid. Hyperspectral profiles were analyzed, and image analysis and processing methods were applied. Results: Studies performed using a hyperspectral camera have shown that kojic acid reduces skin discoloration by increasing skin brightness in 75% of patients tested, reducing skin contrast in approximately 83% and increasing skin homogeneity in approximately 67% of patients.

Compact and ultracompact spectral imagers: technology and applications in biomedical imaging

Significance

Spectral imaging, which includes hyperspectral and multispectral imaging, can provide images in numerous wavelength bands within and beyond the visible light spectrum. Emerging technologies that enable compact, portable spectral imaging cameras can facilitate new applications in biomedical imaging.

Aim

With this review paper, researchers will (1) understand the technological trends of upcoming spectral cameras, (2) understand new specific applications that portable spectral imaging unlocked, and (3) evaluate proper spectral imaging systems for their specific applications.

Approach

We performed a comprehensive literature review in three databases (Scopus, PubMed, and Web of Science). We included only fully realized systems with definable dimensions. To best accommodate many different definitions of "compact," we included a table of dimensions and weights for systems that met our definition.

Results

There is a wide variety of contributions from industry, academic, and hobbyist spaces. A variety of new engineering approaches, such as Fabry-Perot interferometers, spectrally resolved detector array (mosaic array), microelectro-mechanical systems, 3D printing, light-emitting diodes, and smartphones, were used in the construction of compact spectral imaging cameras. In bioimaging applications, these compact devices were used for in vivo and ex vivo diagnosis and surgical settings.

Conclusions

Compact and ultracompact spectral imagers are the future of spectral imaging systems. Researchers in the bioimaging fields are building systems that are low-cost, fast in acquisition time, and mobile enough to be handheld.

Histopathological diagnosis of colon cancer using micro-FTIR hyperspectral imaging and deep learning

BACKGROUND AND OBJECTIVE

Current studies based on digital biopsy images have achieved satisfactory results in detecting colon cancer despite their limited visual spectral range. Such methods may be less accurate when applied to samples taken from the tumor margin region or to samples containing multiple diagnoses. In contrast with the traditional computer vision approach, micro-FTIR hyperspectral images quantify the tissue-light interaction on a histochemical level and characterize different tissue pathologies, as they present a unique spectral signature. Therefore, this paper investigates the possibility of using hyperspectral images acquired over micro-FTIR absorbance spectroscopy to characterize healthy, inflammatory, and tumor colon tissues.

METHODS

The proposed method consists of modeling hyperspectral data into a voxel format to detect the patterns of each voxel using fully connected deep neural network. A web-based computer-aided diagnosis tool for inference is also provided.

RESULTS

Our experiments were performed using the K-fold cross-validation protocol in an intrapatient approach and achieved an overall accuracy of 99% using a deep neural network and 96% using a linear support vector machine. Through the experiments, we noticed the high performance of the method in characterizing such tissues using deep learning and hyperspectral images, indicating that the infrared spectrum contains relevant information and can be used to assist pathologists during the diagnostic process.

Design and Validation of a Custom-Made Hyperspectral Microscope Imaging System for Biomedical Applications

Hyperspectral microscope imaging (HMI) is an emerging modality that integrates spatial information collected by standard laboratory microscopy and the spectral-based contrast obtained by hyperspectral imaging and may be instrumental in establishing novel quantitative diagnostic methodologies, particularly in histopathology. Further expansion of HMI capabilities hinges upon the modularity and versatility of systems and their proper standardization. In this report, we describe the design, calibration, characterization, and validation of the custom-made laboratory HMI system based on a Zeiss Axiotron fully motorized microscope and a custom-developed Czerny-Turner-type monochromator. For these important steps, we rely on a previously designed calibration protocol. Validation of the system demonstrates a performance comparable to classic spectrometry laboratory systems. We further demonstrate validation against a laboratory hyperspectral imaging system for macroscopic samples, enabling future comparison of spectral imaging results across length scales. An example of the utility of our custom-made HMI system on a standard hematoxylin and eosin-stained histology slide is also shown.

Paper-based optical sensors paired with smartphones for biomedical analysis

The traditional analytical methods used for biomedical analysis are expensive and not easy to handle and require sophisticated instruments, thus their application is limited in resource-limited settings. Due to their portability, low cost, and ability to be applied to different analytical techniques, paper-based analytical devices are becoming valuable tools for biomedical analysis. The integration of smartphones into analytical devices has provided the ability to build portable, cost-effective, straightforward analytical devices for biomedical analysis and mobile health. The key aim of this review is to emphasize the recent applications of PADs combined with a smartphone for the optical analysis of biomedical species. We started this review by highlighting the type of papers and their modifications with different materials to prepare the PADs. After that, this review presents various detection methods including colorimetry, fluorescence, and luminescence where the smartphone is used for read-out. In the end, we provided the recent applications of the analysis of different biomedical compounds such as cancer and cardiovascular biomarkers, metal ions, glucose, viruses, etc. We believe that the present review will attract a wide scientific community in the areas of analytical chemistry, sensors, and clinical testing.

Towards Real-Time Hyperspectral Multi-Image Super-Resolution Reconstruction Applied to Histological Samples

Hyperspectral Imaging (HSI) is increasingly adopted in medical applications for the usefulness of understanding the spectral signature of specific organic and non-organic elements. The acquisition of such images is a complex task, and the commercial sensors that can measure such images is scarce down to the point that some of them have limited spatial resolution in the bands of interest. This work proposes an approach to enhance the spatial resolution of hyperspectral histology samples using super-resolution. As the data volume associated to HSI has always been an inconvenience for the image processing in practical terms, this work proposes a relatively low computationally intensive algorithm. Using multiple images of the same scene taken in a controlled environment (hyperspectral microscopic system) with sub-pixel shifts between them, the proposed algorithm can effectively enhance the spatial resolution of the sensor while maintaining the spectral signature of the pixels, competing in performance with other state-of-the-art super-resolution techniques, and paving the way towards its use in real-time applications.

A Video Transformer Network for Thyroid Cancer Detection on Hyperspectral Histologic Images

Hyperspectral imaging is a label-free and non-invasive imaging modality that seeks to capture images in different wavelengths. In this study, we used a vision transformer that was pre-trained from video data to detect thyroid cancer on hyperspectral images. We built a dataset of 49 whole slide hyperspectral images (WS-HSI) of thyroid cancer. To improve training, we introduced 5 new data augmentation methods that transform spectra. We achieved an F-1 score of 88.1% and an accuracy of 89.64% on our test dataset. The transformer network and the whole slide hyperspectral imaging technique can have many applications in digital pathology.

Effects of dimension reduction of hyperspectral images in skin gross pathology

BACKGROUND

Hyperspectral imaging (HSI) is an emerging modality for the gross pathology of the skin. Spectral signatures of HSI could discriminate malignant from benign tissue. Because of inherent redundancies in HSI and in order to facilitate the use of deep-learning models, dimension reduction is a common preprocessing step. The effects of dimension reduction choice, training scope, and number of retained dimensions have not been evaluated on skin HSI for segmentation tasks.

MATERIALS AND METHODS

An in-house dataset of HSI signatures from pigmented skin lesions was prepared and labeled with histology. Eleven different dimension reduction methods were used as preprocessing for tumor margin detection with support vector machines. Cluster-wise principal component analysis (ClusterPCA), a new variant of PCA, was proposed. The scope of application for dimension reduction was also investigated.

RESULTS

The components produced by ClusterPCA show good agreement with the expected optical properties of skin chromophores. Random forest importance performed best during classification. However, all methods suffered from low sensitivity and generalization.

CONCLUSION

Investigation of more complex reduction and segmentation schemes with emphasis on the nature of HSI and optical properties of the skin is necessary. Insights on dimension reduction for skin tissue could facilitate the development of HSI-based systems for cancer margin detection at gross level.

Conditional Generative Adversarial Network (cGAN) for Synthesis of Digital Histologic Images from Hyperspectral Images

Hyperspectral imaging (HSI) has been demonstrated in various digital pathology applications. However, the intrinsic high dimensionality of hyperspectral images makes it difficult for pathologists to visualize the information. The aim of this study is to develop a method to transform hyperspectral images of hemoxylin & eosin (H&E)-stained slides to natural-color RGB histologic images for easy visualization. Hyperspectral images were obtained at 40× magnification with an automated microscopic imaging system and downsampled by various factors to generate data equivalent to different magnifications. High-resolution digital histologic RGB images were cropped and registered to the corresponding hyperspectral images as the ground truth. A conditional generative adversarial network (cGAN) was trained to output natural color RGB images of the histological tissue samples. The generated synthetic RGBs have similar color and sharpness to real RGBs. Image classification was implemented using the real and synthetic RGBs, respectively, with a pretrained network. The classification of tumor and normal tissue using the HSI-synthesized RGBs yielded a comparable but slightly higher accuracy and AUC than the real RGBs. The proposed method can reduce the acquisition time of two imaging modalities while giving pathologists access to the high information density of HSI and the quality visualization of RGBs. This study demonstrated that HSI may provide a potentially better alternative to current RGB-based pathologic imaging and thus make HSI a viable tool for histopathological diagnosis.

Visible- and near-infrared hyperspectral imaging for the quantitative analysis of PD-L1+ cells in human lymphomas: Comparison with fluorescent multiplex immunohistochemistry

INTRODUCTION

We analyzed the expression of PD-L1 in human lymphomas using hyperspectral imaging (HSI) compared to visual assessment (VA) and conventional digital image analysis (DIA) to strengthen further the value of HSI as a tool for the evaluation of brightfield-based immunohistochemistry (IHC). In addition, fluorescent multiplex immunohistochemistry (mIHC) was used as a second detection method to analyze the impact of a different detection method.

MATERIAL AND METHODS

18 cases (6 follicular lymphomas and 12 diffuse large B-cell lymphomas) were stained for PD-L1 by IHC and for PD-L1, CD3, and CD8 by fluorescent mIHC. The percentage of positively stained cells was evaluated with VA, HSI, and DIA for IHC and VA and DIA for mIHC. Results were compared between the different methods of detection and analysis.

RESULTS

An overall high concordance was found between VA, HSI, and DIA in IHC (Cohens Kappa = 0.810^VA/HSI, 0.710 ^VA/DIA, and 0.516 ^HSI/DIA) and for VA^mIHCversus DIA^mIHC (Cohens Kappa = 0.894). Comparing IHC and mIHC general agreement differed depending on the methods compared but reached at most a moderate agreement (Coheńs Kappa between 0.250 and 0.483). This is reflected by the significantly higher percentage of PD-L1+ cells found with mIHC (p^Friedman = 0.014).

CONCLUSION

Our study shows a good concordance for the different analysis methods. Compared to VA and DIA, HSI proved to be a reliable tool for assessing IHC. Understanding the regulation of PD-L1 expression will further enlighten the role of PD-L1 as a biomarker. Therefore it is necessary to develop an instrument, such as HSI, which can offer a reliable and objective evaluation of PD-L1 expression.

An automatic whole-slide hyperspectral imaging microscope

Whole slide imaging (WSI) is a common step used in histopathology to quickly digitize stained histological slides. Digital whole-slide images not only improve the efficiency of labeling but also open the door for computer-aided diagnosis, specifically machine learning-based methods. Hyperspectral imaging (HSI) is an imaging modality that captures data in various wavelengths, some beyond the range of visible lights. In this study, we developed and implemented an automated microscopy system that can acquire hyperspectral whole slide images (HWSI). The system is robust since it consists of parts that can be swapped and bought from different manufacturers. We used the automated system and built a database of 49 HWSI of thyroid cancer. The automatic whole-slide hyperspectral imaging microscope can have many potential applications in biological and medical areas.

Polarized hyperspectral microscopic imaging for collagen visualization on pathologic slides of head and neck squamous cell carcinoma

We developed a polarized hyperspectral microscope to collect four types of Stokes vector data cubes (S0, S1, S2, and S3) of the pathologic slides with head and neck squamous cell carcinoma (HNSCC). Our system consists of an optical light microscope with a movable stage, two polarizers, two liquid crystal variable retarders (LCVRs), and a SnapScan hyperspectral camera. The polarizers and LCVRs work in tandem with the hyperspectral camera to acquire polarized hyperspectral images. Synthetic pseudo-RGB images are generated from the four Stokes vector data cubes based on a transformation function similar to the spectral response of human eye for the visualization of hyperspectral images. Collagen is the most abundant extracellular matrix (ECM) protein in the human body. A major focus of studying the ECM in tumor microenvironment is the role of collagen in both normal and abnormal function. Collagen tends to accumulate in and around tumors during cancer development and growth. In this study, we acquired images from normal regions containing normal cells and collagen fibers and from tumor regions containing cancerous squamous cells and collagen fibers on HNSCC pathologic slides. The preliminary results demonstrated that our customized polarized hyperspectral microscope is able to improve the visualization of collagen on HNSCC pathologic slides under different situations, including thick fibers of normal stroma, thin fibers of normal stroma, fibers of normal muscle cells, fibers accumulated in tumors, fibers accumulated around tumors. Our preliminary results also demonstrated that the customized polarized hyperspectral microscope is capable of extracting the spectral signatures of collagen based on Stokes vector parameters and can have various applications in pathology and oncology.

Estimating quantitative physiological and morphological tissue parameters of murine tumor models using hyperspectral imaging and optical profilometry

Understanding tumors and their microenvironment are essential for successful and accurate disease diagnosis. Tissue physiology and morphology are altered in tumors compared to healthy tissues, and there is a need to monitor tumors and their surrounding tissues, including blood vessels, non-invasively. This preliminary study utilizes a multimodal optical imaging system combining hyperspectral imaging (HSI) and three-dimensional (3D) optical profilometry (OP) to capture hyperspectral images and surface shapes of subcutaneously grown murine tumor models. Hyperspectral images are corrected with 3D OP data and analyzed using the inverse-adding doubling (IAD) method to extract tissue properties such as melanin volume fraction and oxygenation. Blood vessels are segmented using the B-COSFIRE algorithm from oxygenation maps. From 3D OP data, tumor volumes are calculated and compared to manual measurements using a vernier caliper. Results show that tumors can be distinguished from healthy tissue based on most extracted tissue parameters ( p < 0.05 ). Furthermore, blood oxygenation is 50% higher within the blood vessels than in the surrounding tissue, and tumor volumes calculated using 3D OP agree within 26% with manual measurements using a vernier caliper. Results suggest that combining HSI and OP could provide relevant quantitative information about tumors and improve the disease diagnosis.

Signal to noise ratio quantifies the contribution of spectral channels to classification of human head and neck tissues ex vivo using deep learning and multispectral imaging

SIGNIFICANCE

Accurate identification of tissues is critical for performing safe surgery. Combining multispectral imaging (MSI) with deep learning is a promising approach to increasing tissue discrimination and classification. Evaluating the contributions of spectral channels to tissue discrimination is important for improving MSI systems.

AIM

Develop a metric to quantify the contributions of individual spectral channels to tissue classification in MSI.

APPROACH

MSI was integrated into a digital operating microscope with three sensors and seven illuminants. Two convolutional neural network (CNN) models were trained to classify 11 head and neck tissue types using white light (RGB) or MSI images. The signal to noise ratio (SNR) of spectral channels was compared with the impact of channels on tissue classification performance as determined using CNN visualization methods.

RESULTS

Overall tissue classification accuracy was higher with use of MSI images compared with RGB images, both for classification of all 11 tissue types and binary classification of nerve and parotid ( p < 0.001 ). Removing spectral channels with SNR > 20 reduced tissue classification accuracy.

CONCLUSIONS

The spectral channel SNR is a useful metric for both understanding CNN tissue classification and quantifying the contributions of different spectral channels in an MSI system.

Intraoperative Assessment of Tumor Margins in Tissue Sections with Hyperspectral Imaging and Machine Learning

The intraoperative assessment of tumor margins of head and neck cancer is crucial for complete tumor resection and patient outcome. The current standard is to take tumor biopsies during surgery for frozen section analysis by a pathologist after H&E staining. This evaluation is time-consuming, subjective, methodologically limited and underlies a selection bias. Optical methods such as hyperspectral imaging (HSI) are therefore of high interest to overcome these limitations. We aimed to analyze the feasibility and accuracy of an intraoperative HSI assessment on unstained tissue sections taken from seven patients with oral squamous cell carcinoma. Afterwards, the tissue sections were subjected to standard histopathological processing and evaluation. We trained different machine learning models on the HSI data, including a supervised 3D convolutional neural network to perform tumor detection. The results were congruent with the histopathological annotations. Therefore, this approach enables the delineation of tumor margins with artificial HSI-based histopathological information during surgery with high speed and accuracy on par with traditional intraoperative tumor margin assessment (Accuracy: 0.76, Specificity: 0.89, Sensitivity: 0.48). With this, we introduce HSI in combination with ML hyperspectral imaging as a potential new tool for intraoperative tumor margin assessment.