1
|
Jong LJS, Appelman JGC, Sterenborg HJCM, Ruers TJM, Dashtbozorg B. Spatial and Spectral Reconstruction of Breast Lumpectomy Hyperspectral Images. SENSORS (BASEL, SWITZERLAND) 2024; 24:1567. [PMID: 38475103 DOI: 10.3390/s24051567] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Revised: 02/23/2024] [Accepted: 02/27/2024] [Indexed: 03/14/2024]
Abstract
(1) Background: Hyperspectral imaging has emerged as a promising margin assessment technique for breast-conserving surgery. However, to be implicated intraoperatively, it should be both fast and capable of yielding high-quality images to provide accurate guidance and decision-making throughout the surgery. As there exists a trade-off between image quality and data acquisition time, higher resolution images come at the cost of longer acquisition times and vice versa. (2) Methods: Therefore, in this study, we introduce a deep learning spatial-spectral reconstruction framework to obtain a high-resolution hyperspectral image from a low-resolution hyperspectral image combined with a high-resolution RGB image as input. (3) Results: Using the framework, we demonstrate the ability to perform a fast data acquisition during surgery while maintaining a high image quality, even in complex scenarios where challenges arise, such as blur due to motion artifacts, dead pixels on the camera sensor, noise from the sensor's reduced sensitivity at spectral extremities, and specular reflections caused by smooth surface areas of the tissue. (4) Conclusion: This gives the opportunity to facilitate an accurate margin assessment through intraoperative hyperspectral imaging.
Collapse
Affiliation(s)
- Lynn-Jade S Jong
- Image-Guided Surgery, Department of Surgery, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
- Department of Nanobiophysics, Faculty of Science and Technology, University of Twente, Drienerlolaan 5, 7522 NB Enschede, The Netherlands
| | - Jelmer G C Appelman
- Image-Guided Surgery, Department of Surgery, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
- Faculty of Science, Vrije Universiteit Amsterdam, De Boelelaan 1111, 1081 HV Amsterdam, The Netherlands
| | - Henricus J C M Sterenborg
- Image-Guided Surgery, Department of Surgery, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
| | - Theo J M Ruers
- Image-Guided Surgery, Department of Surgery, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
- Department of Nanobiophysics, Faculty of Science and Technology, University of Twente, Drienerlolaan 5, 7522 NB Enschede, The Netherlands
| | - Behdad Dashtbozorg
- Image-Guided Surgery, Department of Surgery, Netherlands Cancer Institute, Plesmanlaan 121, 1066 CX Amsterdam, The Netherlands
| |
Collapse
|
2
|
Ma L, Pruitt K, Fei B. A hyperspectral surgical microscope with super-resolution reconstruction for intraoperative image guidance. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2024; 12930:129300Z. [PMID: 38745746 PMCID: PMC11093589 DOI: 10.1117/12.3008789] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Hyperspectral imaging (HSI) is an emerging imaging modality in medical applications, especially for intraoperative image guidance. A surgical microscope improves surgeons' visualization with fine details during surgery. The combination of HSI and surgical microscope can provide a powerful tool for surgical guidance. However, to acquire high-resolution hyperspectral images, the long integration time and large image file size can be a burden for intraoperative applications. Super-resolution reconstruction allows acquisition of low-resolution hyperspectral images and generates high-resolution HSI. In this work, we developed a hyperspectral surgical microscope and employed our unsupervised super-resolution neural network, which generated high-resolution hyperspectral images with fine textures and spectral characteristics of tissues. The proposed method can reduce the acquisition time and save storage space taken up by hyperspectral images without compromising image quality, which will facilitate the adaptation of hyperspectral imaging technology in intraoperative image guidance.
Collapse
Affiliation(s)
- Ling Ma
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Kelden Pruitt
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Baowei Fei
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
3
|
Ma L, Pruitt K, Fei B. Dual-camera laparoscopic imaging with super-resolution reconstruction for intraoperative hyperspectral image guidance. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2024; 12928:129280I. [PMID: 38752166 PMCID: PMC11094590 DOI: 10.1117/12.3006573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/18/2024]
Abstract
Laparoscopic and robotic surgery, as one type of minimally invasive surgery (MIS), has gained popularity due to the improved surgeon ergonomics, instrument precision, operative time, and postoperative recovery. Hyperspectral imaging (HSI) is an emerging medical imaging modality, which has proved useful for intraoperative image guidance. Snapshot hyperspectral cameras are ideal for intraoperative laparoscopic imaging because of their compact size and light weight, but low spatial resolution can be a limitation. In this work, we developed a dual-camera laparoscopic imaging system that consists of a high-resolution color camera and a snapshot hyperspectral camera, and we employed super-resolution reconstruction to fuse the images from both cameras to generate high-resolution hyperspectral images. The experimental results show that our method can significantly improve the resolution of hyperspectral images without compromising the image quality or spectral signatures. The proposed super-resolution reconstruction method is promising to promote the employment of high-speed hyperspectral imaging in laparoscopic surgery.
Collapse
Affiliation(s)
- Ling Ma
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Kelden Pruitt
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Baowei Fei
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
4
|
Tran MH, Bryarly M, Ma L, Yousuf MS, Price TJ, Fei B. Nerve Detection and Visualization Using Hyperspectral Imaging for Surgical Guidance. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2024; 12930:129302A. [PMID: 38707637 PMCID: PMC11070131 DOI: 10.1117/12.3008470] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/07/2024]
Abstract
During surgery of delicate regions, differentiation between nerve and surrounding tissue is crucial. Hyperspectral imaging (HSI) techniques can enhance the contrast between types of tissue beyond what the human eye can differentiate. Whereas an RGB image captures 3 bands within the visible light range (e.g., 400 nm to 700 nm), HSI can acquire many bands in wavelength increments that highlight regions of an image across a wavelength spectrum. We developed a workflow to identify nerve tissues from other similar tissues such as fat, bone, and muscle. Our workflow uses spectral angle mapper (SAM) and endmember selection. The method is robust for different types of environment and lighting conditions. We validated our workflow on two samples of human tissues. We used a compact HSI system that can image from 400 to 1700 nm to produce HSI of the samples. On these two samples, we achieved an intersection-over-union (IoU) segmentation score of 84.15% and 76.73%, respectively. We showed that our workflow identifies nerve segments that are not easily seen in RGB images. This method is fast, does not rely on special hardware, and can be applied in real time. The hyperspectral imaging and nerve detection approach may provide a powerful tool for image-guided surgery.
Collapse
Affiliation(s)
- Minh Ha Tran
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Michelle Bryarly
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Ling Ma
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | | | - Theodore J. Price
- Department of Neuroscience, University of Texas at Dallas, Richardson, TX
| | - Baowei Fei
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
5
|
Zhou X, Ma L, Mubarak HK, Palsgrove D, Sumer BD, Chen AY, Fei B. Polarized hyperspectral microscopic imaging system for enhancing the visualization of collagen fibers and head and neck squamous cell carcinoma. JOURNAL OF BIOMEDICAL OPTICS 2024; 29:016005. [PMID: 38239390 PMCID: PMC10795499 DOI: 10.1117/1.jbo.29.1.016005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/01/2023] [Revised: 12/04/2023] [Accepted: 12/20/2023] [Indexed: 01/22/2024]
Abstract
Significance Polarized hyperspectral microscopes with the capability of full Stokes vector imaging have potential for many biological and medical applications. Aim The aim of this study is to investigate polarized hyperspectral imaging (PHSI) for improving the visualization of collagen fibers, which is an important biomarker related to tumor development, and improving the differentiation of normal and tumor cells on pathologic slides. Approach We customized a polarized hyperspectral microscopic imaging system comprising an upright microscope with a motorized stage, two linear polarizers, two liquid crystal variable retarders (LCVRs), and a compact SnapScan hyperspectral camera. The polarizers and LCVRs worked in tandem with the hyperspectral camera to acquire polarized hyperspectral images, which were further used to calculate four Stokes vectors: S 0 , S 1 , S 2 , and S 3 . Synthetic RGB images of the Stokes vectors were generated for the visualization of cellular components in PHSI images. Regions of interest of collagen, normal cells, and tumor cells in the synthetic RGB images were selected, and spectral signatures of the selected components were extracted for comparison. Specifically, we qualitatively and quantitatively investigated the enhanced visualization and spectral characteristics of dense fibers and sparse fibers in normal stroma tissue, fibers accumulated within tumors, and fibers accumulated around tumors. Results By employing our customized polarized hyperspectral microscope, we extract the spectral signatures of Stokes vector parameters of collagen as well as of tumor and normal cells. The measurement of Stokes vector parameters increased the image contrast of collagen fibers and cells in the slides. Conclusions With the spatial and spectral information from the Stokes vector data cubes (S 0 , S 1 , S 2 , and S 3 ), our PHSI microscope system enhances the visualization of tumor cells and tumor microenvironment components, thus being beneficial for pathology and oncology.
Collapse
Affiliation(s)
- Ximing Zhou
- The University of Texas at Dallas, Department of Bioengineering, Richardson, Texas, United States
- The University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, Texas, United States
| | - Ling Ma
- The University of Texas at Dallas, Department of Bioengineering, Richardson, Texas, United States
- The University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, Texas, United States
| | - Hasan K. Mubarak
- The University of Texas at Dallas, Department of Bioengineering, Richardson, Texas, United States
- The University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, Texas, United States
| | - Doreen Palsgrove
- The University of Texas Southwestern Medical Center, Department of Pathology, Dallas, Texas, United States
| | - Baran D. Sumer
- The University of Texas Southwestern Medical Center, Department of Otolaryngology, Dallas, Texas, United States
| | - Amy Y. Chen
- Emory University, Department of Otolaryngology, Atlanta, Georgia, United States
| | - Baowei Fei
- The University of Texas at Dallas, Department of Bioengineering, Richardson, Texas, United States
- The University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, Texas, United States
- The University of Texas Southwestern Medical Center, Department of Radiology, Dallas, Texas, United States
| |
Collapse
|
6
|
Al-Thelaya K, Gilal NU, Alzubaidi M, Majeed F, Agus M, Schneider J, Househ M. Applications of discriminative and deep learning feature extraction methods for whole slide image analysis: A survey. J Pathol Inform 2023; 14:100335. [PMID: 37928897 PMCID: PMC10622844 DOI: 10.1016/j.jpi.2023.100335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Revised: 07/17/2023] [Accepted: 07/19/2023] [Indexed: 11/07/2023] Open
Abstract
Digital pathology technologies, including whole slide imaging (WSI), have significantly improved modern clinical practices by facilitating storing, viewing, processing, and sharing digital scans of tissue glass slides. Researchers have proposed various artificial intelligence (AI) solutions for digital pathology applications, such as automated image analysis, to extract diagnostic information from WSI for improving pathology productivity, accuracy, and reproducibility. Feature extraction methods play a crucial role in transforming raw image data into meaningful representations for analysis, facilitating the characterization of tissue structures, cellular properties, and pathological patterns. These features have diverse applications in several digital pathology applications, such as cancer prognosis and diagnosis. Deep learning-based feature extraction methods have emerged as a promising approach to accurately represent WSI contents and have demonstrated superior performance in histology-related tasks. In this survey, we provide a comprehensive overview of feature extraction methods, including both manual and deep learning-based techniques, for the analysis of WSIs. We review relevant literature, analyze the discriminative and geometric features of WSIs (i.e., features suited to support the diagnostic process and extracted by "engineered" methods as opposed to AI), and explore predictive modeling techniques using AI and deep learning. This survey examines the advances, challenges, and opportunities in this rapidly evolving field, emphasizing the potential for accurate diagnosis, prognosis, and decision-making in digital pathology.
Collapse
Affiliation(s)
- Khaled Al-Thelaya
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Nauman Ullah Gilal
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mahmood Alzubaidi
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Fahad Majeed
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Marco Agus
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Jens Schneider
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| | - Mowafa Househ
- Department of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar
| |
Collapse
|
7
|
Tran MH, Gomez O, Fei B. A Video Transformer Network for Thyroid Cancer Detection on Hyperspectral Histologic Images. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12471:1247107. [PMID: 38577581 PMCID: PMC10993530 DOI: 10.1117/12.2654851] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/06/2024]
Abstract
Hyperspectral imaging is a label-free and non-invasive imaging modality that seeks to capture images in different wavelengths. In this study, we used a vision transformer that was pre-trained from video data to detect thyroid cancer on hyperspectral images. We built a dataset of 49 whole slide hyperspectral images (WS-HSI) of thyroid cancer. To improve training, we introduced 5 new data augmentation methods that transform spectra. We achieved an F-1 score of 88.1% and an accuracy of 89.64% on our test dataset. The transformer network and the whole slide hyperspectral imaging technique can have many applications in digital pathology.
Collapse
Affiliation(s)
- Minh Ha Tran
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
| | - Ofelia Gomez
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
| | - Baowei Fei
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|
8
|
Rathgeb A, Ma L, Tran M, Fei B. Extended Depth of Field Imaging for Mosaic Hyperspectral Images. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12471:1247109. [PMID: 38500779 PMCID: PMC10947262 DOI: 10.1117/12.2653923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/20/2024]
Abstract
Hyperspectral imaging (HSI) is an emerging modality for digital pathology. The purpose of this study is to develop an extended depth of field (EDOF) method for mosaic hyperspectral images acquired with a snapshot camera. EDOF is a technique for ensuring that an image is in focus at all points. A stack of mosaicked hyperspectral images of hematoxylin and eosin (H&E)-stained histologic slides were acquired at different positions along the z-axis and used to output a hyperspectral histologic image that was in-focus at every point. Three different methods were compared to achieve a fully focused image. We compared conventional patch-based methods to our proposed growth-based and band-based methods. The Brenner function was used to quantitatively measure the focus quality of each image measured. The results show that both of our proposed methods performed better qualitatively and quantitatively than the patch-based method, with the band-based method performing the best, as it leveraged dividing pixels into their proper wavelengths in addition to spatially, giving the algorithm better contrast to measure. In terms of speed, the band-based method was the fastest, followed by the patch-based method, with the growth-based method being the slowest. Our proposed extended depth of field hyperspectral imaging methods can have immediate applications in digital pathology, especially whole slide imaging, and other microscopic imaging.
Collapse
Affiliation(s)
- Armand Rathgeb
- University of Texas at Dallas, Department of Bioengineering, Richardson, TX
- University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, TX
| | - Ling Ma
- University of Texas at Dallas, Department of Bioengineering, Richardson, TX
- University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, TX
| | - Minh Tran
- University of Texas at Dallas, Department of Bioengineering, Richardson, TX
- University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, TX
| | - Baowei Fei
- University of Texas at Dallas, Department of Bioengineering, Richardson, TX
- University of Texas at Dallas, Center for Imaging and Surgical Innovation, Richardson, TX
- University of Texas Southwestern Medical Center, Department of Radiology, Dallas, TX
| |
Collapse
|
9
|
Ma L, Sherey J, Palsgrove D, Fei B. Conditional Generative Adversarial Network (cGAN) for Synthesis of Digital Histologic Images from Hyperspectral Images. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12471:124711D. [PMID: 38495871 PMCID: PMC10942653 DOI: 10.1117/12.2653715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
Abstract
Hyperspectral imaging (HSI) has been demonstrated in various digital pathology applications. However, the intrinsic high dimensionality of hyperspectral images makes it difficult for pathologists to visualize the information. The aim of this study is to develop a method to transform hyperspectral images of hemoxylin & eosin (H&E)-stained slides to natural-color RGB histologic images for easy visualization. Hyperspectral images were obtained at 40× magnification with an automated microscopic imaging system and downsampled by various factors to generate data equivalent to different magnifications. High-resolution digital histologic RGB images were cropped and registered to the corresponding hyperspectral images as the ground truth. A conditional generative adversarial network (cGAN) was trained to output natural color RGB images of the histological tissue samples. The generated synthetic RGBs have similar color and sharpness to real RGBs. Image classification was implemented using the real and synthetic RGBs, respectively, with a pretrained network. The classification of tumor and normal tissue using the HSI-synthesized RGBs yielded a comparable but slightly higher accuracy and AUC than the real RGBs. The proposed method can reduce the acquisition time of two imaging modalities while giving pathologists access to the high information density of HSI and the quality visualization of RGBs. This study demonstrated that HSI may provide a potentially better alternative to current RGB-based pathologic imaging and thus make HSI a viable tool for histopathological diagnosis.
Collapse
Affiliation(s)
- Ling Ma
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Jeremy Sherey
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
| | - Doreen Palsgrove
- Department of Pathology, University of Texas Southwestern Medical Center, Dallas, TX
| | - Baowei Fei
- Center for Imaging and Surgical Innovation, University of Texas at Dallas, Richardson, TX
- Department of Bioengineering, University of Texas at Dallas, Richardson, TX
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX
| |
Collapse
|