Label-Free Deep Profiling of the Tumor Microenvironment

Toward cancer detection by label-free microscopic imaging in oncological surgery: Techniques, instrumentation and applications

This review examines the clinical application of label-free microscopy and spectroscopy, which are based on optical signals emitted by tissue components. Over the past three decades, a variety of techniques have been investigated with the aim of developing an in situ histopathology method that can rapidly and accurately identify tumor margins during surgical procedures. These techniques can be divided into two groups. One group encompasses techniques exploiting linear optical signals, and includes infrared and Raman microspectroscopy, and autofluorescence microscopy. The second group includes techniques based on nonlinear optical signals, including harmonic generation, coherent Raman scattering, and multiphoton autofluorescence microscopy. Some of these methods provide comparable information, while others are complementary. However, all of them have distinct advantages and disadvantages due to their inherent nature. The first part of the review provides an explanation of the underlying physics of the excitation mechanisms and a description of the instrumentation. It also covers endomicroscopy and data analysis, which are important for understanding the current limitations in implementing label-free techniques in clinical settings. The second part of the review describes the application of label-free microscopy imaging to improve oncological surgery with focus on brain tumors and selected gastrointestinal cancers, and provides a critical assessment of the current state of translation of these methods into clinical practice. Finally, the potential of confocal laser endomicroscopy for the acquisition of autofluorescence is discussed in the context of immediate clinical applications.

Diagnosis of Fibrotic Interstitial Lung Diseases Based on the Combination of Label-Free Quantitative Multiphoton Fiber Histology and Machine Learning

Interstitial lung disease (ILD), characterized by inflammation and fibrosis, often suffers from low diagnostic accuracy and consistency. Traditional hematoxylin and eosin (H&E) staining primarily reveals cellular inflammation with limited detail on fibrosis. To address these issues, we introduce a pioneering label-free quantitative multiphoton fiber histology (MPFH) technique that delineates the intricate characteristics of collagen and elastin fibers for ILD diagnosis. We acquired colocated multiphoton and H&E-stained images from a single tissue slice. Multiphoton imaging was performed on the deparaffinized section to obtain fibrotic tissue information, followed by H&E staining to capture cellular information. This approach was tested in a blinded diagnostic trial among 7 pathologists involving 14 patients with relatively normal lung and 31 patients with ILD (11 idiopathic pulmonary fibrosis/usual interstitial pneumonia, 14 nonspecific interstitial pneumonia, and 6 pleuroparenchymal fibroelastosis). A customized algorithm extracted quantitative fiber indicators from multiphoton images. These indicators, combined with clinical and radiologic features, were used to develop an automatic multiclass ILD classifier. Using MPFH, we can acquire high-quality, colocalized images of collagen fibers, elastin fibers, and cells. We found that the type, distribution, and degree of fibrotic proliferation can effectively distinguish between different subtypes. The blind study showed that MPFH enhanced diagnostic consistency (κ values from 0.56 to 0.72) and accuracy (from 73.0% to 82.5%, P = .0090). The combination of quantitative fiber indicators effectively distinguished between different tissues, with areas under the receiver operating characteristic curves exceeding 0.92. The automatic classifier achieved 93.8% accuracy, closely paralleling the 92.2% accuracy of expert pathologists. The outcomes of our research underscore the transformative potential of MPFH in the field of fibrotic-ILD diagnostics. By integrating quantitative analysis of fiber characteristics with advanced machine learning algorithms, MPFH facilitates the automatic and accurate identification of various fibrotic disease subtypes, showcasing a significant leap forward in precision diagnostics.

Clinical confocal laser endomicroscopy for imaging of autofluorescence signals of human brain tumors and non-tumor brain

PURPOSE

Analysis of autofluorescence holds promise for brain tumor delineation and diagnosis. Therefore, we investigated the potential of a commercial confocal laser scanning endomicroscopy (CLE) system for clinical imaging of brain tumors.

METHODS

A clinical CLE system with fiber probe and 488 nm laser excitation was used to acquire images of tissue autofluorescence. Fresh samples were obtained from routine surgeries (glioblastoma n = 6, meningioma n = 6, brain metastases n = 10, pituitary adenoma n = 2, non-tumor from surgery for the treatment of pharmacoresistant epilepsy n = 2). Additionally, in situ intraoperative label-free CLE was performed in three cases. The autofluorescence images were visually inspected for feature identification and quantification. For reference, tissue cryosections were prepared and further analyzed by label-free multiphoton microscopy and HE histology.

RESULTS

Label-free CLE enabled the acquisition of autofluorescence images for all cases. Autofluorescent structures were assigned to the cytoplasmic compartment of cells, elastin fibers, psammoma bodies and blood vessels by comparison to references. Sparse punctuated autofluorescence was identified in most images across all cases, while dense punctuated autofluorescence was most frequent in glioblastomas. Autofluorescent cells were observed in higher abundancies in images of non-tumor samples. Diffuse autofluorescence, fibers and round fluorescent structures were predominantly found in tumor tissues.

CONCLUSION

Label-free CLE imaging through an approved clinical device was able to visualize the characteristic autofluorescence patterns of human brain tumors and non-tumor brain tissue ex vivo and in situ. Therefore, this approach offers the possibility to obtain intraoperative diagnostic information before resection, importantly independent of any kind of marker or label.

Deep and dynamic metabolic and structural imaging in living tissues

Label-free imaging through two-photon autofluorescence of NAD(P)H allows for nondestructive, high-resolution visualization of cellular activities in living systems. However, its application to thick tissues has been restricted by its limited penetration depth within 300 μm, largely due to light scattering. Here, we demonstrate that the imaging depth for NAD(P)H can be extended to more than 700 μm in living engineered human multicellular microtissues by adopting multimode fiber-based, low repetition rate, high peak power, three-photon excitation of NAD(P)H at 1100 nm. This is achieved by having more than 0.5 megawatts peak power at the band of 1100 ± 25 nm through adaptively modulating multimodal nonlinear pulse propagation with a compact fiber shaper. Moreover, the eightfold increase in pulse energy enables faster imaging of monocyte behaviors in the living multicellular models. These results represent a substantial advance for deep and dynamic imaging of intact living biosystems. The modular design is anticipated to allow wide adoption for demanding imaging applications, including cancer research, immune responses, and tissue engineering.

Predicting DNA damage response in non-small cell lung cancer organoids via simultaneous label-free autofluorescence multiharmonic microscopy

The DNA damage response (DDR) is a fundamental readout for evaluating efficacy of cancer therapeutics, many of which target DNA associated processes. Current techniques to evaluate DDR rely on immunostaining for phosphorylated histone H2AX (γH2AX), which is an indicator of DNA double-strand breaks. While γH2AX immunostaining can provide a snapshot of DDR in fixed cell and tissue samples, this method is technically cumbersome due to temporal monitoring of DDR requiring timepoint replicates, extensive assay development efforts for 3D cell culture samples such as organoids, and time-consuming protocols for γH2AX immunostaining and its evaluation. The goal of this current study is to reduce overall burden on assay duration and development in non-small cell lung cancer (NSCLC) organoids by leveraging label-free multiphoton imaging. In this study, simultaneous label-free autofluorescence multiharmonic (SLAM) microscopy was used to provide rich intracellular information based on endogenous contrasts. SLAM microscopy enables imaging of live samples eliminating the need to generate sacrificial sample replicates and has improved image acquisition in 3D space over conventional confocal microscopy. Predictive modeling between label-free SLAM microscopy and γH2AX immunostained images confirmed strong correlation between SLAM image features and γH2AX signal. Across multiple DNA targeting chemotherapeutics and multiple patient-derived NSCLC organoid lines, the optical redox ratio and third harmonic generation channels were used to robustly predict DDR. Imaging via SLAM microscopy can be used to more rapidly predict DDR in live 3D NSCLC organoids with minimal sample handling and without labeling.

Large field-of-view metabolic profiling of murine brain tissue following morphine incubation using label-free multiphoton microscopy

BACKGROUND

Although the effects on neural activation and glucose consumption caused by opiates such as morphine are known, the metabolic machinery underlying opioid use and misuse is not fully explored. Multiphoton microscopy (MPM) techniques have been developed for optical imaging at high spatial resolution. Despite the increased use of MPM for neural imaging, the use of intrinsic optical contrast has seen minimal use in neuroscience.

NEW METHOD

We present a label-free, multimodal microscopy technique for metabolic profiling of murine brain tissue following incubation with morphine sulfate (MSO4). We evaluate two- and three-photon excited autofluorescence, and second and third harmonic generation to determine meaningful intrinsic contrast mechanisms in brain tissue using simultaneous label-free, autofluorescence multi-harmonic (SLAM) microscopy.

RESULTS

Regional differences quantified in the cortex, caudate, and thalamus of the brain demonstrate region-specific changes to metabolic profiles measured from FAD intensity, along with brain-wide quantification. While the overall intensity of FAD signal significantly decreased after morphine incubation, this metabolic molecule accumulated near the nucleus accumbens.

COMPARISON WITH EXISTING METHODS

Histopathology requires tissue fixation and staining to determine cell type and morphology, lacking information about cellular metabolism. Tools such as fMRI or PET imaging have been widely used, but lack cellular resolution. SLAM microscopy obviates the need for tissue preparation, permitting immediate use and imaging of tissue with subcellular resolution in its native environment.

CONCLUSIONS

This study demonstrates the utility of SLAM microscopy for label-free investigations of neural metabolism, especially the intensity changes in FAD autofluorescence and structural morphology from third-harmonic generation.

Noninvasive Nonlinear Optical Computational Histology

Cancer remains a global health challenge, demanding early detection and accurate diagnosis for improved patient outcomes. An intelligent paradigm is introduced that elevates label-free nonlinear optical imaging with contrastive patch-wise learning, yielding stain-free nonlinear optical computational histology (NOCH). NOCH enables swift, precise diagnostic analysis of fresh tissues, reducing patient anxiety and healthcare costs. Nonlinear modalities are evaluated, including stimulated Raman scattering and multiphoton imaging, for their ability to enhance tumor microenvironment sensitivity, pathological analysis, and cancer examination. Quantitative analysis confirmed that NOCH images accurately reproduce nuclear morphometric features across different cancer stages. Key diagnostic features, such as nuclear morphology, size, and nuclear-cytoplasmic contrast, are well preserved. NOCH models also demonstrate promising generalization when applied to other pathological tissues. The study unites label-free nonlinear optical imaging with histopathology using contrastive learning to establish stain-free computational histology. NOCH provides a rapid, non-invasive, and precise approach to surgical pathology, holding immense potential for revolutionizing cancer diagnosis and surgical interventions.

Label-free biomedical optical imaging

Label-free optical imaging employs natural and nondestructive approaches for the visualisation of biomedical samples for both biological assays and clinical diagnosis. Currently, this field revolves around multiple broad technology-oriented communities, each with a specific focus on a particular modality despite the existence of shared challenges and applications. As a result, biologists or clinical researchers who require label-free imaging are often not aware of the most appropriate modality to use. This manuscript presents a comprehensive review of and comparison among different label-free imaging modalities and discusses common challenges and applications. We expect this review to facilitate collaborative interactions between imaging communities, push the field forward and foster technological advancements, biophysical discoveries, as well as clinical detection, diagnosis, and monitoring of disease.

Restoration of metabolic functional metrics from label-free, two-photon human tissue images using multiscale deep-learning-based denoising algorithms

Significance

Label-free, two-photon excited fluorescence (TPEF) imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, noise and other artifacts present in these images severely complicate the extraction of biologically useful information.

Aim

We aim to employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-signal-to-noise ratio (SNR), TPEF images.

Approach

TPEF images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used to assess the impact of various denoising models, preprocessing methods, and data on metrics of image quality and the recovery of six metrics of metabolic function from the images relative to ground truth images.

Results

Optimized recovery of the redox ratio and mitochondrial organization is achieved using a novel algorithm based on deep denoising in the wavelet transform domain. This algorithm also leads to significant improvements in peak-SNR (PSNR) and structural similarity index measure (SSIM) for all images. Interestingly, other models yield even higher PSNR and SSIM improvements, but they are not optimal for recovery of metabolic function metrics.

Conclusions

Denoising algorithms can recover diagnostically useful information from low SNR label-free TPEF images and will be useful for the clinical translation of such imaging.

A deep learning-based stripe self-correction method for stitched microscopic images

Stitched fluorescence microscope images inevitably exist in various types of stripes or artifacts caused by uncertain factors such as optical devices or specimens, which severely affects the image quality and downstream quantitative analysis. Here, we present a deep learning-based Stripe Self-Correction method, so-called SSCOR. Specifically, we propose a proximity sampling scheme and adversarial reciprocal self-training paradigm that enable SSCOR to utilize stripe-free patches sampled from the stitched microscope image itself to correct their adjacent stripe patches. Comparing to off-the-shelf approaches, SSCOR can not only adaptively correct non-uniform, oblique, and grid stripes, but also remove scanning, bubble, and out-of-focus artifacts, achieving the state-of-the-art performance across different imaging conditions and modalities. Moreover, SSCOR does not require any physical parameter estimation, patch-wise manual annotation, or raw stitched information in the correction process. This provides an intelligent prior-free image restoration solution for microscopists or even microscope companies, thus ensuring more precise biomedical applications for researchers.

Nonlinear Imaging Histopathology: A Pipeline to Correlate Gold-Standard Hematoxylin and Eosin Staining With Modern Nonlinear Microscopy

Hematoxylin and eosin (H&E) staining, the century-old technique, has been the gold standard tool for pathologists to detect anomalies in tissues and diseases such as cancer. H&E staining is a cumbersome, time-consuming process that delays and wastes precious minutes during an intraoperative diagnosis. However, even in the modern era, real-time label-free imaging techniques such as simultaneous label-free autofluorescence multiharmonic (SLAM) microscopy have delivered several more layers of information to characterize a tissue with high precision. Still, they have yet to translate to the clinic. The slow translation rate can be attributed to the lack of direct comparisons between the old and new techniques. Our approach to solving this problem is to: 1) reduce dimensions by pre-sectioning the tissue in 500 μm slices, and 2) produce fiducial laser markings which appear in both SLAM and histological imaging. High peak-power femtosecond laser pulses enable ablation in a controlled and contained manner. We perform laser marking on a grid of points encompassing the SLAM region of interest. We optimize laser power, numerical aperture, and timing to produce axially extended marking, hence multilayered fiducial markers, with minimal damage to the surrounding tissues. We performed this co-registration over an area of 3 × 3 mm2 of freshly excised mouse kidney and intestine, followed by standard H&E staining. Reduced dimensionality and the use of laser markings provided a comparison of the old and new techniques, giving a wealth of correlative information and elevating the potential of translating nonlinear microscopy to the clinic for rapid pathological assessment.

Restoration of metabolic functional metrics from label-free, two-photon cervical tissue images using multiscale deep-learning-based denoising algorithms

Label-free, two-photon imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, this modality suffers from low signal arising from limitations imposed by the maximum permissible dose of illumination and the need for rapid image acquisition to avoid motion artifacts. Recently, deep learning methods have been developed to facilitate the extraction of quantitative information from such images. Here, we employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-SNR, two-photon images. Two-photon excited fluorescence (TPEF) images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used. We assess the impact of the specific denoising model, loss function, data transformation, and training dataset on established metrics of image restoration when comparing denoised single frame images with corresponding six frame averages, considered as the ground truth. We further assess the restoration accuracy of six metrics of metabolic function from the denoised images relative to ground truth images. Using a novel algorithm based on deep denoising in the wavelet transform domain, we demonstrate optimal recovery of metabolic function metrics. Our results highlight the promise of denoising algorithms to recover diagnostically useful information from low SNR label-free two-photon images and their potential importance in the clinical translation of such imaging.

Diffractive interconnects: all-optical permutation operation using diffractive networks

Permutation matrices form an important computational building block frequently used in various fields including, e.g., communications, information security, and data processing. Optical implementation of permutation operators with relatively large number of input-output interconnections based on power-efficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to all-optically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output field-of-view using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can all-optically perform arbitrarily selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in, e.g., security, image encryption, and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THz-bands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks.

Deep learning-based high-speed, large-field, and high-resolution multiphoton imaging

Multiphoton microscopy is a formidable tool for the pathological analysis of tumors. The physical limitations of imaging systems and the low efficiencies inherent in nonlinear processes have prevented the simultaneous achievement of high imaging speed and high resolution. We demonstrate a self-alignment dual-attention-guided residual-in-residual generative adversarial network trained with various multiphoton images. The network enhances image contrast and spatial resolution, suppresses noise, and scanning fringe artifacts, and eliminates the mutual exclusion between field of view, image quality, and imaging speed. The network may be integrated into commercial microscopes for large-scale, high-resolution, and low photobleaching studies of tumor environments.

Sub-60-fs, compact 1100-nm fiber laser system based on double-pass pre-chirp managed amplification

An ytterbium-doped, single-stage, double-pass nonlinear fiber amplification system was fabricated for amplifying an 1100-nm mode-locking fiber laser. Pre-chirp managed amplification (PCMA) was applied in realizing the nonlinear amplification process with an all-polarization-maintaining (PM) fiber construction. The system can deliver 19.8-nJ, 58.7-fs, 24.4-MHz amplified signal pulses with a 10-dB spectral range spanning from 1049 nm to 1130 nm. Further experimental investigations were conducted in exploring the dynamics of the double-pass nonlinear amplification process. This compact 1100-nm ultrafast fiber laser can be implemented for multi-photon microscopy (MPM) with deep penetration depth.

Label-free multimodal nonlinear optical imaging of needle biopsy cores for intraoperative cancer diagnosis

SIGNIFICANCE

Needle biopsy (NB) procedures are important for the initial diagnosis of many types of cancer. However, the possibility of NB specimens being unable to provide diagnostic information, (i.e., non-diagnostic sampling) and the time-consuming histological evaluation process can cause delays in diagnoses that affect patient care.

AIM

We aim to demonstrate the advantages of this label-free multimodal nonlinear optical imaging (NLOI) technique as a non-destructive point-of-procedure evaluation method for NB tissue cores, for the visualization and characterization of the tissue microenvironment.

APPROACH

A portable, label-free, multimodal NLOI system combined second-harmonic generation (SHG) and third-harmonic generation and two- and three-photon autofluorescence (2PF, 3PF) microscopy. It was used for intraoperative imaging of fresh NB tissue cores acquired during canine cancer surgeries, which involved liver, lung, and mammary tumors as well as soft-tissue sarcoma; in total, eight canine patients were recruited. An added tissue culture chamber enabled the use of this NLOI system for longitudinal imaging of fresh NB tissue cores taken from an induced rat mammary tumor and healthy mouse livers.

RESULTS

The intraoperative NLOI system was used to assess fresh canine NB specimens during veterinary cancer surgeries. Histology-like morphological features were visualized by the combination of four NLOI modalities at the point-of-procedure. The NLOI results provided quantitative information on the tissue microenvironment such as the collagen fiber orientation using Fourier-domain SHG analysis and metabolic profiling by optical redox ratio (ORR) defined by 2PF/(2PF + 3PF). The analyses showed that the canine mammary tumor had more randomly oriented collagen fibers compared to the tumor margin, and hepatocarcinoma had a wider distribution of ORR with a lower mean value compared to the liver fibrosis and the normal-appearing liver. Moreover, the loss of metabolic information during tissue degradation of fresh murine NB specimens was shown by overall intensity decreases in all channels and an increase of mean ORR from 0.94 (standard deviation 0.099) to 0.97 (standard deviation 0.077) during 1-h longitudinal imaging of a rat mammary tumor NB specimen. The tissue response to staurosporine (STS), an apoptotic inducer, from fresh murine liver NB specimens was also observed. The mean ORR decreased from 0.86 to 0.74 in the first 40 min and then increased to 0.8 during the rest of the hour of imaging, compared to the imaging results without the addition of STS, which showed a continuous increase of ORR from 0.72 to 0.75.

CONCLUSIONS

A label-free, multimodal NLOI platform reveals microstructural and metabolic information of the fresh NB cores during intraoperative cancer imaging. This system has been demonstrated on animal models to show its potential to provide a more comprehensive histological assessment and a better understanding of the unperturbed tumor microenvironment. Considering tissue degradation, or loss of viability upon fixation, this intraoperative NLOI system has the advantage of immediate assessment of freshly excised tissue specimens at the point of procedure.

Deep learning autofluorescence-harmonic microscopy

Laser scanning microscopy has inherent tradeoffs between imaging speed, field of view (FOV), and spatial resolution due to the limitations of sophisticated mechanical and optical setups, and deep learning networks have emerged to overcome these limitations without changing the system. Here, we demonstrate deep learning autofluorescence-harmonic microscopy (DLAM) based on self-alignment attention-guided residual-in-residual dense generative adversarial networks to close the gap between speed, FOV, and quality. Using the framework, we demonstrate label-free large-field multimodal imaging of clinicopathological tissues with enhanced spatial resolution and running time advantages. Statistical quality assessments show that the attention-guided residual dense connections minimize the persistent noise, distortions, and scanning fringes that degrade the autofluorescence-harmonic images and avoid reconstruction artifacts in the output images. With the advantages of high contrast, high fidelity, and high speed in image reconstruction, DLAM can act as a powerful tool for the noninvasive evaluation of diseases, neural activity, and embryogenesis.