Compressive hyperspectral time-resolved wide-field fluorescence lifetime imaging

Deep compressed multichannel adaptive optics scanning light ophthalmoscope

Adaptive optics scanning light ophthalmoscopy (AOSLO) reveals individual retinal cells and their function, microvasculature, and micropathologies in vivo. As compared to the single-channel offset pinhole and two-channel split-detector nonconfocal AOSLO designs, by providing multidirectional imaging capabilities, a recent generation of multidetector and (multi-)offset aperture AOSLO modalities has been demonstrated to provide critical information about retinal microstructures. However, increasing detection channels requires expensive optical components and/or critically increases imaging time. To address this issue, we present an innovative combination of machine learning and optics as an integrated technology to compressively capture 12 nonconfocal channel AOSLO images simultaneously. Imaging of healthy participants and diseased subjects using the proposed deep compressed multichannel AOSLO showed enhanced visualization of rods, cones, and mural cells with over an order-of-magnitude improvement in imaging speed as compared to conventional offset aperture imaging. To facilitate the adaptation and integration with other in vivo microscopy systems, we made optical design, acquisition, and computational reconstruction codes open source.

Enhanced fluorescence lifetime imaging microscopy denoising via principal component analysis

Fluorescence Lifetime Imaging Microscopy (FLIM) quantifies the autofluorescence lifetime to measure cellular metabolism, therapeutic efficacy, and disease progression. These dynamic processes are intrinsically heterogeneous, increasing the complexity of the signal analysis. Often noise reduction strategies that combine thresholding and non-selective data smoothing filters are applied. These can result in error introduction and data loss. To mitigate these issues, we develop noise-corrected principal component analysis (NC-PCA). This approach isolates the signal of interest by selectively identifying and removing the noise. To validate NC-PCA, a secondary analysis of FLIM images of patient-derived colorectal cancer organoids exposed to a range of therapeutics was performed. First, we demonstrate that NC-PCA decreases the uncertainty up to 4-fold in comparison to conventional analysis with no data loss. Then, using a merged data set, we show that NC-PCA, unlike conventional methods, identifies multiple metabolic states. Thus, NC-PCA provides an enabling tool to advance FLIM analysis across fields.

Rapid Wide-Field Correlative Mapping of Electronic and Vibrational Ultrafast Dynamics in Solids

Coupling between electronic and vibrational degrees of freedom in solids is responsible for many fundamental material properties, including superconductivity, ferroelectricity, high thermal conductivity, and indirect bandgap emission among many others. In heterogeneous materials electronic-vibrational coupling gives rise to spatial correlations between the electronic relaxation properties and vibrational dynamics. Visualizing and mapping these correlations may lead to important physical insights for applications that include electronics, optoelectronics, and energy technologies. However, due to the vastly different energy scales involved, measuring and correlating electronic and vibrational properties is challenging. While in principle, ultrafast pulses with sufficient bandwidth generate excited-state population and vibrational coherence signatures, the need to measure the signal point-by-point across the sample results in relatively slow acquisition, leading to an increased risk of sample photodamage and rendering the measurements highly susceptible to noise. Here, we introduce Parallel Rapid Imaging with Spectroscopic Mapping (PRISM), an ultrafast, wide-field, and coherent imaging technique, that allowed for the simultaneous acquisition of electronic state decay in the 0-10 ps range and vibrational spectra in the structurally sensitive low-frequency 5-600 cm-1 range. The exceptionally high speed of PRISM, exceeding 1.6 million time-resolved traces per second, enabled the mapping of electronic and vibrational properties across 80,000 pixels simultaneously in few-layer tungsten diselenide and perovskite materials. Correlations between the population and coherence maps reveal spatial heterogeneity not observed by either measurement alone. The ability to map electronic-vibrational coupling makes PRISM particularly well-suited for fundamental studies of complex solids and a wide range of materials applications.

Light sheet illumination in single-molecule localization microscopy for imaging of cellular architectures and molecular dynamics

Single-molecule localization microscopy has revealed cellular architectures and molecular dynamics beyond the diffraction limit of light. However, imaging thick samples presents challenges from increased fluorescence background. Light sheet illumination, which utilizes a plane of light for optical sectioning, is effective in reducing fluorescence background, photobleaching, and photodamage. Here, we present the principles of single-molecule localization microscopy and light sheet illumination, followed by an introduction to light sheet microscopy geometries and their imaging applications. Finally, we discuss light sheet illumination approaches for high- and super-resolution imaging of biological structures and dynamics.

A broadband hyperspectral image sensor with high spatio-temporal resolution

Hyperspectral imaging provides high-dimensional spatial-temporal-spectral information showing intrinsic matter characteristics1-5. Here we report an on-chip computational hyperspectral imaging framework with high spatial and temporal resolution. By integrating different broadband modulation materials on the image sensor chip, the target spectral information is non-uniformly and intrinsically coupled to each pixel with high light throughput. Using intelligent reconstruction algorithms, multi-channel images can be recovered from each frame, realizing real-time hyperspectral imaging. Following this framework, we fabricated a broadband visible-near-infrared (400-1,700 nm) hyperspectral image sensor using photolithography, with an average light throughput of 74.8% and 96 wavelength channels. The demonstrated resolution is 1,024 × 1,024 pixels at 124 fps. We demonstrated its wide applications, including chlorophyll and sugar quantification for intelligent agriculture, blood oxygen and water quality monitoring for human health, textile classification and apple bruise detection for industrial automation, and remote lunar detection for astronomy. The integrated hyperspectral image sensor weighs only tens of grams and can be assembled on various resource-limited platforms or equipped with off-the-shelf optical systems. The technique transforms the challenge of high-dimensional imaging from a high-cost manufacturing and cumbersome system to one that is solvable through on-chip compression and agile computation.

Overcoming photon and spatiotemporal sparsity in fluorescence lifetime imaging with SparseFLIM

Fluorescence lifetime imaging microscopy (FLIM) provides quantitative readouts of biochemical microenvironments, holding great promise for biomedical imaging. However, conventional FLIM relies on slow photon counting routines to accumulate sufficient photon statistics, restricting acquisition speeds. Here we demonstrate SparseFLIM, an intelligent paradigm for achieving high-fidelity FLIM reconstruction from sparse photon measurements. We develop a coupled bidirectional propagation network that enriches photon counts and recovers hidden spatial-temporal information. Quantitative analysis shows over tenfold photon enrichment, dramatically improving signal-to-noise ratio, lifetime accuracy, and correlation compared to the original sparse data. SparseFLIM enables reconstructing spatially and temporally undersampled FLIM at full resolution and channel count. The model exhibits strong generalization across experimental modalities including multispectral FLIM and in vivo endoscopic FLIM. This work establishes deep learning as a promising approach to enhance fluorescence lifetime imaging and transcend limitations imposed by the inherent codependence between measurement duration and information content.

Imaging quality enhancement in photon-counting single-pixel imaging via an ADMM-based deep unfolding network in small animal fluorescence imaging

Photon-counting single-pixel imaging (SPI) can image under low-light conditions with high-sensitivity detection. However, the imaging quality of these systems will degrade due to the undersampling and intrinsic photon-noise in practical applications. Here, we propose a deep unfolding network based on the Bayesian maximum a posterior (MAP) estimation and alternating direction method of multipliers (ADMM) algorithm. The reconstruction framework adopts a learnable denoiser by convolutional neural network (CNN) instead of explicit function with hand-crafted prior. Our method enhances the imaging quality compared to traditional methods and data-driven CNN under different photon-noise levels at a low sampling rate of 8%. Using our method, the sensitivity of photon-counting SPI prototype system for fluorescence imaging can reach 7.4 pmol/ml. In-vivo imaging of a mouse bearing tumor demonstrates an 8-times imaging efficiency improvement.

Advancements in fluorescence lifetime imaging microscopy Instrumentation: Towards high speed and 3D

Fluorescence lifetime imaging microscopy (FLIM) is a powerful imaging tool offering molecular specific insights into samples through the measurement of fluorescence decay time, with promising applications in diverse research fields. However, to acquire two-dimensional lifetime images, conventional FLIM relies on extensive scanning in both the spatial and temporal domain, resulting in much slower acquisition rates compared to intensity-based approaches. This problem is further magnified in three-dimensional imaging, as it necessitates additional scanning along the depth axis. Recent advancements have aimed to enhance the speed and three-dimensional imaging capabilities of FLIM. This review explores the progress made in addressing these challenges and discusses potential directions for future developments in FLIM instrumentation.

New approaches for low phototoxicity imaging of living cells and tissues

Fluorescence microscopy is a powerful tool used in scientific and medical research, but it is inextricably linked to phototoxicity. Neglecting phototoxicity can lead to erroneous or inconclusive results. Recently, several reports have addressed this issue, but it is still underestimated by many researchers, even though it can lead to cell death. Phototoxicity can be reduced by appropriate microscopic techniques and carefully designed experiments. This review focuses on recent strategies to reduce phototoxicity in microscopic imaging of living cells and tissues. We describe digital image processing and new hardware solutions. We point out new modifications of microscopy methods and hope that this review will interest microscopy hardware engineers. Our aim is to underscore the challenges and potential solutions integral to the design of microscopy systems. Simultaneously, we intend to engage biologists, offering insight into the latest technological advancements in imaging that can enhance their understanding and practice.

Time-gated single-pixel imaging of Cherenkov emission from a medical linear accelerator

Cherenkov imaging is an ideal tool for real-time in vivo verification of a radiation therapy dose. Given that radiation is pulsed from a medical linear accelerator (LINAC) together with weak Cherenkov emissions, time-gated high-sensitivity imaging is required for robust measurements. Instead of using an expensive camera system with limited efficiency of detection in each pixel, a single-pixel imaging (SPI) approach that maintains promising sensitivity over the entire spectral band could be used to provide a low-cost and viable alternative. A prototype SPI system was developed and demonstrated here in Cherenkov imaging of LINAC dose delivery to a water tank. Validation experiments were performed using four regular fields and an intensity-modulated radiotherapy (IMRT) delivery plan. The Cherenkov image-based projection percent depth dose curves (pPDDs) were compared to pPDDs simulated by the treatment planning system (TPS), with an overall average error of 0.48, 0.42, 0.65, and 1.08% for the 3, 5, 7, and 9 cm square beams, respectively. The composite image of the IMRT plan achieved a 85.9% pass rate using 3%/3 mm gamma index criteria, in comparing Cherenkov intensity and TPS dose. This study validates the feasibility of applying SPI to the Cherenkov imaging of radiotherapy dose for the first time to our knowledge.

Wide-field illumination diffuse optical tomography within a framework of single-pixel time-domain spatial frequency domain imaging

We present a wide-field illumination time-domain (TD) diffusion optical tomography (DOT) for three-dimensional (3-D) reconstruction within a shallow region under the illuminated surface of the turbid medium. The methodological foundation is laid on the single-pixel spatial frequency domain (SFD) imaging that facilitates the adoption of the well-established time-correlated single-photon counting (TCSPC)-based TD detection and generalized pulse spectrum techniques (GPST)-based reconstruction. To ameliorate the defects of the conventional diffusion equation (DE) in the forward modeling of TD-SFD-DOT, mainly the low accuracy in the near-field region and in profiling early-photon migration, we propose a modified model employing the time-dependent δ-P1 approximation and verify its improved accuracy in comparison with both the Monte Carlo and DE-based ones. For a simplified inversion process, a modified GPST approach is extended to TD-SFD-DOT that enables the effective separation of the absorption and scattering coefficients using a steady-state equivalent strategy. Furthermore, we set up a single-pixel TD-SFD-DOT system that employs the TCSPC-based TD detection in the SFD imaging framework. For assessments of the reconstruction approach and the system performance, phantom experiments are performed for a series of scenarios. The results show the effectiveness of the proposed methodology for rapid 3-D reconstruction of the absorption and scattering coefficients within a depth range of about 5 mean free pathlengths.

Applications of machine learning in time-domain fluorescence lifetime imaging: a review

Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.

Multiplexed near infrared fluorescence lifetime imaging in turbid media

Significance

Fluorescence lifetime imaging (FLI) plays a pivotal role in enhancing our understanding of biological systems, providing a valuable tool for non-invasive exploration of biomolecular and cellular dynamics, both in vitro and in vivo. Its ability to selectively target and multiplex various entities, alongside heightened sensitivity and specificity, offers rapid and cost-effective insights.

Aim

Our aim is to investigate the multiplexing capabilities of near-infrared (NIR) FLI within a scattering medium that mimics biological tissues. We strive to develop a comprehensive understanding of FLI's potential for multiplexing diverse targets within a complex, tissue-like environment.

Approach

We introduce an innovative Monte Carlo (MC) simulation approach that accurately describes the scattering behavior of fluorescent photons within turbid media. Applying phasor analyses, we enable the multiplexing of distinct targets within a single FLI image. Leveraging the state-of-the-art single-photon avalanche diode (SPAD) time-gated camera, SPAD512S, we conduct experimental wide-field FLI in the NIR regime.

Results

Our study demonstrates the successful multiplexing of dual targets within a single FLI image, reaching a depth of 1 cm within tissue-like phantoms. Through our novel MC simulation approach and phasor analyses, we showcase the effectiveness of our methodology in overcoming the challenges posed by scattering media.

Conclusions

This research underscores the potential of NIR FLI for multiplexing applications in complex biological environments. By combining advanced simulation techniques with cutting-edge experimental tools, we introduce significant results in the non-invasive exploration of biomolecular dynamics, to advance the field of FLI research.

Nanocrystals for Deep-Tissue In Vivo Luminescence Imaging in the Near-Infrared Region

In vivo imaging technologies have emerged as a powerful tool for both fundamental research and clinical practice. In particular, luminescence imaging in the tissue-transparent near-infrared (NIR, 700-1700 nm) region offers tremendous potential for visualizing biological architectures and pathophysiological events in living subjects with deep tissue penetration and high imaging contrast owing to the reduced light-tissue interactions of absorption, scattering, and autofluorescence. The distinctive quantum effects of nanocrystals have been harnessed to achieve exceptional photophysical properties, establishing them as a promising category of luminescent probes. In this comprehensive review, the interactions between light and biological tissues, as well as the advantages of NIR light for in vivo luminescence imaging, are initially elaborated. Subsequently, we focus on achieving deep tissue penetration and improved imaging contrast by optimizing the performance of nanocrystal fluorophores. The ingenious design strategies of NIR nanocrystal probes are discussed, along with their respective biomedical applications in versatile in vivo luminescence imaging modalities. Finally, thought-provoking reflections on the challenges and prospects for future clinical translation of nanocrystal-based in vivo luminescence imaging in the NIR region are wisely provided.

Fast data fitting scheme for compressive multispectral fluorescence lifetime imaging

A single-pixel camera combined with compressive sensing techniques is a promising fluorescence microscope scheme for acquiring a multidimensional dataset (space, spectrum, and lifetime) and for reducing the measurement time with respect to conventional microscope schemes. However, upon completing the acquisition, a computational step is necessary for image reconstruction and data analysis, which can be time-consuming, potentially canceling out the beneficial effect of compressive sensing. In this work, we propose and experimentally validate a fast-fit workflow based on global analysis and multiple linear fits, which significantly reduces the computation time from tens of minutes to less than 1 s. Moreover, as the method is interlaced with the measurement flow, it can be applied in parallel with the acquisitions.

Macroscopic temporally and spectrally resolved fluorescence imaging enhanced by laser-wavelength multiplexing

We present a laser scanning system for macroscopic samples that records fully resolved decay curves in individual pixels, resolves the images in 16 wavelength channels, and records simultaneously at several laser wavelengths. By using confocal detection, the system delivers images that are virtually free of lateral scattering and out-of-focus haze. Image formats can be up to 256 × 256 pixels and up to 1024 time channels. We demonstrate the performance of the system both on model experiments with fluorescent micro-beads and on the tumor model in the living mice.

Optical single-pixel volumetric imaging by three-dimensional light-field illumination

Three-dimensional single-pixel imaging (3D SPI) has become an attractive imaging modality for both biomedical research and optical sensing. 3D-SPI techniques generally depend on time-of-flight or stereovision principle to extract depth information from backscattered light. However, existing implementations for these two optical schemes are limited to surface mapping of 3D objects at depth resolutions, at best, at the millimeter level. Here, we report 3D light-field illumination single-pixel microscopy (3D-LFI-SPM) that enables volumetric imaging of microscopic objects with a near-diffraction-limit 3D optical resolution. Aimed at 3D space reconstruction, 3D-LFI-SPM optically samples the 3D Fourier spectrum by combining 3D structured light-field illumination with single-element intensity detection. We build a 3D-LFI-SPM prototype that provides an imaging volume of ∼390 × 390 × 3,800 μm3 and achieves 2.7-μm lateral resolution and better than 37-μm axial resolution. Its capability of 3D visualization of label-free optical absorption contrast is demonstrated by imaging single algal cells in vivo. Our approach opens broad perspectives for 3D SPI with potential applications in various fields, such as biomedical functional imaging.

Single photon single pixel imaging into thick scattering medium

Imaging into thick scattering medium is a long-standing challenge. Beyond the quasi-ballistic regime, multiple scattering scrambles the spatiotemporal information of incident/emitted light, making canonical imaging based on light focusing nearly impossible. Diffusion optical tomography (DOT) is one of the most popular approach to look inside scattering medium, but quantitatively inverting the diffusion equation is ill-posed, and prior information of the medium is typically necessary, which is nontrivial to obtain. Here, we show theoretically and experimentally that, by synergizing the one-way light scattering characteristic of single pixel imaging with ultrasensitive single photon detection and a metric-guided image reconstruction, single photon single pixel imaging can serve as a simple and powerful alternative to DOT for imaging into thick scattering medium without prior knowledge or inverting the diffusion equation. We demonstrated an image resolution of 12 mm inside a 60 mm thick (∼ 78 mean free paths) scattering medium.

Deep learning-based fusion of widefield diffuse optical tomography and micro-CT structural priors for accurate 3D reconstructions

Widefield illumination and detection strategies leveraging structured light have enabled fast and robust probing of tissue properties over large surface areas and volumes. However, when applied to diffuse optical tomography (DOT) applications, they still require a time-consuming and expert-centric solving of an ill-posed inverse problem. Deep learning (DL) models have been recently proposed to facilitate this challenging step. Herein, we expand on a previously reported deep neural network (DNN) -based architecture (modified AUTOMAP - ModAM) for accurate and fast reconstructions of the absorption coefficient in 3D DOT based on a structured light illumination and detection scheme. Furthermore, we evaluate the improved performances when incorporating a micro-CT structural prior in the DNN-based workflow, named Z-AUTOMAP. This Z-AUTOMAP significantly improves the widefield imaging process's spatial resolution, especially in the transverse direction. The reported DL-based strategies are validated both in silico and in experimental phantom studies using spectral micro-CT priors. Overall, this is the first successful demonstration of micro-CT and DOT fusion using deep learning, greatly enhancing the prospect of rapid data-integration strategies, often demanded in challenging pre-clinical scenarios.

Nanosecond compressive fluorescence lifetime microscopy imaging via the RATS method with a direct reconstruction of lifetime maps

The RAndom Temporal Signals (RATS) method has proven to be a useful and versatile method for measuring photoluminescence (PL) dynamics and fluorescence lifetime imaging (FLIM). Here, we present two fundamental development steps in the method. First, we demonstrate that by using random digital laser modulation in RATS, it is possible to implement the measurement of PL dynamics with temporal resolution in units of nanoseconds. Secondly, we propose an alternative approach to evaluating FLIM measurements based on a single-pixel camera experiment. In contrast to the standard evaluation, which requires a lengthy iterative reconstruction of PL maps for each timepoint, here we use a limited set of predetermined PL lifetimes and calculate the amplitude maps corresponding to each lifetime. The alternative approach significantly saves post-processing time and, in addition, in a system with noise present, it shows better stability in terms of the accuracy of the FLIM spectrogram. Besides simulations that confirmed the functionality of the extension, we implemented the new advancements into a microscope optical setup for mapping PL dynamics on the micrometer scale. The presented principles were also verified experimentally by mapping a LuAG:Ce crystal surface.

Programmed Control of Fluorescence Blinking Patterns based on Electron Transfer in DNA

Fluorescence imaging uses changes in the fluorescence intensity and emission wavelength to analyze multiple targets simultaneously. To increase the number of targets that can be identified simultaneously, fluorescence blinking can be used as an additional parameter. To understand and eventually control blinking, we used DNA as a platform to elucidate the processes of electron transfer (ET) leading to blinking, down to the rate constants. With a fixed ET distance, various blinking patterns were observed depending on the DNA sequence between the donor and acceptor units of the DNA platform. The blinking pattern was successfully described with a combination of ET rate constants. Therefore, molecules with various blinking patterns can be developed by tuning ET. It is expected that the number of targets that can be analyzed simultaneously will increase by the power of the number of blinking patterns.

Tunable image projection spectrometry

We present tunable image projection spectrometry (TIPS), a Fourier-domain line-scan spectral imager with a tunable compression ratio. Compared to state-of-the-art spatial-domain pushbroom hyperspectral cameras, TIPS requires much fewer measurements and provides a higher light throughput. Using a rotating Dove prism and a cylindrical field lens, TIPS scans an input scene in the Fourier domain and captures a subset of multi-angled one-dimensional (1D) en face projections of the input scene, allowing a tailored data compression ratio for a given scene. We demonstrate the spectral imaging capability of TIPS with a hematoxylin and eosin (H&E) stained pathology slide. Moreover, we showed the spectral information obtained can be further converted to depths when combining TIPS with a low-coherence full-field spectral-domain interferometer.

Computational macroscopic lifetime imaging and concentration unmixing of autofluorescence

Single-pixel computational imaging can leverage highly sensitive detectors that concurrently acquire data across spectral and temporal domains. For molecular imaging, such methodology enables to collect rich intensity and lifetime multiplexed fluorescence datasets. Herein we report on the application of a single-pixel structured light-based platform for macroscopic imaging of tissue autofluorescence. The super-continuum visible excitation and hyperspectral single-pixel detection allow for parallel characterization of autofluorescence intensity and lifetime. Furthermore, we exploit a deep learning based data processing pipeline, to perform autofluorescence unmixing while yielding the autofluorophores' concentrations. The full scheme (setup and processing) is validated in silico and in vitro with clinically relevant autofluorophores flavin adenine dinucleotide, riboflavin, and protoporphyrin. The presented results demonstrate the potential of the methodology for macroscopically quantifying the intensity and lifetime of autofluorophores, with higher specificity for cases of mixed emissions, which are ubiquitous in autofluorescence and multiplexed in vivo imaging.

Polarity-Sensitive and Membrane-Specific Probe Quantitatively Monitoring Ferroptosis through Fluorescence Lifetime Imaging

As a new form of regulated cell death, ferroptosis is closely related to various diseases. To interpret this biological behavior and monitor related pathological processes, it is necessary to develop appropriate detection strategies and tools. Considering that ferroptosis is featured with remarkable lipid peroxidation of various cell membranes, it is logical to detect membranes' structural and environmental changes for the direct assessment of ferroptosis. For this sake, we designed novel polarity-sensitive fluorescent probes Mem-C₁C₁₈ and Mem-C₁₈C₁₈, which have superior plasma membrane anchorage, high brightness, and sensitive responses to environmental polarity by changing their fluorescence lifetimes. Mem-C₁C₁₈ with much less tendency to aggregate than Mem-C₁₈C₁₈ outperformed the latter in high resolution fluorescence labeling of artificial vesicle membranes and plasma membranes of live cells. Thus, Mem-C₁C₁₈ was selected to monitor plasma membranes damaged along ferroptosis process for the first time, in combination with the technique of fluorescence lifetime imaging (FLIM). After treating HeLa cells with Erastin, a typical ferroptosis inducer, the mean fluorescence lifetime of Mem-C₁C₁₈ displayed a considerable increase from 3.00 to 4.93 ns, with a 64% increase (corresponding to the polarity parameter Δf increased from 0.213 to 0.232). Therefore, our idea to utilize a probe to quantitate the changes in polarity of plasma membranes proves to be an effective method in the evaluation of the ferroptosis process.

Secondary Complementary Balancing Compressive Imaging with a Free-Space Balanced Amplified Photodetector

Single-pixel imaging (SPI) has attracted widespread attention because it generally uses a non-pixelated photodetector and a digital micromirror device (DMD) to acquire the object image. Since the modulated patterns seen from two reflection directions of the DMD are naturally complementary, one can apply complementary balanced measurements to greatly improve the measurement signal-to-noise ratio and reconstruction quality. However, the balance between two reflection arms significantly determines the quality of differential measurements. In this work, we propose and demonstrate a simple secondary complementary balancing mechanism to minimize the impact of the imbalance on the imaging system. In our SPI setup, we used a silicon free-space balanced amplified photodetector with 5 mm active diameter which could directly output the difference between two optical input signals in two reflection arms. Both simulation and experimental results have demonstrated that the use of secondary complementary balancing can result in a better cancellation of direct current components of measurements, and can acquire an image quality slightly better than that of single-arm single-pixel complementary measurement scheme (which is free from the trouble of optical imbalance) and over 20 times better than that of double-arm dual-pixel complementary measurement scheme under optical imbalance conditions.

An autofocus algorithm considering wavelength changes for large scale microscopic hyperspectral pathological imaging system

Microscopic hyperspectral imaging technology has been widely used to acquire pathological information of tissue sections. Autofocus is one of the most important steps in microscopic hyperspectral imaging systems to capture large scale or even whole slide images of pathological slides with high quality and high speed. However, there are quite few autofocus algorithm put forward for the microscopic hyperspectral imaging system. Therefore, this article proposes a Laplace operator based autofocus algorithm for microscopic hyperspectral imaging system which takes the influence of wavelength changes into consideration. Through the proposed algorithm, the focal length for each wavelength can be adjusted automatically to ensure that each single band image can be autofocused precisely with adaptive image sharpness evaluation method. In addition, to increase the capture speed, the relationship of wavelength and focal length is derived and the focal offsets among different single band images are calculated for pre-focusing. We have employed the proposed method on our own datasets and the experimental results show that it can capture large-scale microscopic hyperspectral pathology images with high precise.

Noise effect on 2D photoluminescence decay analysis using the RATS method in a single-pixel camera configuration

Using a random temporal signal for sample excitation (RATS method) is a new, capable approach to measuring photoluminescence (PL) dynamics. The method can be used in single-point measurement (0D), but also it can be converted to PL decay imaging (2D) using a single-pixel camera configuration. In both cases, the reconstruction of the PL decay and PL snapshot is affected by ubiquitous noise. This article provides a detailed analysis of the noise effect on the RATS method and possible strategies for its suppression. We carried out an extensive set of simulations focusing on the effect of noise introduced through the random excitation signal and the corresponding PL waveform. We show that the PL signal noise level is critical for the method. Furthermore, we analyze the role of acquisition time, where we demonstrate the need for a non-periodic excitation signal. We show that it is beneficial to increase the acquisition time and that increasing the number of measurements in the single-pixel camera configuration has a minimal effect above a certain threshold. Finally, we study the effect of a regularization parameter used in the deconvolution step, and we observe that there is an optimum value set by the noise present in the PL dataset. Our results provide a guideline for optimization of the RATS measurement, but we also study effects generally occurring in PL decay measurements methods relying on the deconvolution step.

Deep learning in macroscopic diffuse optical imaging

SIGNIFICANCE

Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis.

AIM

We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI).

APPROACH

First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography.

RESULTS

The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships.

CONCLUSIONS

The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.

Compressed sensing in fluorescence microscopy

Compressed sensing (CS) is a signal processing approach that solves ill-posed inverse problems, from under-sampled data with respect to the Nyquist criterium. CS exploits sparsity constraints based on the knowledge of prior information, relative to the structure of the object in the spatial or other domains. It is commonly used in image and video compression as well as in scientific and medical applications, including computed tomography and magnetic resonance imaging. In the field of fluorescence microscopy, it has been demonstrated to be valuable for fast and high-resolution imaging, from single-molecule localization, super-resolution to light-sheet microscopy. Furthermore, CS has found remarkable applications in the field of mesoscopic imaging, facilitating the study of small animals' organs and entire organisms. This review article illustrates the working principles of CS, its implementations in optical imaging and discusses several relevant uses of CS in the field of fluorescence imaging from super-resolution microscopy to mesoscopy.

Macroscopic Fluorescence Lifetime Imaging for Monitoring of Drug-Target Engagement

Precision medicine promises to improve therapeutic efficacy while reducing adverse effects, especially in oncology. However, despite great progresses in recent years, precision medicine for cancer treatment is not always part of routine care. Indeed, the ability to specifically tailor therapies to distinct patient profiles requires still significant improvements in targeted therapy development as well as decreases in drug treatment failures. In this regard, preclinical animal research is fundamental to advance our understanding of tumor biology, and diagnostic and therapeutic response. Most importantly, the ability to measure drug-target engagement accurately in live and intact animals is critical in guiding the development and optimization of targeted therapy. However, a major limitation of preclinical molecular imaging modalities is their lack of capability to directly and quantitatively discriminate between drug accumulation and drug-target engagement at the pathological site. Recently, we have developed Macroscopic Fluorescence Lifetime Imaging (MFLI) as a unique feature of optical imaging to quantitate in vivo drug-target engagement. MFLI quantitatively reports on nanoscale interactions via lifetime-sensing of Förster Resonance Energy Transfer (FRET) in live, intact animals. Hence, MFLI FRET acts as a direct reporter of receptor dimerization and target engagement via the measurement of the fraction of labeled-donor entity undergoing binding to its respective receptor. MFLI is expected to greatly impact preclinical imaging and also adjacent fields such as image-guided surgery and drug development.

Single-Detector Spectrometer Using a Superconducting Nanowire

Designing a spectrometer without the need for wavelength multiplexing optics can effectively reduce the complexity and physical footprint. On the basis of the computational spectroscopic strategy and combining a broadband-responsive dynamic detector, we successfully demonstrate an optics-free single-detector spectrometer that maps the tunable quantum efficiency of a superconducting nanowire into a matrix to build a solvable mathematical equation. Such a spectrometer can realize a broadband spectral responsivity ranging from 660 to 1900 nm. The spectral resolution at the telecom is sub-10 nm, exceeding the energy resolving capacity of existing infrared single-photon detectors. Meanwhile, benefiting from the optics-free setup, precise time-of-flight measurements can be simultaneously achieved. We have demonstrated a spectral LiDAR with eight spectral channels. This spectrometer scheme paves the way for applying superconducting nanowire detectors in multifunctional spectroscopy and represents a conceptual advancement for on-chip spectroscopy and spectral imaging.

3D denoised completion network for deep single-pixel reconstruction of hyperspectral images

Single-pixel imaging acquires an image by measuring its coefficients in a transform domain, thanks to a spatial light modulator. However, as measurements are sequential, only a few coefficients can be measured in the real-time applications. Therefore, single-pixel reconstruction is usually an underdetermined inverse problem that requires regularization to obtain an appropriate solution. Combined with a spectral detector, the concept of single-pixel imaging allows for hyperspectral imaging. While each channel can be reconstructed independently, we propose to exploit the spectral redundancy between channels to regularize the reconstruction problem. In particular, we introduce a denoised completion network that includes 3D convolution filters. Contrary to black-box approaches, our network combines the classical Tikhonov theory with the deep learning methodology, leading to an explainable network. Considering both simulated and experimental data, we demonstrate that the proposed approach yields hyperspectral images with higher quantitative metrics than the approaches developed for grayscale images.

Full spectrum fluorescence lifetime imaging with 0.5 nm spectral and 50 ps temporal resolution

The use of optical techniques to interrogate wide ranging samples from semiconductors to biological tissue for rapid analysis and diagnostics has gained wide adoption over the past decades. The desire to collect ever more spatially, spectrally and temporally detailed optical signatures for sample characterization has specifically driven a sharp rise in new optical microscopy technologies. Here we present a high-speed optical scanning microscope capable of capturing time resolved images across 512 spectral and 32 time channels in a single acquisition with the potential for ~0.2 frames per second (256 × 256 image pixels). Each pixel in the resulting images contains a detailed data cube for the study of diverse time resolved light driven phenomena. This is enabled by integration of system control electronics and on-chip processing which overcomes the challenges presented by high data volume and low imaging speed, often bottlenecks in previous systems.

High data volumes from multidimensional imaging techniques can lead to slow collection and processing times. Here, the authors implement multispectral fluorescence lifetime imaging microscopy (FLIM) that uses time-correlated photon counting technology to reach simultaneously high imaging rates combined with high spectral and temporal resolution.

Compressed sensing time-resolved spectrometer for quantification of light absorbers in turbid media

Time-resolved (TR) spectroscopy is well-suited to address the challenges of quantifying light absorbers in highly scattering media such as living tissue; however, current TR spectrometers are either based on expensive array detectors or rely on wavelength scanning. Here, we introduce a TR spectrometer architecture based on compressed sensing (CS) and time-correlated single-photon counting. Using both CS and basis scanning, we demonstrate that-in homogeneous and two-layer tissue-mimicking phantoms made of Intralipid and Indocyanine Green-the CS method agrees with or outperforms uncompressed approaches. Further, we illustrate the superior depth sensitivity of TR spectroscopy and highlight the potential of the device to quantify absorption changes in deeper (>1 cm) tissue layers.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

A Polarity-Sensitive Ratiometric Fluorescence Probe for Monitoring Changes in Lipid Droplets and Nucleus during Ferroptosis

Ferroptosis regulates cell death through reactive oxygen species (ROS)-associated lipid peroxide accumulation, which is expected to affect the structure and polarity of lipid droplets (LDs), but with no clear evidence. Herein, we report the first example of an LD/nucleus dual-targeted ratiometric fluorescent probe, CQPP, for monitoring polarity changes in the cellular microenvironment. Due to the donor-acceptor structure of CQPP, it offers ratiometric fluorescence emission and fluorescence lifetime signals that reflect polarity variations. Using nucleus imaging as a reference, CQPP was applied to report the increase in LD polarity and the homogenization of polarity between LDs and cytoplasm in the ferroptosis model. This LD/nucleus dual-targeted fluorescent probe shows the great potential of using fluorescence imaging to study ferroptosis and ferroptosis-related diseases.

Single-pixel image reconstruction from experimental data using neural networks

Single-pixel cameras that measure image coefficients have various promising applications, in particular for hyper-spectral imaging. Here, we investigate deep neural networks that when fed with experimental data can output high-quality images in real time. Assuming that the measurements are corrupted by mixed Poisson-Gaussian noise, we propose to map the raw data from the measurement domain to the image domain based on a Tikhonov regularization. This step can be implemented as the first layer of a deep neural network, followed by any architecture of layers that acts in the image domain. We also describe a framework for training the network in the presence of noise. In particular, our approach includes an estimation of the image intensity and experimental parameters, together with a normalization scheme that allows varying noise levels to be handled during training and testing. Finally, we present results from simulations and experimental acquisitions with varying noise levels. Our approach yields images with improved peak signal-to-noise ratios, even for noise levels that were foreseen during the training of the networks, which makes the approach particularly suitable to deal with experimental data. Furthermore, while this approach focuses on single-pixel imaging, it can be adapted for other computational optics problems.

Luminescence lifetime imaging of three-dimensional biological objects

A major focus of current biological studies is to fill the knowledge gaps between cell, tissue and organism scales. To this end, a wide array of contemporary optical analytical tools enable multiparameter quantitative imaging of live and fixed cells, three-dimensional (3D) systems, tissues, organs and organisms in the context of their complex spatiotemporal biological and molecular features. In particular, the modalities of luminescence lifetime imaging, comprising fluorescence lifetime imaging (FLI) and phosphorescence lifetime imaging microscopy (PLIM), in synergy with Förster resonance energy transfer (FRET) assays, provide a wealth of information. On the application side, the luminescence lifetime of endogenous molecules inside cells and tissues, overexpressed fluorescent protein fusion biosensor constructs or probes delivered externally provide molecular insights at multiple scales into protein-protein interaction networks, cellular metabolism, dynamics of molecular oxygen and hypoxia, physiologically important ions, and other physical and physiological parameters. Luminescence lifetime imaging offers a unique window into the physiological and structural environment of cells and tissues, enabling a new level of functional and molecular analysis in addition to providing 3D spatially resolved and longitudinal measurements that can range from microscopic to macroscopic scale. We provide an overview of luminescence lifetime imaging and summarize key biological applications from cells and tissues to organisms.

Multispectral compressive fluorescence lifetime imaging microscopy with a SPAD array detector

Multispectral/hyperspectral fluorescence lifetime imaging microscopy (λFLIM) is a promising tool for studying functional and structural biological processes. The rich information content provided by a multidimensional dataset is often in contrast with the acquisition speed. In this work, we develop and experimentally demonstrate a wide-field λFLIM setup, based on a novel time-resolved 18×1 single-photon avalanche diode array detector working in a single-pixel camera scheme, which parallelizes the spectral detection, reducing measurement time. The proposed system, which implements a single-pixel camera with a compressive sensing scheme, represents an optimal microscopy framework towards the design of λFLIM setups.

Fluorescence lifetime imaging via spatio-temporal speckle patterns in a single-pixel camera configuration

Photoluminescence (PL) spectroscopy offers excellent methods for mapping the PL decay on the nanosecond time scale. However, capturing maps of emission dynamics on the microsecond timescale can be highly time-consuming. We present a new approach to fluorescence lifetime imaging (FLIM), which combines the concept of random temporal speckles excitation (RATS) with the concept of a single-pixel camera based on spatial speckles. The spatio-temporal speckle pattern makes it possible to map PL dynamics with unmatched simplicity. Moreover, the method can acquire all the data necessary to map PL decay on the microsecond timescale within minutes. We present proof-of-principle measurements for two samples and compare the reconstructed decays to the non-imaging measurements. Finally, we discuss the effect of the preprocessing routine and other factors on the reconstruction noise level. The presented method is suitable for lifetime imaging processes in several samples, including monitoring charge carrier dynamics in perovskites or monitoring solid-state luminophores with a long lifetime of PL.

Applications of compressive sensing in spatial frequency domain imaging

SIGNIFICANCE

Spatial frequency domain imaging (SFDI) is an imaging modality that projects spatially modulated light patterns to determine optical property maps for absorption and reduced scattering of biological tissue via a pixel-by-pixel data acquisition and analysis procedure. Compressive sensing (CS) is a signal processing methodology which aims to reproduce the original signal with a reduced number of measurements, addressing the pixel-wise nature of SFDI. These methodologies have been combined for complex heterogenous data in both the image detection and data analysis stage in a compressive sensing SFDI (cs-SFDI) approach, showing reduction in both the data acquisition and overall computational time.

AIM

Application of CS in SFDI data acquisition and image reconstruction significantly improves data collection and image recovery time without loss of quantitative accuracy.

APPROACH

cs-SFDI has been applied to an increased heterogenic sample from the AppSFDI data set (back of the hand), highlighting the increased number of CS measurements required as compared to simple phantoms to accurately obtain optical property maps. A novel application of CS to the parameter recovery stage of image analysis has also been developed and validated.

RESULTS

Dimensionality reduction has been demonstrated using the increased heterogenic sample at both the acquisition and analysis stages. A data reduction of 30% for the cs-SFDI and up to 80% for the parameter recover was achieved as compared to traditional SFDI, while maintaining an error of <10  %   for the recovered optical property maps.

CONCLUSION

The application of data reduction through CS demonstrates additional capabilities for multi- and hyperspectral SFDI, providing advanced optical and physiological property maps.

High compression deep learning based single-pixel hyperspectral macroscopic fluorescence lifetime imaging in vivo

Single pixel imaging frameworks facilitate the acquisition of high-dimensional optical data in biological applications with photon starved conditions. However, they are still limited to slow acquisition times and low pixel resolution. Herein, we propose a convolutional neural network for fluorescence lifetime imaging with compressed sensing at high compression (NetFLICS-CR), which enables in vivo applications at enhanced resolution, acquisition and processing speeds, without the need for experimental training datasets. NetFLICS-CR produces intensity and lifetime reconstructions at 128 × 128 pixel resolution over 16 spectral channels while using only up to 1% of the required measurements, therefore reducing acquisition times from ∼2.5 hours at 50% compression to ∼3 minutes at 99% compression. Its potential is demonstrated in silico, in vitro and for mice in vivo through the monitoring of receptor-ligand interactions in liver and bladder and further imaging of intracellular delivery of the clinical drug Trastuzumab to HER2-positive breast tumor xenografts. The data acquisition time and resolution improvement through NetFLICS-CR, facilitate the translation of single pixel macroscopic flurorescence lifetime imaging (SP-MFLI) for in vivo monitoring of lifetime properties and drug uptake.

Multiplexed non-invasive tumor imaging of glucose metabolism and receptor-ligand engagement using dark quencher FRET acceptor

Rationale: Following an ever-increased focus on personalized medicine, there is a continuing need to develop preclinical molecular imaging modalities to guide the development and optimization of targeted therapies. Near-Infrared (NIR) Macroscopic Fluorescence Lifetime Förster Resonance Energy Transfer (MFLI-FRET) imaging offers a unique method to robustly quantify receptor-ligand engagement in live intact animals, which is critical to assess the delivery efficacy of therapeutics. However, to date, non-invasive imaging approaches that can simultaneously measure cellular drug delivery efficacy and metabolic response are lacking. A major challenge for the implementation of concurrent optical and MFLI-FRET in vivo whole-body preclinical imaging is the spectral crowding and cross-contamination between fluorescent probes. Methods: We report on a strategy that relies on a dark quencher enabling simultaneous assessment of receptor-ligand engagement and tumor metabolism in intact live mice. Several optical imaging approaches, such as in vitro NIR FLI microscopy (FLIM) and in vivo wide-field MFLI, were used to validate a novel donor-dark quencher FRET pair. IRDye 800CW 2-deoxyglucose (2-DG) imaging was multiplexed with MFLI-FRET of NIR-labeled transferrin FRET pair (Tf-AF700/Tf-QC-1) to monitor tumor metabolism and probe uptake in breast tumor xenografts in intact live nude mice. Immunohistochemistry was used to validate in vivo imaging results. Results: First, we establish that IRDye QC-1 (QC-1) is an effective NIR dark acceptor for the FRET-induced quenching of donor Alexa Fluor 700 (AF700). Second, we report on simultaneous in vivo imaging of the metabolic probe 2-DG and MFLI-FRET imaging of Tf-AF700/Tf-QC-1 uptake in tumors. Such multiplexed imaging revealed an inverse relationship between 2-DG uptake and Tf intracellular delivery, suggesting that 2-DG signal may predict the efficacy of intracellular targeted delivery. Conclusions: Overall, our methodology enables for the first time simultaneous non-invasive monitoring of intracellular drug delivery and metabolic response in preclinical studies.

UNMIX-ME: spectral and lifetime fluorescence unmixing via deep learning

Hyperspectral fluorescence lifetime imaging allows for the simultaneous acquisition of spectrally resolved temporal fluorescence emission decays. In turn, the acquired rich multidimensional data set enables simultaneous imaging of multiple fluorescent species for a comprehensive molecular assessment of biotissues. However, to enable quantitative imaging, inherent spectral overlap between the considered fluorescent probes and potential bleed-through must be considered. Such a task is performed via either spectral or lifetime unmixing, typically independently. Herein, we present "UNMIX-ME" (unmix multiple emissions), a deep learning-based fluorescence unmixing routine, capable of quantitative fluorophore unmixing by simultaneously using both spectral and temporal signatures. UNMIX-ME was trained and validated using an in silico framework replicating the data acquisition process of a compressive hyperspectral fluorescent lifetime imaging platform (HMFLI). It was benchmarked against a conventional LSQ method for tri and quadri-exponential simulated samples. Last, UNMIX-ME's potential was assessed for NIR FRET in vitro and in vivo preclinical applications.

Super Sub-Nyquist Single-Pixel Imaging by Total Variation Ascending Ordering of the Hadamard Basis

Single pixel imaging (SPI) captures images without array detectors or raster scanning. When combined with compressive sensing techniques it enables novel solutions for high-speed optical imaging and spectroscopy. However, when it comes to the real-time capture and analysis of a fast event, the challenge is the inherent trade-off between frame rate and image resolution. Due to the lack of sufficient sparsity and the intrinsic iterative process, conventional compressed sensing techniques have limited improvement in capturing natural scenes and displaying the images in real time. In this work, we demonstrate a novel alternative compressive imaging approach employing an efficient and easy-implementation sampling scheme based on reordering the deterministic Hadamard basis through their total variation. By this means, the number of measurements and acquisition are reduced significantly without needing complex minimization algorithms. We can recover a 128 × 128 image with a sampling ratio of 5% at the signal peak signal-to-noise ratio (PSNR) of 23.8 dB, achieving super sub-Nyquist sampling SPI. Compared to other widely used sampling e.g. standard Hadamard protocols and Gaussian matrix methods, this approach results in a significant improvement both in the compression ratio and image reconstruction quality, enabling SPI for high frame rate imaging or video applications.

Single-shot ultrafast imaging attaining 70 trillion frames per second

Real-time imaging of countless femtosecond dynamics requires extreme speeds orders of magnitude beyond the limits of electronic sensors. Existing femtosecond imaging modalities either require event repetition or provide single-shot acquisition with no more than 10¹³ frames per second (fps) and 3 × 10² frames. Here, we report compressed ultrafast spectral photography (CUSP), which attains several new records in single-shot multi-dimensional imaging speeds. In active mode, CUSP achieves both 7 × 10¹³ fps and 10³ frames simultaneously by synergizing spectral encoding, pulse splitting, temporal shearing, and compressed sensing-enabling unprecedented quantitative imaging of rapid nonlinear light-matter interaction. In passive mode, CUSP provides four-dimensional (4D) spectral imaging at 0.5 × 10¹² fps, allowing the first single-shot spectrally resolved fluorescence lifetime imaging microscopy (SR-FLIM). As a real-time multi-dimensional imaging technology with the highest speeds and most frames, CUSP is envisioned to play instrumental roles in numerous pivotal scientific studies without the need for event repetition.

Diffraction-unlimited imaging based on conventional optical devices

We propose a computational paradigm where off-the-shelf optical devices can be used to image objects in a scene well beyond their native optical resolution. By design, our approach is generic, does not require active illumination, and is applicable to several types of optical devices. It only requires the placement of a spatial light modulator some distance from the optical system. In this paper, we first introduce the acquisition strategy together with the reconstruction framework. We then conduct practical experiments with a webcam that confirm that this approach can image objects with substantially enhanced spatial resolution compared to the performance of the native optical device. We finally discuss potential applications, current limitations, and future research directions.