Deep learning model for ultrafast multifrequency optical property extractions for spatial frequency domain imaging

Deep-learning-enabled spatial frequency domain imaging of the spatiotemporal dynamics of skin physiology

Significance

Spatial frequency domain imaging (SFDI) is an emerging optical imaging modality for visualizing tissue absorption and scattering properties. This approach is promising for noninvasive wide field-of-view (FOV) monitoring of biophysiological processes in vivo.

Aim

We aim to develop deep-learning-enabled spatial frequency domain imaging (SFDI-net) for real-time large FOV imaging of the optical, structural, and physiological properties and demonstrate its application for probing the spatiotemporal dynamics of skin physiology.

Approach

SFDI-net, based on mapping of a two-layer structure into an equivalent homogeneous medium for spatially modulated light and with a convolutional neural network architecture, produces two-dimensional maps of optical, structural, and physiological parameters for bilayered tissue, including cutaneous hemoglobin concentration, oxygen saturation, scattering properties (reduced scattering coefficient and scattering power), melanin content, surface roughness, and epidermal thickness, with visible spatially modulated light at the camera frame rate.

Results

Compared with traditional approaches, SFDI-net achieves a real-time inversion speed and significantly improves image quality by effectively suppressing noise while preserving tissue structure without oversmoothing. We demonstrate the application of the SFDI-net for monitoring the spatiotemporal dynamics of forearm skin physiology in reactive hyperemia and rhythmic respiration and reveal their intricate patterns in hemodynamics.

Conclusions

Deep-learning-enabled spatial frequency domain imaging and SFDI-net may offer insights into the cardiorespiratory system and have promising clinical utility for disease diagnosis, surveillance, and therapeutic assessment. Future hardware and software advancements will bring SFDI-net to clinical practice.

GMOPNet: A GAN-MLP two-stage network for optical properties measurement of kiwifruit and peaches with spatial frequency domain imaging

Spatial frequency domain imaging (SFDI) is an imaging technique using spatially modulated illumination for measurement of optical properties. Conventional SFDI methods require capturing at least six images, making it time-consuming. This study presents a Generative Adversarial Network-Multi-Layer Perceptron (GAN-MLP) two-stage network (GMOPNet) for extracting high-precision optical properties of kiwifruit and peaches from a single SFDI image, enabling real-time continuous wide-band SFDI. The GMOPNet we proposed leverages the GAN to predict diffuse reflectance, followed by the MLP with Monte Carlo prior knowledge to predict optical properties. Our method achieves mean absolute percentage errors (MAPE) of 5.91% for the absorption coefficient (μa) and 5.23% for the reduced scattering coefficient ( [Formula: see text] ), reducing acquisition and processing time significantly, with single inference taking 31.13 ms. The MAPE of the μa and the [Formula: see text] were 6.73% and 6.34% for kiwifruit and 5.80% and 6.65% for peaches, respectively.

Deep-learning approach to stratified reconstructions of tissue absorption and scattering in time-domain spatial frequency domain imaging

Significance

The conventional optical properties (OPs) reconstruction in spatial frequency domain (SFD) imaging, like the lookup table (LUT) method, causes OPs aliasing and yields only average OPs without depth resolution. Integrating SFD imaging with time-resolved (TR) measurements enhances space-TR information, enabling improved reconstruction of absorption (μ a ) and reduced scattering (μ s ' ) coefficients at various depths.

Aim

To achieve the stratified reconstruction of OPs and the separation between μ a and μ s ' , using deep learning workflow based on the temporal and spatial information provided by time-domain SFD imaging technique, while enhancing the reconstruction accuracy.

Approach

Two data processing methods are employed for the OPs reconstruction with TR-SFD imaging, one is full TR data, and the other is the featured data extracted from the full TR data (E , continuous-wave component, ⟨ t ⟩ , mean time of flight). We compared their performance using a series of simulation and phantom validations.

Results

Compared to the LUT approach, utilizing full TR, E and ⟨ t ⟩ datasets yield high-resolution OPs reconstruction results. Among the three datasets employed, full TR demonstrates the optimal accuracy.

Conclusions

Utilizing the data obtained from SFD and TR measurement techniques allows for achieving high-resolution separation reconstruction of μ a and μ s ' at different depths within 5 mm.

Compressed single-shot 3D photoacoustic imaging with a single-element transducer

Three-dimensional (3D) photoacoustic imaging (PAI) can provide rich information content and has gained increasingly more attention in various biomedical applications. However, current 3D PAI methods either involves pointwise scanning of the 3D volume using a single-element transducer, which can be time-consuming, or requires an array of transducers, which is known to be complex and expensive. By utilizing a 3D encoder and compressed sensing techniques, we develop a new imaging modality that is capable of single-shot 3D PAI using a single-element transducer. The proposed method is validated with phantom study, which demonstrates single-shot 3D imaging of different objects and 3D tracking of a moving object. After one-time calibration, while the system could perform single-shot 3D imaging for different objects, the calibration could remain effective over 7 days, which is highly beneficial for practical translation. Overall, the experimental results showcase the potential of this technique for both scientific research and clinical applications.

Applications of optical property measurement for quality evaluation of agri-food products: a review

Spectroscopic techniques coupled with chemometric approaches have been widely used for quality evaluation of agricultural and food (agri-food) products due to the nondestructive, simple, fast, and easy characters. However, these techniques face the issues or challenges of relatively weak robustness, generalizability, and applicability in modeling and prediction because they measure the aggregate amount of light interaction with tissues, resulting in the combined effect of absorption and scattering of photons. Optical property measurement could separate absorption from scattering, providing new insights into more reliable prediction performance in quality evaluation, which is attracting increasing attention. In this review, a brief overview of the currently popular measurement techniques, in terms of light transfer principles and data analysis algorithms, is first presented. Then, the emphases are put on the recent advances of these techniques for measuring optical properties of agri-food products since 2000. Corresponding applications on qualitative and quantitative analyses of quality evaluation, as well as light transfer simulations within tissues, were reviewed. Furthermore, the leading groups working on optical property measurement worldwide are highlighted, which is the first summary to the best of our knowledge. Finally, challenges for optical property measurement are discussed, and some viewpoints on future research directions are also given.

Real-time and accurate estimation ex vivo of four basic optical properties from thin tissue based on a cascade forward neural network

Double integrating sphere measurements obtained from thin ex vivo tissues provides more spectral information and hence allows full estimation of all basic optical properties (OPs) theoretically. However, the ill-conditioned nature of the OP determination increases excessively with the reduction in tissue thickness. Therefore, it is crucial to develop a model for thin ex vivo tissues that is robust to noise. Herein, we present a deep learning solution to precisely extract four basic OPs in real-time from thin ex vivo tissues, leveraging a dedicated cascade forward neural network (CFNN) for each OP with an additional introduced input of the refractive index of the cuvette holder. The results show that the CFNN-based model enables accurate and fast evaluation of OPs, as well as robustness to noise. Our proposed method overcomes the highly ill-conditioned restriction of OP evaluation and can distinguish the effects of slight changes in measurable quantities without any a priori knowledge.

Machine Learning Diffuse Optical Tomography Using Extreme Gradient Boosting and Genetic Programming

Diffuse optical tomography (DOT) is a non-invasive method for detecting breast cancer; however, it struggles to produce high-quality images due to the complexity of scattered light and the limitations of traditional image reconstruction algorithms. These algorithms can be affected by boundary conditions and have a low imaging accuracy, a shallow imaging depth, a long computation time, and a high signal-to-noise ratio. However, machine learning can potentially improve the performance of DOT by being better equipped to solve inverse problems, perform regression, classify medical images, and reconstruct biomedical images. In this study, we utilized a machine learning model called "XGBoost" to detect tumors in inhomogeneous breasts and applied a post-processing technique based on genetic programming to improve accuracy. The proposed algorithm was tested using simulated DOT measurements from complex inhomogeneous breasts and evaluated using the cosine similarity metrics and root mean square error loss. The results showed that the use of XGBoost and genetic programming in DOT could lead to more accurate and non-invasive detection of tumors in inhomogeneous breasts compared to traditional methods, with the reconstructed breasts having an average cosine similarity of more than 0.97 ± 0.07 and average root mean square error of around 0.1270 ± 0.0031 compared to the ground truth.

Extracting Tissue Optical Properties and Detecting Bruised Tissue in Pears Quickly and Accurately Based on Spatial Frequency Domain Imaging and Machine Learning

Recently, Spatial Frequency Domain Imaging (SFDI) has gradually become an alternative method to extract tissue optical properties (OPs), as it provides a wide-field, no-contact acquisition. SFDI extracts OPs by least-square fitting (LSF) based on the diffuse approximation equation, but there are shortcomings in the speed and accuracy of extracting OPs. This study proposed a Long Short-term Memory Regressor (LSTMR) solution to extract tissue OPs. This method allows for fast and accurate extraction of tissue OPs. Firstly, the imaging system was developed, which is more compact and portable than conventional SFDI systems. Next, numerical simulation was performed using the Monte Carlo forward model to obtain the dataset, and then the mapping model was established using the dataset. Finally, the model was applied to detect the bruised tissue of 'crown' pears. The results show that the mean absolute errors of the absorption coefficient and the reduced scattering coefficient are no more than 0.32% and 0.21%, and the bruised tissue of 'crown' pears can be highlighted by the change of OPs. Compared with the LSF, the speed of extracting tissue OPs is improved by two orders of magnitude, and the accuracy is greatly improved. The study contributes to the rapid and accurate extraction of tissue OPs based on SFDI and has great potential in food safety assessment.

Training generative adversarial networks for optical property mapping using synthetic image data

We demonstrate the training of a generative adversarial network (GAN) for the prediction of optical property maps (scattering and absorption) using spatial frequency domain imaging (SFDI) image data sets that are generated synthetically with a free open-source 3D modelling and rendering software, Blender. The flexibility of Blender is exploited to simulate 5 models with real-life relevance to clinical SFDI of diseased tissue: flat samples containing a single material, flat samples containing 2 materials, flat samples containing 3 materials, flat samples with spheroidal tumours and cylindrical samples with spheroidal tumours. The last case is particularly relevant as it represents wide-field imaging inside a tubular organ e.g. the gastro-intestinal tract. In all 5 scenarios we show the GAN provides an accurate reconstruction of the optical properties from single SFDI images with a mean normalised error ranging from 1.0-1.2% for absorption and 1.1%-1.2% for scattering, resulting in visually improved contrast for tumour spheroid structures. This compares favourably with the ∼10% absorption error and ∼10% scattering error achieved using GANs on experimental SFDI data. Next, we perform a bi-directional cross-validation of our synthetically-trained GAN, retrained with 90% synthetic and 10% experimental data to encourage domain transfer, with a GAN trained fully on experimental data and observe visually accurate results with an error of 6.3%-10.3% for absorption and 6.6%-11.9% for scattering. Our synthetically trained GAN is therefore highly relevant to real experimental samples but provides the significant added benefits of large training datasets, perfect ground-truths and the ability to test realistic imaging geometries, e.g. inside cylinders, for which no conventional single-shot demodulation algorithms exist. In the future, we expect that the application of techniques such as domain adaptation or training on hybrid real-synthetic datasets will create a powerful tool for fast, accurate production of optical property maps for real clinical imaging systems.

Ultracompact Deep Neural Network for Ultrafast Optical Property Extraction in Spatial Frequency Domain Imaging (SFDI)

Spatial frequency domain imaging (SFDI) is a powerful, label-free imaging technique capable of the wide-field quantitative mapping of tissue optical properties and, subsequently, chromophore concentrations. While SFDI hardware acquisition methods have advanced towards video-rate, the inverse problem (i.e., the mapping of acquired diffuse reflectance to optical properties) has remained a bottleneck for real-time data processing and visualization. Deep learning methods are adept at fitting nonlinear patterns, and may be ideal for rapidly solving the SFDI inverse problem. While current deep neural networks (DNN) are growing increasingly larger and more complex (e.g., with millions of parameters or more), our study shows that it can also be beneficial to move in the other direction, i.e., make DNNs that are smaller and simpler. Here, we propose an ultracompact, two-layer, fully connected DNN structure (each layer with four and two neurons, respectively) for ultrafast optical property extractions, which is 30×–600× faster than current methods with a similar or improved accuracy, allowing for an inversion time of 5.5 ms for 696 × 520 pixels. We further demonstrated the proposed inverse model in numerical simulations, and comprehensive phantom characterization, as well as offering in vivo measurements of dynamic physiological processes. We further demonstrated that the computation time could achieve another 200× improvement with a GPU device. This deep learning structure will help to enable fast and accurate real-time SFDI measurements, which are crucial for pre-clinical, clinical, and industrial applications.

Phase function estimation from a diffuse optical image via deep learning

Objective. The phase function is a key element of a light propagation model for Monte Carlo (MC) simulation, which is usually fitted with an analytic function with associated parameters. In recent years, machine learning methods were reported to estimate the parameters of the phase function of a particular form such as the Henyey–Greenstein phase function but, to our knowledge, no studies have been performed to determine the form of the phase function. Approach. Here we design a convolutional neural network (CNN) to estimate the phase function from a diffuse optical image without any explicit assumption on the form of the phase function. Specifically, we use a Gaussian mixture model (GMM) as an example to represent the phase function generally and learn the model parameters accurately. The GMM is selected because it provides the analytic expression of phase function to facilitate deflection angle sampling in MC simulation, and does not significantly increase the number of free parameters. Main Results. Our proposed method is validated on MC-simulated reflectance images of typical biological tissues using the Henyey–Greenstein phase function with different anisotropy factors. The mean squared error of the phase function is 0.01 and the relative error of the anisotropy factor is 3.28%. Significance. We propose the first data-driven CNN-based inverse MC model to estimate the form of scattering phase function. The effects of field of view and spatial resolution are analyzed and the findings provide guidelines for optimizing the experimental protocol in practical applications.

Deep learning in macroscopic diffuse optical imaging

SIGNIFICANCE

Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis.

AIM

We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI).

APPROACH

First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography.

RESULTS

The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships.

CONCLUSIONS

The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.

Spatial frequency domain imaging technology based on Fourier single-pixel imaging

SIGNIFICANCE

Optical properties (absorption coefficient and scattering coefficient) of tissue are the most critical parameters for disease diagnosis-based optical method. In recent years, researchers proposed spatial frequency domain imaging (SFDI) to quantitatively map tissue optical properties in a broad field of contactless imaging. To solve the limitations in wavebands unsuitable for silicon-based sensor technology, a compressed sensing (CS) algorithm is used to reproduce the original signal by a single-pixel detectors. Currently, the existing single-pixel SFDI method mainly uses a random sampling policy to extract and recover signals in the acquisition stage. However, these methods are memory-hungry and time-consuming, and they cannot generate discernible results under low sampling rate. Explorations on high performance and efficiency single-pixel SFDI are of great significance for clinical application.

AIM

Fourier single-pixel imaging can reconstruct signals with less time and space costs and has fewer reconstruction errors. We focus on an SFDI algorithm based on Fourier single-pixel imaging and propose our Fourier single-pixel image-based spatial frequency domain imaging method (FSI-SFDI).

APPROACH

First, we use Fourier single-pixel imaging algorithm to collect and compress signals and SFDI algorithm to generate optical parameters. Given the basis that the main energy of general image signals is concentrated in the range of low frequency of Fourier frequency domain, our FSI-SFDI uses a circular-sampling scheme to sample data points in the low-frequency region. Then, we reconstruct the image details from these points by optimization-based inverse-FFT method.

RESULTS

Our algorithm is tested on simulated data. Results show that the root mean square error (RMSE) of optical parameters is lower than 5% when the data reduction is 92%, and it can generate discernible optical parameter image with low sampling rate. We can observe that our FSI-SFDI primarily recovers the optical properties while keeping the RMSE under the upper bound of 4.5% when we use an image with 512  ×  512 resolution as the example for calculation and analysis. Not only that but also our algorithm consumes less space and time for an image with 256  ×  256 resolution, the signal reconstruction takes only 1.65 ms, and requires less RAM memory. Compared to CS-SFDI method, our FSI-SFDI can reduce the required number of measurements through optimizing algorithm.

CONCLUSIONS

Moreover, FSI-SFDI is capable of recovering high-quality resolvable images with lower sampling rate, higher-resolution images with less memory and time consumed than previous CS-SFDI method, which is very promising for clinical data collection and medical analysis.

Halftone spatial frequency domain imaging enables kilohertz high-speed label-free non-contact quantitative mapping of optical properties for strongly turbid media

The ability to quantify optical properties (i.e., absorption and scattering) of strongly turbid media has major implications on the characterization of biological tissues, fluid fields, and many others. However, there are few methods that can provide wide-field quantification of optical properties, and none is able to perform quantitative optical property imaging with high-speed (e.g., kilohertz) capabilities. Here we develop a new imaging modality termed halftone spatial frequency domain imaging (halftone-SFDI), which is approximately two orders of magnitude faster than the state-of-the-art, and provides kilohertz high-speed, label-free, non-contact, wide-field quantification for the optical properties of strongly turbid media. This method utilizes halftone binary patterned illumination to target the spatial frequency response of turbid media, which is then mapped to optical properties using model-based analysis. We validate the halftone-SFDI on an array of phantoms with a wide range of optical properties as well as in vivo human tissue. We demonstrate with an in vivo rat brain cortex imaging study, and show that halftone-SFDI can longitudinally monitor the absolute concentration as well as spatial distribution of functional chromophores in tissue. We also show that halftone-SFDI can spatially map dual-wavelength optical properties of a highly dynamic flow field at kilohertz speed. Together, these results highlight the potential of halftone-SFDI to enable new capabilities in fundamental research and translational studies including brain science and fluid dynamics.

Developing diagnostic assessment of breast lumpectomy tissues using radiomic and optical signatures

High positive margin rates in oncologic breast-conserving surgery are a pressing clinical problem. Volumetric X-ray scanning is emerging as a powerful ex vivo specimen imaging technique for analyzing resection margins, but X-rays lack contrast between non-malignant and malignant fibrous tissues. In this study, combined micro-CT and wide-field optical image radiomics were developed to classify malignancy of breast cancer tissues, demonstrating that X-ray/optical radiomics improve malignancy classification. Ninety-two standardized features were extracted from co-registered micro-CT and optical spatial frequency domain imaging samples extracted from 54 breast tumors exhibiting seven tissue subtypes confirmed by microscopic histological analysis. Multimodal feature sets improved classification performance versus micro-CT alone when adipose samples were included (AUC = 0.88 vs. 0.90; p-value = 3.65e-11) and excluded, focusing the classification task on exclusively non-malignant fibrous versus malignant tissues (AUC = 0.78 vs. 0.85; p-value = 9.33e-14). Extending the radiomics approach to high-dimensional optical data-termed "optomics" in this study-offers a promising optical image analysis technique for cancer detection. Radiomic feature data and classification source code are publicly available.

In vivo two-dimensional quantitative imaging of skin and cutaneous microcirculation with perturbative spatial frequency domain imaging (p-SFDI)

We introduce perturbative spatial frequency domain imaging (p-SFDI) for fast two-dimensional (2D) mapping of the optical properties and physiological characteristics of skin and cutaneous microcirculation using spatially modulated visible light. Compared to the traditional methods for recovering 2D maps through a pixel-by-pixel inversion, p-SFDI significantly shortens parameter retrieval time, largely avoids the random fitting errors caused by measurement noise, and enhances the image reconstruction quality. The efficacy of p-SFDI is demonstrated by in vivo imaging forearm of one healthy subject, recovering the 2D spatial distribution of cutaneous hemoglobin concentration, oxygen saturation, scattering properties, the melanin content, and the epidermal thickness over a large field of view. Furthermore, the temporal and spatial variations in physiological parameters under the forearm reactive hyperemia protocol are revealed, showing its applications in monitoring temporal and spatial dynamics.

Deep learning-based method to accurately estimate breast tissue optical properties in the presence of the chest wall

SIGNIFICANCE

In general, image reconstruction methods used in diffuse optical tomography (DOT) are based on diffusion approximation, and they consider the breast tissue as a homogenous, semi-infinite medium. However, the semi-infinite medium assumption used in DOT reconstruction is not valid when the chest wall is underneath the breast tissue.

AIM

We aim to reduce the chest wall's effect on the estimated average optical properties of breast tissue and obtain accurate forward model for DOT reconstruction.

APPROACH

We propose a deep learning-based neural network approach where a convolution neural network (CNN) is trained to simultaneously obtain accurate optical property values for both the breast tissue and the chest wall.

RESULTS

The CNN model shows great promise in reducing errors in estimating the optical properties of the breast tissue in the presence of a shallow chest wall. For patient data, the CNN model predicted the breast tissue optical absorption coefficient, which was independent of chest wall depth.

CONCLUSIONS

Our proposed method can be readily used in DOT and diffuse spectroscopy measurements to improve the accuracy of estimated tissue optical properties.

Imaging sub-diffuse optical properties of cancerous and normal skin tissue using machine learning-aided spatial frequency domain imaging

SIGNIFICANCE

Sub-diffuse optical properties may serve as useful cancer biomarkers, and wide-field heatmaps of these properties could aid physicians in identifying cancerous tissue. Sub-diffuse spatial frequency domain imaging (sd-SFDI) can reveal such wide-field maps, but the current time cost of experimentally validated methods for rendering these heatmaps precludes this technology from potential real-time applications.

AIM

Our study renders heatmaps of sub-diffuse optical properties from experimental sd-SFDI images in real time and reports these properties for cancerous and normal skin tissue subtypes.

APPROACH

A phase function sampling method was used to simulate sd-SFDI spectra over a wide range of optical properties. A machine learning model trained on these simulations and tested on tissue phantoms was used to render sub-diffuse optical property heatmaps from sd-SFDI images of cancerous and normal skin tissue.

RESULTS

The model accurately rendered heatmaps from experimental sd-SFDI images in real time. In addition, heatmaps of a small number of tissue samples are presented to inform hypotheses on sub-diffuse optical property differences across skin tissue subtypes.

CONCLUSION

These results bring the overall process of sd-SFDI a fundamental step closer to real-time speeds and set a foundation for future real-time medical applications of sd-SFDI such as image guided surgery.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Modeling and Synthesis of Breast Cancer Optical Property Signatures With Generative Models

Is it possible to find deterministic relationships between optical measurements and pathophysiology in an unsupervised manner and based on data alone? Optical property quantification is a rapidly growing biomedical imaging technique for characterizing biological tissues that shows promise in a range of clinical applications, such as intraoperative breast-conserving surgery margin assessment. However, translating tissue optical properties to clinical pathology information is still a cumbersome problem due to, amongst other things, inter- and intrapatient variability, calibration, and ultimately the nonlinear behavior of light in turbid media. These challenges limit the ability of standard statistical methods to generate a simple model of pathology, requiring more advanced algorithms. We present a data-driven, nonlinear model of breast cancer pathology for real-time margin assessment of resected samples using optical properties derived from spatial frequency domain imaging data. A series of deep neural network models are employed to obtain sets of latent embeddings that relate optical data signatures to the underlying tissue pathology in a tractable manner. These self-explanatory models can translate absorption and scattering properties measured from pathology, while also being able to synthesize new data. The method was tested on a total of 70 resected breast tissue samples containing 137 regions of interest, achieving rapid optical property modeling with errors only limited by current semi-empirical models, allowing for mass sample synthesis and providing a systematic understanding of dataset properties, paving the way for deep automated margin assessment algorithms using structured light imaging or, in principle, any other optical imaging technique seeking modeling. Code is available.

Profile-based intensity and frequency corrections for single-snapshot spatial frequency domain imaging

We have proposed the profile-based intensity and frequency corrections for single-snapshot spatial frequency domain (SFD) imaging to mitigate surface profile effects on the measured intensity and spatial frequency in extracting the optical properties. In the scheme, the spatially modulated frequency of the projected sinusoidal pattern is adaptively adjusted according to the sample surface profile, reducing distortions of the modulation amplitude in the single-snapshot demodulation and errors in the optical property extraction. The profile effects on both the measured intensities of light incident onto and reflected from the sample are then compensated using Minnaert's correction to obtain the true diffuse reflectance of the sample. We have validated the method by phantom experiments using a highly sensitive SFD imaging system based on the single-pixel photon-counting detection and assessed error reductions in extracting the absorption and reduced scattering coefficients by an average of 40% and 10%, respectively. Further, an in vivo topography experiment of the opisthenar vessels has demonstrated its clinical feasibility.

Gradient-assisted focusing light through scattering media

Focusing light through a scattering medium is a longstanding challenge in biomedical optics, to which wavefront shaping is a powerful solution. The state-of-the-art feedback-based approach is the widely used genetic algorithm method. However, it can only achieve relatively low enhancement of the focus, and the genetic algorithm is known to be time-consuming. To tackle those issues, we propose a gradient-assisted strategy for wavefront shaping. The proposed method conducts optimization in the function distribution space. Specifically, when optimizing the parameters along each iteration, the consequent function distribution changes within a distance as measured by the Kullback-Leibler divergence. Taking advantage of the gradient information, the proposed method is over 60× faster to obtain the same peak-to-background ratio (PBR) level. Compared with the genetic algorithm that is able to optimize a number of 64×64 phase segments, the proposed gradient strategy is able to optimize 256×256 phase segments, and gives 20× higher focus enhancement as quantified by the PBR.

Speckle illumination SFDI for projector-free optical property mapping

Spatial frequency domain imaging can map tissue scattering and absorption properties over a wide field of view, making it useful for clinical applications such as wound assessment and surgical guidance. This technique has previously required the projection of fully characterized illumination patterns. Here, we show that random and unknown speckle illumination can be used to sample the modulation transfer function of tissues at known spatial frequencies, allowing the quantitative mapping of optical properties with simple laser diode illumination. We compute low- and high-spatial frequency response parameters from the local power spectral density for each pixel and use a lookup table to accurately estimate absorption and scattering coefficients in tissue phantoms, in vivo human hand, and ex vivo swine esophagus. Because speckle patterns can be generated over a large depth of field and field of view with simple coherent illumination, this approach may enable optical property mapping in new form-factors and applications, including endoscopy.

Direct mapping from diffuse reflectance to chromophore concentrations in multi-fx spatial frequency domain imaging (SFDI) with a deep residual network (DRN)

Spatial frequency domain imaging (SFDI) is an emerging technology that enables label-free, non-contact, and wide-field mapping of tissue chromophore contents, such as oxy- and deoxy-hemoglobin concentrations. It has been shown that the use of more than two spatial frequencies (multi-fx ) can vastly improve measurement accuracy and reduce chromophore estimation uncertainties, but real-time multi-fx SFDI for chromophore monitoring has been limited in practice due to the slow speed of available chromophore inversion algorithms. Existing inversion algorithms have to first convert the multi-fx diffuse reflectance to optical absorptions, and then solve a set of linear equations to estimate chromophore concentrations. In this work, we present a deep learning framework, noted as a deep residual network (DRN), that is able to directly map from diffuse reflectance to chromophore concentrations. The proposed DRN is over 10x faster than the state-of-the-art method for chromophore inversion and enables 25x improvement on the frame rate for in vivo real-time oxygenation mapping. The proposed deep learning model will help enable real-time and highly accurate chromophore monitoring with multi-fx SFDI.

Real-time, wide-field and high-quality single snapshot imaging of optical properties with profile correction using deep learning

The development of real-time, wide-field and quantitative diffuse optical imaging methods to visualize functional and structural biomarkers of living tissues is a pressing need for numerous clinical applications including image-guided surgery. In this context, Spatial Frequency Domain Imaging (SFDI) is an attractive method allowing for the fast estimation of optical properties using the Single Snapshot of Optical Properties (SSOP) approach. Herein, we present a novel implementation of SSOP based on a combination of deep learning network at the filtering stage and Graphics Processing Units (GPU) capable of simultaneous high visual quality image reconstruction, surface profile correction and accurate optical property (OP) extraction in real-time across large fields of view. In the most optimal implementation, the presented methodology demonstrates megapixel profile-corrected OP imaging with results comparable to that of profile-corrected SFDI, with a processing time of 18 ms and errors relative to SFDI method less than 10% in both profilometry and profile-corrected OPs. This novel processing framework lays the foundation for real-time multispectral quantitative diffuse optical imaging for surgical guidance and healthcare applications. All code and data used for this work is publicly available at www.healthphotonics.org under the resources tab.

Deep learning model for ultrafast quantification of blood flow in diffuse correlation spectroscopy

Diffuse correlation spectroscopy (DCS) is increasingly used in the optical imaging field to assess blood flow in humans due to its non-invasive, real-time characteristics and its ability to provide label-free, bedside monitoring of blood flow changes. Previous DCS studies have utilized a traditional curve fitting of the analytical or Monte Carlo models to extract the blood flow changes, which are computationally demanding and less accurate when the signal to noise ratio decreases. Here, we present a deep learning model that eliminates this bottleneck by solving the inverse problem more than 2300% faster, with equivalent or improved accuracy compared to the nonlinear fitting with an analytical method. The proposed deep learning inverse model will enable real-time and accurate tissue blood flow quantification with the DCS technique.

Adaptive Boosting (AdaBoost)-based multiwavelength spatial frequency domain imaging and characterization for ex vivo human colorectal tissue assessment

The current gold standard diagnostic test for colorectal cancer remains histological inspections of endoluminal neoplasia in biopsy specimens. However, biopsy site selection requires visual inspection of the bowel, typically with a white-light endoscope. Therefore, this technique is poorly suited to detect small or innocuous-appearing lesions. We hypothesize that an alternative modality-multiwavelength spatial frequency domain imaging (SFDI)-would be able to differentiate various colorectal neoplasia from normal tissue. In this ex vivo study of human colorectal tissues, we report the optical absorption and scattering signatures of normal, adenomatous polyp and cancer specimens. An abnormal vs. normal adaptive boosting (AdaBoost) classifier is trained to dichotomize tissue based on SFDI imaging characteristics, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.95 is achieved. We conclude that AdaBoost-based multiwavelength SFDI can differentiate abnormal from normal colorectal tissues, potentially improving endoluminal screening of the distal gastrointestinal tract in the future.

GANPOP: Generative Adversarial Network Prediction of Optical Properties From Single Snapshot Wide-Field Images

We present a deep learning framework for wide-field, content-aware estimation of absorption and scattering coefficients of tissues, called Generative Adversarial Network Prediction of Optical Properties (GANPOP). Spatial frequency domain imaging is used to obtain ground-truth optical properties at 660 nm from in vivo human hands and feet, freshly resected human esophagectomy samples, and homogeneous tissue phantoms. Images of objects with either flat-field or structured illumination are paired with registered optical property maps and are used to train conditional generative adversarial networks that estimate optical properties from a single input image. We benchmark this approach by comparing GANPOP to a single-snapshot optical property (SSOP) technique, using a normalized mean absolute error (NMAE) metric. In human gastrointestinal specimens, GANPOP with a single structured-light input image estimates the reduced scattering and absorption coefficients with 60% higher accuracy than SSOP while GANPOP with a single flat-field illumination image achieves similar accuracy to SSOP. When applied to both in vivo and ex vivo swine tissues, a GANPOP model trained solely on structured-illumination images of human specimens and phantoms estimates optical properties with approximately 46% improvement over SSOP, indicating adaptability to new, unseen tissue types. Given a training set that appropriately spans the target domain, GANPOP has the potential to enable rapid and accurate wide-field measurements of optical properties.

Real-time optical properties and oxygenation imaging using custom parallel processing in the spatial frequency domain

The development of real-time, wide-field and quantitative diffuse optical imaging methods is becoming increasingly popular for biological and medical applications. Recent developments introduced a novel approach for real-time multispectral acquisition in the spatial frequency domain using spatio-temporal modulation of light. Using this method, optical properties maps (absorption and reduced scattering) could be obtained for two wavelengths (665 nm and 860 nm). These maps, in turn, are used to deduce oxygen saturation levels in tissues. However, while the acquisition was performed in real-time, processing was performed post-acquisition and was not in real-time. In the present article, we present CPU and GPU processing implementations for this method with special emphasis on processing time. The obtained results show that the proposed custom direct method using a General Purpose Graphic Processing Unit (GPGPU) and C CUDA (Compute Unified Device Architecture) implementation enables 1.6 milliseconds processing time for a 1 Mega-pixel image with a maximum average error of 0.1% in extracting optical properties.

Spatial frequency domain imaging in 2019: principles, applications, and perspectives

Spatial frequency domain imaging (SFDI) has witnessed very rapid growth over the last decade, owing to its unique capabilities for imaging optical properties and chromophores over a large field-of-view and in a rapid manner. We provide a comprehensive review of the principles of this imaging method as of 2019, review the modeling of light propagation in this domain, describe acquisition methods, provide an understanding of the various implementations and their practical limitations, and finally review applications that have been published in the literature. Importantly, we also introduce a group effort by several key actors in the field for the dissemination of SFDI, including publications, advice in hardware and implementations, and processing code, all freely available online.

Spatial frequency domain imaging in 2019: principles, applications, and perspectives

Spatial frequency domain imaging (SFDI) has witnessed very rapid growth over the last decade, owing to its unique capabilities for imaging optical properties and chromophores over a large field-of-view and in a rapid manner. We provide a comprehensive review of the principles of this imaging method as of 2019, review the modeling of light propagation in this domain, describe acquisition methods, provide an understanding of the various implementations and their practical limitations, and finally review applications that have been published in the literature. Importantly, we also introduce a group effort by several key actors in the field for the dissemination of SFDI, including publications, advice in hardware and implementations, and processing code, all freely available online.

Burn wound classification model using spatial frequency-domain imaging and machine learning

Accurate assessment of burn severity is critical for wound care and the course of treatment. Delays in classification translate to delays in burn management, increasing the risk of scarring and infection. To this end, numerous imaging techniques have been used to examine tissue properties to infer burn severity. Spatial frequency-domain imaging (SFDI) has also been used to characterize burns based on the relationships between histologic observations and changes in tissue properties. Recently, machine learning has been used to classify burns by combining optical features from multispectral or hyperspectral imaging. Rather than employ models of light propagation to deduce tissue optical properties, we investigated the feasibility of using SFDI reflectance data at multiple spatial frequencies, with a support vector machine (SVM) classifier, to predict severity in a porcine model of graded burns. Calibrated reflectance images were collected using SFDI at eight wavelengths (471 to 851 nm) and five spatial frequencies (0 to 0.2  mm  -  1). Three models were built from subsets of this initial dataset. The first subset included data taken at all wavelengths with the planar (0  mm  -  1) spatial frequency, the second comprised data at all wavelengths and spatial frequencies, and the third used all collected data at values relative to unburned tissue. These data subsets were used to train and test cubic SVM models, and compared against burn status 28 days after injury. Model accuracy was established through leave-one-out cross-validation testing. The model based on images obtained at all wavelengths and spatial frequencies predicted burn severity at 24 h with 92.5% accuracy. The model composed of all values relative to unburned skin was 94.4% accurate. By comparison, the model that employed only planar illumination was 88.8% accurate. This investigation suggests that the combination of SFDI with machine learning has potential for accurately predicting burn severity.

Real-time, wide-field, and quantitative oxygenation imaging using spatiotemporal modulation of light

Quantitative diffuse optical imaging has the potential to provide valuable functional information about tissue status, such as oxygen saturation or blood content to healthcare practitioners in real time. However, significant technical challenges have so far prevented such tools from being deployed in the clinic. Toward achieving this goal, prior research introduced methods based on spatial frequency domain imaging (SFDI) that allow real-time (within milliseconds) wide-field imaging of optical properties but at a single wavelength. However, for this technology to be useful to clinicians, images must be displayed in terms of metrics related to the physiological state of the tissue, hence interpretable to guide decision-making. For this purpose, recent developments introduced multispectral SFDI methods for rapid imaging of oxygenation parameters up to 16 frames per seconds (fps). We introduce real-time, wide-field, and quantitative blood parameters imaging using spatiotemporal modulation of light. Using this method, we are able to quantitatively obtain optical properties at 100 fps at two wavelengths (665 and 860 nm), and therefore oxygenation, oxyhemoglobin, and deoxyhemoglobin, using a single camera with, at most, 4.2% error in comparison with standard SFDI acquisitions.