GANPOP: Generative Adversarial Network Prediction of Optical Properties From Single Snapshot Wide-Field Images

Deep-learning-enabled spatial frequency domain imaging of the spatiotemporal dynamics of skin physiology

Significance

Spatial frequency domain imaging (SFDI) is an emerging optical imaging modality for visualizing tissue absorption and scattering properties. This approach is promising for noninvasive wide field-of-view (FOV) monitoring of biophysiological processes in vivo.

Aim

We aim to develop deep-learning-enabled spatial frequency domain imaging (SFDI-net) for real-time large FOV imaging of the optical, structural, and physiological properties and demonstrate its application for probing the spatiotemporal dynamics of skin physiology.

Approach

SFDI-net, based on mapping of a two-layer structure into an equivalent homogeneous medium for spatially modulated light and with a convolutional neural network architecture, produces two-dimensional maps of optical, structural, and physiological parameters for bilayered tissue, including cutaneous hemoglobin concentration, oxygen saturation, scattering properties (reduced scattering coefficient and scattering power), melanin content, surface roughness, and epidermal thickness, with visible spatially modulated light at the camera frame rate.

Results

Compared with traditional approaches, SFDI-net achieves a real-time inversion speed and significantly improves image quality by effectively suppressing noise while preserving tissue structure without oversmoothing. We demonstrate the application of the SFDI-net for monitoring the spatiotemporal dynamics of forearm skin physiology in reactive hyperemia and rhythmic respiration and reveal their intricate patterns in hemodynamics.

Conclusions

Deep-learning-enabled spatial frequency domain imaging and SFDI-net may offer insights into the cardiorespiratory system and have promising clinical utility for disease diagnosis, surveillance, and therapeutic assessment. Future hardware and software advancements will bring SFDI-net to clinical practice.

GMOPNet: A GAN-MLP two-stage network for optical properties measurement of kiwifruit and peaches with spatial frequency domain imaging

Spatial frequency domain imaging (SFDI) is an imaging technique using spatially modulated illumination for measurement of optical properties. Conventional SFDI methods require capturing at least six images, making it time-consuming. This study presents a Generative Adversarial Network-Multi-Layer Perceptron (GAN-MLP) two-stage network (GMOPNet) for extracting high-precision optical properties of kiwifruit and peaches from a single SFDI image, enabling real-time continuous wide-band SFDI. The GMOPNet we proposed leverages the GAN to predict diffuse reflectance, followed by the MLP with Monte Carlo prior knowledge to predict optical properties. Our method achieves mean absolute percentage errors (MAPE) of 5.91% for the absorption coefficient (μa) and 5.23% for the reduced scattering coefficient ( [Formula: see text] ), reducing acquisition and processing time significantly, with single inference taking 31.13 ms. The MAPE of the μa and the [Formula: see text] were 6.73% and 6.34% for kiwifruit and 5.80% and 6.65% for peaches, respectively.

Speckle-illumination spatial frequency domain imaging with a stereo laparoscope for profile-corrected optical property mapping

Significance

Laparoscopic surgery presents challenges in localizing oncological margins due to poor contrast between healthy and malignant tissues. Optical properties can uniquely identify various tissue types and disease states with high sensitivity and specificity, making it a promising tool for surgical guidance. Although spatial frequency domain imaging (SFDI) effectively measures quantitative optical properties, its deployment in laparoscopy is challenging due to the constrained imaging environment. Thus, there is a need for compact structured illumination techniques to enable accurate, quantitative endogenous contrast in minimally invasive surgery.

Aim

We introduce a compact, two-camera laparoscope that incorporates both active stereo depth estimation and speckle-illumination SFDI (si-SFDI) to map profile-corrected, pixel-level absorption (μ a ), and reduced scattering (μ s ' ) optical properties in images of tissues with complex geometries.

Approach

We used a multimode fiber-coupled 639-nm laser illumination to generate high-contrast speckle patterns on the object. These patterns were imaged through a modified commercial stereo laparoscope for optical property estimation via si-SFDI. Compared with the original si-SFDI work, which required ≥ 10 images of randomized speckle patterns for accurate optical property estimations, our approach approximates the DC response using a laser speckle reducer (LSR) and consequently requires only two images. In addition, we demonstrate 3D profilometry using active stereo from low-coherence RGB laser flood illumination. Sample topography was then used to correct for measured intensity variations caused by object height and surface angle differences with respect to a calibration phantom. The low-contrast RGB speckle pattern was blurred using an LSR to approximate incoherent white light illumination. We validated profile-corrected si-SFDI against conventional SFDI in phantoms with simple and complex geometries, as well as in a human finger in vivo time-series constriction study.

Results

Laparoscopic si-SFDI optical property measurements agreed with conventional SFDI measurements when measuring flat tissue phantoms, exhibiting an error of 6.4% for absorption and 5.8% for reduced scattering. Profile-correction improved the accuracy for measurements of phantoms with complex geometries, particularly for absorption, where it reduced the error by 23.7%. An in vivo finger constriction study further validated laparoscopic si-SFDI, demonstrating an error of 8.2% for absorption and 5.8% for reduced scattering compared with conventional SFDI. Moreover, the observed trends in optical properties due to physiological changes were consistent with previous studies.

Conclusions

Our stereo-laparoscopic implementation of si-SFDI provides a simple method to obtain accurate optical property maps through a laparoscope for flat and complex geometries. This has the potential to provide quantitative endogenous contrast for minimally invasive surgical guidance.

A Best-Fitting B-Spline Neural Network Approach to the Prediction of Advection-Diffusion Physical Fields with Absorption and Source Terms

This paper proposed a two-dimensional steady-state field prediction approach that combines B-spline functions and a fully connected neural network. In this approach, field data, which are determined by corresponding control vectors, are fitted by a selected B-spline function set, yielding the corresponding best-fitting weight vectors, and then a fully connected neural network is trained using those weight vectors and control vectors. The trained neural network first predicts a weight vector using a given control vector, and then the corresponding field can be restored via the selected B-spline set. This method was applied to learn and predict two-dimensional steady advection-diffusion physical fields with absorption and source terms, and its accuracy and performance were tested and verified by a series of numerical experiments with different B-spline sets, boundary conditions, field gradients, and field states. The proposed method was finally compared with a generative adversarial network (GAN) and a physics-informed neural network (PINN). The results indicated that the B-spline neural network could predict the tested physical fields well; the overall error can be reduced by expanding the selected B-spline set. Compared with GAN and PINN, the proposed method also presented the advantages of a high prediction accuracy, less demand for training data, and high training efficiency.

Deep-learning approach to stratified reconstructions of tissue absorption and scattering in time-domain spatial frequency domain imaging

Significance

The conventional optical properties (OPs) reconstruction in spatial frequency domain (SFD) imaging, like the lookup table (LUT) method, causes OPs aliasing and yields only average OPs without depth resolution. Integrating SFD imaging with time-resolved (TR) measurements enhances space-TR information, enabling improved reconstruction of absorption (μ a ) and reduced scattering (μ s ' ) coefficients at various depths.

Aim

To achieve the stratified reconstruction of OPs and the separation between μ a and μ s ' , using deep learning workflow based on the temporal and spatial information provided by time-domain SFD imaging technique, while enhancing the reconstruction accuracy.

Approach

Two data processing methods are employed for the OPs reconstruction with TR-SFD imaging, one is full TR data, and the other is the featured data extracted from the full TR data (E , continuous-wave component, ⟨ t ⟩ , mean time of flight). We compared their performance using a series of simulation and phantom validations.

Results

Compared to the LUT approach, utilizing full TR, E and ⟨ t ⟩ datasets yield high-resolution OPs reconstruction results. Among the three datasets employed, full TR demonstrates the optimal accuracy.

Conclusions

Utilizing the data obtained from SFD and TR measurement techniques allows for achieving high-resolution separation reconstruction of μ a and μ s ' at different depths within 5 mm.

Colonoscopy 3D video dataset with paired depth from 2D-3D registration

Screening colonoscopy is an important clinical application for several 3D computer vision techniques, including depth estimation, surface reconstruction, and missing region detection. However, the development, evaluation, and comparison of these techniques in real colonoscopy videos remain largely qualitative due to the difficulty of acquiring ground truth data. In this work, we present a Colonoscopy 3D Video Dataset (C3VD) acquired with a high definition clinical colonoscope and high-fidelity colon models for benchmarking computer vision methods in colonoscopy. We introduce a novel multimodal 2D-3D registration technique to register optical video sequences with ground truth rendered views of a known 3D model. The different modalities are registered by transforming optical images to depth maps with a Generative Adversarial Network and aligning edge features with an evolutionary optimizer. This registration method achieves an average translation error of 0.321 millimeters and an average rotation error of 0.159 degrees in simulation experiments where error-free ground truth is available. The method also leverages video information, improving registration accuracy by 55.6% for translation and 60.4% for rotation compared to single frame registration. 22 short video sequences were registered to generate 10,015 total frames with paired ground truth depth, surface normals, optical flow, occlusion, six degree-of-freedom pose, coverage maps, and 3D models. The dataset also includes screening videos acquired by a gastroenterologist with paired ground truth pose and 3D surface models. The dataset and registration source code are available at https://durr.jhu.edu/C3VD.

Applications of optical property measurement for quality evaluation of agri-food products: a review

Spectroscopic techniques coupled with chemometric approaches have been widely used for quality evaluation of agricultural and food (agri-food) products due to the nondestructive, simple, fast, and easy characters. However, these techniques face the issues or challenges of relatively weak robustness, generalizability, and applicability in modeling and prediction because they measure the aggregate amount of light interaction with tissues, resulting in the combined effect of absorption and scattering of photons. Optical property measurement could separate absorption from scattering, providing new insights into more reliable prediction performance in quality evaluation, which is attracting increasing attention. In this review, a brief overview of the currently popular measurement techniques, in terms of light transfer principles and data analysis algorithms, is first presented. Then, the emphases are put on the recent advances of these techniques for measuring optical properties of agri-food products since 2000. Corresponding applications on qualitative and quantitative analyses of quality evaluation, as well as light transfer simulations within tissues, were reviewed. Furthermore, the leading groups working on optical property measurement worldwide are highlighted, which is the first summary to the best of our knowledge. Finally, challenges for optical property measurement are discussed, and some viewpoints on future research directions are also given.

Optimal Hyperspectral Band Selection for Tissue Oxygenation Mapping with Generative Adversarial Network

Tissue oxygenation assessment using hyperspectral imaging is an emerging technique for the diagnosis and pre- and post-treatment monitoring of ischemic patients. However, the high spectral resolution of hyperspectral imaging leads to large data sizes and a long imaging time. In this study, we propose a method that utilizes multi-objective evolutionary algorithms to determine the optimal hyperspectral band combination when developing a deep learning model for predicting tissue oxygenation from hyperspectral images. Our results confirm that the deep learning model effectively predicts tissue oxygenation images for various oxygenation states. Moreover, we demonstrate that a high-performance prediction model can be developed using only a small number of spectral bands, indicating the potential for more efficient non-contact tissue oxygenation mapping with the proposed method.Clinical Relevance- The proposed method allows for the non-contact and efficient acquisition of two-dimensional tissue oxygenation information in various oxygenation states.

Designing and simulating realistic spatial frequency domain imaging systems using open-source 3D rendering software

Spatial frequency domain imaging (SFDI) is a low-cost imaging technique that maps absorption and reduced scattering coefficients, offering improved contrast for important tissue structures such as tumours. Practical SFDI systems must cope with various imaging geometries including imaging planar samples ex vivo, imaging inside tubular lumen in vivo e.g. for endoscopy, and measuring tumours or polyps of varying morphology. There is a need for a design and simulation tool to accelerate design of new SFDI systems and simulate realistic performance under these scenarios. We present such a system implemented using open-source 3D design and ray-tracing software Blender that simulates media with realistic absorption and scattering in a wide range of geometries. By using Blender's Cycles ray-tracing engine, our system simulates effects such as varying lighting, refractive index changes, non-normal incidence, specular reflections and shadows, enabling realistic evaluation of new designs. We first demonstrate quantitative agreement between Monte-Carlo simulated absorption and reduced scattering coefficients with those simulated from our Blender system, achieving 16 % discrepancy in absorption coefficient and 18 % in reduced scattering coefficient. However, we then show that using an empirically derived look-up table the errors reduce to 1 % and 0.7 % respectively. Next, we simulate SFDI mapping of absorption, scattering and shape for simulated tumour spheroids, demonstrating enhanced contrast. Finally we demonstrate SFDI mapping inside a tubular lumen, which highlighted a important design insight: custom look-up tables must be generated for different longitudinal sections of the lumen. With this approach we achieved 2 % absorption error and 2 % scattering error. We anticipate our simulation system will aid in the design of novel SFDI systems for key biomedical applications.

Unrolled-DOT: an interpretable deep network for diffuse optical tomography

Significance

Imaging through scattering media is critical in many biomedical imaging applications, such as breast tumor detection and functional neuroimaging. Time-of-flight diffuse optical tomography (ToF-DOT) is one of the most promising methods for high-resolution imaging through scattering media. ToF-DOT and many traditional DOT methods require an image reconstruction algorithm. Unfortunately, this algorithm often requires long computational runtimes and may produce lower quality reconstructions in the presence of model mismatch or improper hyperparameter tuning.

Aim

We used a data-driven unrolled network as our ToF-DOT inverse solver. The unrolled network is faster than traditional inverse solvers and achieves higher reconstruction quality by accounting for model mismatch.

Approach

Our model "Unrolled-DOT" uses the learned iterative shrinkage thresholding algorithm. In addition, we incorporate a refinement U-Net and Visual Geometry Group (VGG) perceptual loss to further increase the reconstruction quality. We trained and tested our model on simulated and real-world data and benchmarked against physics-based and learning-based inverse solvers.

Results

In experiments on real-world data, Unrolled-DOT outperformed learning-based algorithms and achieved over 10× reduction in runtime and mean-squared error, compared to traditional physics-based solvers.

Conclusion

We demonstrated a learning-based ToF-DOT inverse solver that achieves state-of-the-art performance in speed and reconstruction quality, which can aid in future applications for noninvasive biomedical imaging.

Training generative adversarial networks for optical property mapping using synthetic image data

We demonstrate the training of a generative adversarial network (GAN) for the prediction of optical property maps (scattering and absorption) using spatial frequency domain imaging (SFDI) image data sets that are generated synthetically with a free open-source 3D modelling and rendering software, Blender. The flexibility of Blender is exploited to simulate 5 models with real-life relevance to clinical SFDI of diseased tissue: flat samples containing a single material, flat samples containing 2 materials, flat samples containing 3 materials, flat samples with spheroidal tumours and cylindrical samples with spheroidal tumours. The last case is particularly relevant as it represents wide-field imaging inside a tubular organ e.g. the gastro-intestinal tract. In all 5 scenarios we show the GAN provides an accurate reconstruction of the optical properties from single SFDI images with a mean normalised error ranging from 1.0-1.2% for absorption and 1.1%-1.2% for scattering, resulting in visually improved contrast for tumour spheroid structures. This compares favourably with the ∼10% absorption error and ∼10% scattering error achieved using GANs on experimental SFDI data. Next, we perform a bi-directional cross-validation of our synthetically-trained GAN, retrained with 90% synthetic and 10% experimental data to encourage domain transfer, with a GAN trained fully on experimental data and observe visually accurate results with an error of 6.3%-10.3% for absorption and 6.6%-11.9% for scattering. Our synthetically trained GAN is therefore highly relevant to real experimental samples but provides the significant added benefits of large training datasets, perfect ground-truths and the ability to test realistic imaging geometries, e.g. inside cylinders, for which no conventional single-shot demodulation algorithms exist. In the future, we expect that the application of techniques such as domain adaptation or training on hybrid real-synthetic datasets will create a powerful tool for fast, accurate production of optical property maps for real clinical imaging systems.

Optical clearing and testing of lung tissue using inhalation aerosols: prospects for monitoring the action of viral infections

Optical clearing of the lung tissue aims to make it more transparent to light by minimizing light scattering, thus allowing reconstruction of the three-dimensional structure of the tissue with a much better resolution. This is of great importance for monitoring of viral infection impact on the alveolar structure of the tissue and oxygen transport. Optical clearing agents (OCAs) can provide not only lesser light scattering of tissue components but also may influence the molecular transport function of the alveolar membrane. Air-filled lungs present significant challenges for optical imaging including optical coherence tomography (OCT), confocal and two-photon microscopy, and Raman spectroscopy, because of the large refractive-index mismatch between alveoli walls and the enclosed air-filled region. During OCT imaging, the light is strongly backscattered at each air–tissue interface, such that image reconstruction is typically limited to a single alveolus. At the same time, the filling of these cavities with an OCA, to which water (physiological solution) can also be attributed since its refractive index is much higher than that of air will lead to much better tissue optical transmittance. This review presents general principles and advances in the field of tissue optical clearing (TOC) technology, OCA delivery mechanisms in lung tissue, studies of the impact of microbial and viral infections on tissue response, and antimicrobial and antiviral photodynamic therapies using methylene blue (MB) and indocyanine green (ICG) dyes as photosensitizers.

Phase function estimation from a diffuse optical image via deep learning

Objective. The phase function is a key element of a light propagation model for Monte Carlo (MC) simulation, which is usually fitted with an analytic function with associated parameters. In recent years, machine learning methods were reported to estimate the parameters of the phase function of a particular form such as the Henyey–Greenstein phase function but, to our knowledge, no studies have been performed to determine the form of the phase function. Approach. Here we design a convolutional neural network (CNN) to estimate the phase function from a diffuse optical image without any explicit assumption on the form of the phase function. Specifically, we use a Gaussian mixture model (GMM) as an example to represent the phase function generally and learn the model parameters accurately. The GMM is selected because it provides the analytic expression of phase function to facilitate deflection angle sampling in MC simulation, and does not significantly increase the number of free parameters. Main Results. Our proposed method is validated on MC-simulated reflectance images of typical biological tissues using the Henyey–Greenstein phase function with different anisotropy factors. The mean squared error of the phase function is 0.01 and the relative error of the anisotropy factor is 3.28%. Significance. We propose the first data-driven CNN-based inverse MC model to estimate the form of scattering phase function. The effects of field of view and spatial resolution are analyzed and the findings provide guidelines for optimizing the experimental protocol in practical applications.

Deep learning in macroscopic diffuse optical imaging

SIGNIFICANCE

Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis.

AIM

We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI).

APPROACH

First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography.

RESULTS

The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships.

CONCLUSIONS

The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.

Spatial frequency domain imaging technology based on Fourier single-pixel imaging

SIGNIFICANCE

Optical properties (absorption coefficient and scattering coefficient) of tissue are the most critical parameters for disease diagnosis-based optical method. In recent years, researchers proposed spatial frequency domain imaging (SFDI) to quantitatively map tissue optical properties in a broad field of contactless imaging. To solve the limitations in wavebands unsuitable for silicon-based sensor technology, a compressed sensing (CS) algorithm is used to reproduce the original signal by a single-pixel detectors. Currently, the existing single-pixel SFDI method mainly uses a random sampling policy to extract and recover signals in the acquisition stage. However, these methods are memory-hungry and time-consuming, and they cannot generate discernible results under low sampling rate. Explorations on high performance and efficiency single-pixel SFDI are of great significance for clinical application.

AIM

Fourier single-pixel imaging can reconstruct signals with less time and space costs and has fewer reconstruction errors. We focus on an SFDI algorithm based on Fourier single-pixel imaging and propose our Fourier single-pixel image-based spatial frequency domain imaging method (FSI-SFDI).

APPROACH

First, we use Fourier single-pixel imaging algorithm to collect and compress signals and SFDI algorithm to generate optical parameters. Given the basis that the main energy of general image signals is concentrated in the range of low frequency of Fourier frequency domain, our FSI-SFDI uses a circular-sampling scheme to sample data points in the low-frequency region. Then, we reconstruct the image details from these points by optimization-based inverse-FFT method.

RESULTS

Our algorithm is tested on simulated data. Results show that the root mean square error (RMSE) of optical parameters is lower than 5% when the data reduction is 92%, and it can generate discernible optical parameter image with low sampling rate. We can observe that our FSI-SFDI primarily recovers the optical properties while keeping the RMSE under the upper bound of 4.5% when we use an image with 512  ×  512 resolution as the example for calculation and analysis. Not only that but also our algorithm consumes less space and time for an image with 256  ×  256 resolution, the signal reconstruction takes only 1.65 ms, and requires less RAM memory. Compared to CS-SFDI method, our FSI-SFDI can reduce the required number of measurements through optimizing algorithm.

CONCLUSIONS

Moreover, FSI-SFDI is capable of recovering high-quality resolvable images with lower sampling rate, higher-resolution images with less memory and time consumed than previous CS-SFDI method, which is very promising for clinical data collection and medical analysis.

Developing diagnostic assessment of breast lumpectomy tissues using radiomic and optical signatures

High positive margin rates in oncologic breast-conserving surgery are a pressing clinical problem. Volumetric X-ray scanning is emerging as a powerful ex vivo specimen imaging technique for analyzing resection margins, but X-rays lack contrast between non-malignant and malignant fibrous tissues. In this study, combined micro-CT and wide-field optical image radiomics were developed to classify malignancy of breast cancer tissues, demonstrating that X-ray/optical radiomics improve malignancy classification. Ninety-two standardized features were extracted from co-registered micro-CT and optical spatial frequency domain imaging samples extracted from 54 breast tumors exhibiting seven tissue subtypes confirmed by microscopic histological analysis. Multimodal feature sets improved classification performance versus micro-CT alone when adipose samples were included (AUC = 0.88 vs. 0.90; p-value = 3.65e-11) and excluded, focusing the classification task on exclusively non-malignant fibrous versus malignant tissues (AUC = 0.78 vs. 0.85; p-value = 9.33e-14). Extending the radiomics approach to high-dimensional optical data-termed "optomics" in this study-offers a promising optical image analysis technique for cancer detection. Radiomic feature data and classification source code are publicly available.

Deep learning-based method to accurately estimate breast tissue optical properties in the presence of the chest wall

SIGNIFICANCE

In general, image reconstruction methods used in diffuse optical tomography (DOT) are based on diffusion approximation, and they consider the breast tissue as a homogenous, semi-infinite medium. However, the semi-infinite medium assumption used in DOT reconstruction is not valid when the chest wall is underneath the breast tissue.

AIM

We aim to reduce the chest wall's effect on the estimated average optical properties of breast tissue and obtain accurate forward model for DOT reconstruction.

APPROACH

We propose a deep learning-based neural network approach where a convolution neural network (CNN) is trained to simultaneously obtain accurate optical property values for both the breast tissue and the chest wall.

RESULTS

The CNN model shows great promise in reducing errors in estimating the optical properties of the breast tissue in the presence of a shallow chest wall. For patient data, the CNN model predicted the breast tissue optical absorption coefficient, which was independent of chest wall depth.

CONCLUSIONS

Our proposed method can be readily used in DOT and diffuse spectroscopy measurements to improve the accuracy of estimated tissue optical properties.

Imaging sub-diffuse optical properties of cancerous and normal skin tissue using machine learning-aided spatial frequency domain imaging

SIGNIFICANCE

Sub-diffuse optical properties may serve as useful cancer biomarkers, and wide-field heatmaps of these properties could aid physicians in identifying cancerous tissue. Sub-diffuse spatial frequency domain imaging (sd-SFDI) can reveal such wide-field maps, but the current time cost of experimentally validated methods for rendering these heatmaps precludes this technology from potential real-time applications.

AIM

Our study renders heatmaps of sub-diffuse optical properties from experimental sd-SFDI images in real time and reports these properties for cancerous and normal skin tissue subtypes.

APPROACH

A phase function sampling method was used to simulate sd-SFDI spectra over a wide range of optical properties. A machine learning model trained on these simulations and tested on tissue phantoms was used to render sub-diffuse optical property heatmaps from sd-SFDI images of cancerous and normal skin tissue.

RESULTS

The model accurately rendered heatmaps from experimental sd-SFDI images in real time. In addition, heatmaps of a small number of tissue samples are presented to inform hypotheses on sub-diffuse optical property differences across skin tissue subtypes.

CONCLUSION

These results bring the overall process of sd-SFDI a fundamental step closer to real-time speeds and set a foundation for future real-time medical applications of sd-SFDI such as image guided surgery.

Machine learning for real-time optical property recovery in interstitial photodynamic therapy: a stimulation-based study

With the continued development of non-toxic photosensitizer drugs, interstitial photodynamic therapy (iPDT) is showing more favorable outcomes in recent clinical trials. IPDT planning is crucial to further increase the treatment efficacy. However, it remains a major challenge to generate a high-quality, patient-specific plan due to uncertainty in tissue optical properties (OPs), µ a and µ s . These parameters govern how light propagates inside tissues, and any deviation from the planning-assumed values during treatment could significantly affect the treatment outcome. In this work, we increase the robustness of iPDT against OP variations by using machine learning models to recover the patient-specific OPs from light dosimetry measurements and then re-optimizing the diffusers' optical powers to adapt to these OPs in real time. Simulations on virtual brain tumor models show that reoptimizing the power allocation with the recovered OPs significantly reduces uncertainty in the predicted light dosimetry for all tissues involved.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Modeling and Synthesis of Breast Cancer Optical Property Signatures With Generative Models

Is it possible to find deterministic relationships between optical measurements and pathophysiology in an unsupervised manner and based on data alone? Optical property quantification is a rapidly growing biomedical imaging technique for characterizing biological tissues that shows promise in a range of clinical applications, such as intraoperative breast-conserving surgery margin assessment. However, translating tissue optical properties to clinical pathology information is still a cumbersome problem due to, amongst other things, inter- and intrapatient variability, calibration, and ultimately the nonlinear behavior of light in turbid media. These challenges limit the ability of standard statistical methods to generate a simple model of pathology, requiring more advanced algorithms. We present a data-driven, nonlinear model of breast cancer pathology for real-time margin assessment of resected samples using optical properties derived from spatial frequency domain imaging data. A series of deep neural network models are employed to obtain sets of latent embeddings that relate optical data signatures to the underlying tissue pathology in a tractable manner. These self-explanatory models can translate absorption and scattering properties measured from pathology, while also being able to synthesize new data. The method was tested on a total of 70 resected breast tissue samples containing 137 regions of interest, achieving rapid optical property modeling with errors only limited by current semi-empirical models, allowing for mass sample synthesis and providing a systematic understanding of dataset properties, paving the way for deep automated margin assessment algorithms using structured light imaging or, in principle, any other optical imaging technique seeking modeling. Code is available.

Speckle illumination SFDI for projector-free optical property mapping

Spatial frequency domain imaging can map tissue scattering and absorption properties over a wide field of view, making it useful for clinical applications such as wound assessment and surgical guidance. This technique has previously required the projection of fully characterized illumination patterns. Here, we show that random and unknown speckle illumination can be used to sample the modulation transfer function of tissues at known spatial frequencies, allowing the quantitative mapping of optical properties with simple laser diode illumination. We compute low- and high-spatial frequency response parameters from the local power spectral density for each pixel and use a lookup table to accurately estimate absorption and scattering coefficients in tissue phantoms, in vivo human hand, and ex vivo swine esophagus. Because speckle patterns can be generated over a large depth of field and field of view with simple coherent illumination, this approach may enable optical property mapping in new form-factors and applications, including endoscopy.

Direct mapping from diffuse reflectance to chromophore concentrations in multi-fx spatial frequency domain imaging (SFDI) with a deep residual network (DRN)

Spatial frequency domain imaging (SFDI) is an emerging technology that enables label-free, non-contact, and wide-field mapping of tissue chromophore contents, such as oxy- and deoxy-hemoglobin concentrations. It has been shown that the use of more than two spatial frequencies (multi-fx ) can vastly improve measurement accuracy and reduce chromophore estimation uncertainties, but real-time multi-fx SFDI for chromophore monitoring has been limited in practice due to the slow speed of available chromophore inversion algorithms. Existing inversion algorithms have to first convert the multi-fx diffuse reflectance to optical absorptions, and then solve a set of linear equations to estimate chromophore concentrations. In this work, we present a deep learning framework, noted as a deep residual network (DRN), that is able to directly map from diffuse reflectance to chromophore concentrations. The proposed DRN is over 10x faster than the state-of-the-art method for chromophore inversion and enables 25x improvement on the frame rate for in vivo real-time oxygenation mapping. The proposed deep learning model will help enable real-time and highly accurate chromophore monitoring with multi-fx SFDI.

Rapid tissue oxygenation mapping from snapshot structured-light images with adversarial deep learning

SIGNIFICANCE

Spatial frequency-domain imaging (SFDI) is a powerful technique for mapping tissue oxygen saturation over a wide field of view. However, current SFDI methods either require a sequence of several images with different illumination patterns or, in the case of single-snapshot optical properties (SSOP), introduce artifacts and sacrifice accuracy.

AIM

We introduce OxyGAN, a data-driven, content-aware method to estimate tissue oxygenation directly from single structured-light images.

APPROACH

OxyGAN is an end-to-end approach that uses supervised generative adversarial networks. Conventional SFDI is used to obtain ground truth tissue oxygenation maps for ex vivo human esophagi, in vivo hands and feet, and an in vivo pig colon sample under 659- and 851-nm sinusoidal illumination. We benchmark OxyGAN by comparing it with SSOP and a two-step hybrid technique that uses a previously developed deep learning model to predict optical properties followed by a physical model to calculate tissue oxygenation.

RESULTS

When tested on human feet, cross-validated OxyGAN maps tissue oxygenation with an accuracy of 96.5%. When applied to sample types not included in the training set, such as human hands and pig colon, OxyGAN achieves a 93% accuracy, demonstrating robustness to various tissue types. On average, OxyGAN outperforms SSOP and a hybrid model in estimating tissue oxygenation by 24.9% and 24.7%, respectively. Finally, we optimize OxyGAN inference so that oxygenation maps are computed ∼10 times faster than previous work, enabling video-rate, 25-Hz imaging.

CONCLUSIONS

Due to its rapid acquisition and processing speed, OxyGAN has the potential to enable real-time, high-fidelity tissue oxygenation mapping that may be useful for many clinical applications.

Real-time, wide-field and high-quality single snapshot imaging of optical properties with profile correction using deep learning

The development of real-time, wide-field and quantitative diffuse optical imaging methods to visualize functional and structural biomarkers of living tissues is a pressing need for numerous clinical applications including image-guided surgery. In this context, Spatial Frequency Domain Imaging (SFDI) is an attractive method allowing for the fast estimation of optical properties using the Single Snapshot of Optical Properties (SSOP) approach. Herein, we present a novel implementation of SSOP based on a combination of deep learning network at the filtering stage and Graphics Processing Units (GPU) capable of simultaneous high visual quality image reconstruction, surface profile correction and accurate optical property (OP) extraction in real-time across large fields of view. In the most optimal implementation, the presented methodology demonstrates megapixel profile-corrected OP imaging with results comparable to that of profile-corrected SFDI, with a processing time of 18 ms and errors relative to SFDI method less than 10% in both profilometry and profile-corrected OPs. This novel processing framework lays the foundation for real-time multispectral quantitative diffuse optical imaging for surgical guidance and healthcare applications. All code and data used for this work is publicly available at www.healthphotonics.org under the resources tab.