1
|
Gouzou D, Taimori A, Haloubi T, Finlayson N, Wang Q, Hopgood JR, Vallejo M. Applications of machine learning in time-domain fluorescence lifetime imaging: a review. Methods Appl Fluoresc 2024; 12:022001. [PMID: 38055998 PMCID: PMC10851337 DOI: 10.1088/2050-6120/ad12f7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/30/2023] [Revised: 09/25/2023] [Accepted: 12/06/2023] [Indexed: 12/08/2023]
Abstract
Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.
Collapse
Affiliation(s)
- Dorian Gouzou
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| | - Ali Taimori
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Tarek Haloubi
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Neil Finlayson
- Neil Finlayson is with Institute for Integrated Micro and Nano Systems, School of Engineering, University ofEdinburgh, Edinburgh EH9 3FF, United Kingdom
| | - Qiang Wang
- Qiang Wang is with Centre for Inflammation Research, University of Edinburgh, Edinburgh, EH16 4TJ, United Kingdom
| | - James R Hopgood
- Tarek Haloubi, Ali Taimori, and James R. Hopgood are with Institute for Imaging, Data and Communication, School of Engineering, University of Edinburgh, Edinburgh, EH9 3FG, United Kingdom
| | - Marta Vallejo
- Dorian Gouzou and Marta Vallejo are with Institute of Signals, Sensors and Systems, School of Engineering and Physical Sciences, Heriot Watt University, Edinburgh, EH14 4AS, United Kingdom
| |
Collapse
|
2
|
Zang Z, Xiao D, Wang Q, Jiao Z, Chen Y, Li DDU. Compact and robust deep learning architecture for fluorescence lifetime imaging and FPGA implementation. Methods Appl Fluoresc 2023; 11. [PMID: 36863024 DOI: 10.1088/2050-6120/acc0d9] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2022] [Accepted: 03/01/2023] [Indexed: 03/04/2023]
Abstract
This paper reports a bespoke adder-based deep learning network for time-domain fluorescence lifetime imaging (FLIM). By leveraging thel1-norm extraction method, we propose a 1D Fluorescence Lifetime AdderNet (FLAN) without multiplication-based convolutions to reduce the computational complexity. Further, we compressed fluorescence decays in temporal dimension using a log-scale merging technique to discard redundant temporal information derived as log-scaling FLAN (FLAN+LS). FLAN+LS achieves 0.11 and 0.23 compression ratios compared with FLAN and a conventional 1D convolutional neural network (1D CNN) while maintaining high accuracy in retrieving lifetimes. We extensively evaluated FLAN and FLAN+LS using synthetic and real data. A traditional fitting method and other non-fitting, high-accuracy algorithms were compared with our networks for synthetic data. Our networks attained a minor reconstruction error in different photon-count scenarios. For real data, we used fluorescent beads' data acquired by a confocal microscope to validate the effectiveness of real fluorophores, and our networks can differentiate beads with different lifetimes. Additionally, we implemented the network architecture on a field-programmable gate array (FPGA) with a post-quantization technique to shorten the bit-width, thereby improving computing efficiency. FLAN+LS on hardware achieves the highest computing efficiency compared to 1D CNN and FLAN. We also discussed the applicability of our network and hardware architecture for other time-resolved biomedical applications using photon-efficient, time-resolved sensors.
Collapse
Affiliation(s)
- Zhenya Zang
- Department of Biomedical Engineering, University of Strathclyde, Glasgow G1 1XQ, United Kingdom
| | - Dong Xiao
- Department of Biomedical Engineering, University of Strathclyde, Glasgow G1 1XQ, United Kingdom
| | - Quan Wang
- Department of Biomedical Engineering, University of Strathclyde, Glasgow G1 1XQ, United Kingdom
| | - Ziao Jiao
- Department of Biomedical Engineering, University of Strathclyde, Glasgow G1 1XQ, United Kingdom
| | - Yu Chen
- Department of Physics, University of Strathclyde, Glasgow G4 0NG, United Kingdom
| | - David Day Uei Li
- Department of Biomedical Engineering, University of Strathclyde, Glasgow G1 1XQ, United Kingdom
| |
Collapse
|
3
|
Ochoa M, Smith JT, Gao S, Intes X. Computational macroscopic lifetime imaging and concentration unmixing of autofluorescence. J Biophotonics 2022; 15:e202200133. [PMID: 36546622 PMCID: PMC10026351 DOI: 10.1002/jbio.202200133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 07/06/2022] [Accepted: 07/12/2022] [Indexed: 06/17/2023]
Abstract
Single-pixel computational imaging can leverage highly sensitive detectors that concurrently acquire data across spectral and temporal domains. For molecular imaging, such methodology enables to collect rich intensity and lifetime multiplexed fluorescence datasets. Herein we report on the application of a single-pixel structured light-based platform for macroscopic imaging of tissue autofluorescence. The super-continuum visible excitation and hyperspectral single-pixel detection allow for parallel characterization of autofluorescence intensity and lifetime. Furthermore, we exploit a deep learning based data processing pipeline, to perform autofluorescence unmixing while yielding the autofluorophores' concentrations. The full scheme (setup and processing) is validated in silico and in vitro with clinically relevant autofluorophores flavin adenine dinucleotide, riboflavin, and protoporphyrin. The presented results demonstrate the potential of the methodology for macroscopically quantifying the intensity and lifetime of autofluorophores, with higher specificity for cases of mixed emissions, which are ubiquitous in autofluorescence and multiplexed in vivo imaging.
Collapse
Affiliation(s)
- Marien Ochoa
- Center for Modeling, Simulation and Imaging in Medicine (CeMSIM), Rensselaer Polytechnic Institute, Troy, New York, USA
| | - Jason T Smith
- Center for Modeling, Simulation and Imaging in Medicine (CeMSIM), Rensselaer Polytechnic Institute, Troy, New York, USA
| | - Shan Gao
- Center for Modeling, Simulation and Imaging in Medicine (CeMSIM), Rensselaer Polytechnic Institute, Troy, New York, USA
| | - Xavier Intes
- Center for Modeling, Simulation and Imaging in Medicine (CeMSIM), Rensselaer Polytechnic Institute, Troy, New York, USA
| |
Collapse
|
4
|
Schweitzer D, Haueisen J, Klemm M. Suppression of natural lens fluorescence in fundus autofluorescence measurements: review of hardware solutions. Biomed Opt Express 2022; 13:5151-5170. [PMID: 36425615 PMCID: PMC9664869 DOI: 10.1364/boe.462559] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/13/2022] [Revised: 07/25/2022] [Accepted: 07/25/2022] [Indexed: 06/16/2023]
Abstract
Fluorescence lifetime imaging ophthalmoscopy (FLIO), a technique for investigating metabolic changes in the eye ground, can reveal the first signs of diseases related to metabolism. The fluorescence of the natural lens overlies the fundus fluorescence. Although the influence of natural lens fluorescence can be somewhat decreased with mathematical models, excluding this influence during the measurement by using hardware enables more exact estimation of the fundus fluorescence. Here, we analyze four 1-photon excitation hardware solutions to suppress the influence of natural lens fluorescence: aperture stop separation, confocal scanning laser ophthalmoscopy, combined confocal scanning laser ophthalmoscopy and aperture stop separation, and dual point confocal scanning laser ophthalmoscopy. The effect of each principle is demonstrated in examples. The best suppression is provided by the dual point principle, realized with a confocal scanning laser ophthalmoscope. In this case, in addition to the fluorescence of the whole eye, the fluorescence of the anterior part of the eye is detected from a non-excited spot of the fundus. The intensity and time-resolved fluorescence spectral data of the fundus are derived through the subtraction of the simultaneously measured fluorescence of the excited and non-excited spots. Advantages of future 2-photon fluorescence excitation are also discussed. This study provides the first quantitative evaluation of hardware principles to suppress the fluorescence of the natural lens during measurements of fundus autofluorescence.
Collapse
Affiliation(s)
- D. Schweitzer
- Department of Ophthalmology, University Hospital Jena, Am Klinikum 1, 07747 Jena, Germany
| | - J. Haueisen
- Institute of Biomedical Engineering and Informatics, POB 100565, 98694 Ilmenau, Germany
| | - M. Klemm
- Institute of Biomedical Engineering and Informatics, POB 100565, 98694 Ilmenau, Germany
| |
Collapse
|
5
|
Klein L, Kristoffersen AS, Touš J, Žídek K. Versatile compressive microscope for hyperspectral transmission and fluorescence lifetime imaging. Opt Express 2022; 30:15708-15720. [PMID: 35473285 DOI: 10.1364/oe.455049] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/03/2022] [Accepted: 04/01/2022] [Indexed: 06/14/2023]
Abstract
Increasing demand for multimodal characterization and imaging of new materials entails the combination of various methods in a single microscopic setup. Hyperspectral imaging of transmission spectra or photoluminescence (PL) decay imaging count among the most used methods. Nevertheless, these methods require very different working conditions and instrumentation. Therefore, combining the methods into a single microscopic system is seldom implemented. Here we demonstrate a novel versatile microscope based on single-pixel imaging, where we use a simple optical configuration to measure the hyperspectral information, as well as fluorescence lifetime imaging (FLIM). The maps are inherently spatially matched and can be taken with spectral resolution limited by the resolution of the used spectrometer (3 nm) or temporal resolution set by PL decay measurement (120 ps). We verify the system's performance by its comparison to the standard FLIM and non-imaging transmission spectroscopy. Our approach enabled us to switch between a broad field-of-view and micrometer resolution without changing the optical configuration. At the same time, the used design opens the possibility to add a variety of other characterization methods. This article demonstrates a simple, affordable way of complex material studies with huge versatility for the imaging parameters.
Collapse
|
6
|
Acuña-Rodriguez JP, Mena-Vega JP, Argüello-Miranda O. Live-cell fluorescence spectral imaging as a data science challenge. Biophys Rev 2022; 14:579-597. [PMID: 35528031 PMCID: PMC9043069 DOI: 10.1007/s12551-022-00941-x] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2021] [Accepted: 03/09/2022] [Indexed: 12/13/2022] Open
Abstract
Live-cell fluorescence spectral imaging is an evolving modality of microscopy that uses specific properties of fluorophores, such as excitation or emission spectra, to detect multiple molecules and structures in intact cells. The main challenge of analyzing live-cell fluorescence spectral imaging data is the precise quantification of fluorescent molecules despite the weak signals and high noise found when imaging living cells under non-phototoxic conditions. Beyond the optimization of fluorophores and microscopy setups, quantifying multiple fluorophores requires algorithms that separate or unmix the contributions of the numerous fluorescent signals recorded at the single pixel level. This review aims to provide both the experimental scientist and the data analyst with a straightforward description of the evolution of spectral unmixing algorithms for fluorescence live-cell imaging. We show how the initial systems of linear equations used to determine the concentration of fluorophores in a pixel progressively evolved into matrix factorization, clustering, and deep learning approaches. We outline potential future trends on combining fluorescence spectral imaging with label-free detection methods, fluorescence lifetime imaging, and deep learning image analysis.
Collapse
Affiliation(s)
- Jessy Pamela Acuña-Rodriguez
- grid.412889.e0000 0004 1937 0706Center for Geophysical Research (CIGEFI), University of Costa Rica, San Pedro, San José Costa Rica
- grid.412889.e0000 0004 1937 0706School of Physics, University of Costa Rica, 2060 San Pedro, San José Costa Rica
| | - Jean Paul Mena-Vega
- grid.412889.e0000 0004 1937 0706School of Physics, University of Costa Rica, 2060 San Pedro, San José Costa Rica
| | - Orlando Argüello-Miranda
- grid.40803.3f0000 0001 2173 6074Department of Plant and Microbial Biology, North Carolina State University, 112 DERIEUX PLACE, Raleigh, NC 27695-7612 USA
| |
Collapse
|
7
|
Nizam NI, Ochoa M, Smith JT, Intes X. 3D k-space reflectance fluorescence tomography via deep learning. Opt Lett 2022; 47:1533-1536. [PMID: 35290357 PMCID: PMC9335514 DOI: 10.1364/ol.450935] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/10/2021] [Accepted: 02/16/2022] [Indexed: 06/14/2023]
Abstract
We report on the potential to perform image reconstruction in 3D k-space reflectance fluorescence tomography (FT) using deep learning (DL). Herein, we adopt a modified AUTOMAP architecture and develop a training methodology that leverages an open-source Monte-Carlo-based simulator to generate a large dataset. Using an enhanced EMNIST (EEMNIST) dataset as an embedded contrast function allows us to train the network efficiently. The optical strategy utilizes k-space illumination in a reflectance configuration to probe tissue in the mesoscopic regime with high sensitivity and resolution. The proposed DL model training and validation is performed with both in silico data and a phantom experiment. Overall, our results indicate that the approach can correctly reconstruct both single and multiple fluorescent embedding(s) in a 3D volume. Furthermore, the presented technique is shown to outperform the traditional approaches [least-squares (LSQ) and total-variation minimization (TVAL)], especially at higher depths. We, therefore, expect the proposed computational technique to have future implications in preclinical studies.
Collapse
Affiliation(s)
- Navid Ibtehaj Nizam
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA
| | - Marien Ochoa
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA
| | - Jason T. Smith
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA
| | - Xavier Intes
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY 12180, USA
- Center for Modeling, Simulation and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, NY 12180, USA
| |
Collapse
|
8
|
Smith JT, Ochoa M, Faulkner D, Haskins G, Intes X. Deep learning in macroscopic diffuse optical imaging. J Biomed Opt 2022; 27:JBO-210288VRR. [PMID: 35218169 PMCID: PMC8881080 DOI: 10.1117/1.jbo.27.2.020901] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/14/2021] [Accepted: 02/09/2022] [Indexed: 05/02/2023]
Abstract
SIGNIFICANCE Biomedical optics system design, image formation, and image analysis have primarily been guided by classical physical modeling and signal processing methodologies. Recently, however, deep learning (DL) has become a major paradigm in computational modeling and has demonstrated utility in numerous scientific domains and various forms of data analysis. AIM We aim to comprehensively review the use of DL applied to macroscopic diffuse optical imaging (DOI). APPROACH First, we provide a layman introduction to DL. Then, the review summarizes current DL work in some of the most active areas of this field, including optical properties retrieval, fluorescence lifetime imaging, and diffuse optical tomography. RESULTS The advantages of using DL for DOI versus conventional inverse solvers cited in the literature reviewed herein are numerous. These include, among others, a decrease in analysis time (often by many orders of magnitude), increased quantitative reconstruction quality, robustness to noise, and the unique capability to learn complex end-to-end relationships. CONCLUSIONS The heavily validated capability of DL's use across a wide range of complex inverse solving methodologies has enormous potential to bring novel DOI modalities, otherwise deemed impractical for clinical translation, to the patient's bedside.
Collapse
Affiliation(s)
- Jason T Smith
- Rensselaer Polytechnic Institute, Department of Biomedical Engineering, Troy, New York, United States
| | - Marien Ochoa
- Rensselaer Polytechnic Institute, Department of Biomedical Engineering, Troy, New York, United States
| | - Denzel Faulkner
- Rensselaer Polytechnic Institute, Department of Biomedical Engineering, Troy, New York, United States
| | - Grant Haskins
- Rensselaer Polytechnic Institute, Department of Biomedical Engineering, Troy, New York, United States
| | - Xavier Intes
- Rensselaer Polytechnic Institute, Center for Modeling, Simulation and Imaging for Medicine, Troy, Ne, United States
| |
Collapse
|
9
|
Abstract
Precision medicine promises to improve therapeutic efficacy while reducing adverse effects, especially in oncology. However, despite great progresses in recent years, precision medicine for cancer treatment is not always part of routine care. Indeed, the ability to specifically tailor therapies to distinct patient profiles requires still significant improvements in targeted therapy development as well as decreases in drug treatment failures. In this regard, preclinical animal research is fundamental to advance our understanding of tumor biology, and diagnostic and therapeutic response. Most importantly, the ability to measure drug-target engagement accurately in live and intact animals is critical in guiding the development and optimization of targeted therapy. However, a major limitation of preclinical molecular imaging modalities is their lack of capability to directly and quantitatively discriminate between drug accumulation and drug-target engagement at the pathological site. Recently, we have developed Macroscopic Fluorescence Lifetime Imaging (MFLI) as a unique feature of optical imaging to quantitate in vivo drug-target engagement. MFLI quantitatively reports on nanoscale interactions via lifetime-sensing of Förster Resonance Energy Transfer (FRET) in live, intact animals. Hence, MFLI FRET acts as a direct reporter of receptor dimerization and target engagement via the measurement of the fraction of labeled-donor entity undergoing binding to its respective receptor. MFLI is expected to greatly impact preclinical imaging and also adjacent fields such as image-guided surgery and drug development.
Collapse
Affiliation(s)
- Marien Ochoa
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA
| | - Alena Rudkouskaya
- Department of Cellular and Molecular Physiology, Albany Medical College, Albany, NY, USA
| | - Jason T Smith
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA
| | - Xavier Intes
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, NY, USA
| | - Margarida Barroso
- Department of Cellular and Molecular Physiology, Albany Medical College, Albany, NY, USA.
| |
Collapse
|
10
|
Devos O, Ghaffari M, Vitale R, de Juan A, Sliwa M, Ruckebusch C. Multivariate Curve Resolution Slicing of Multiexponential Time-Resolved Spectroscopy Fluorescence Data. Anal Chem 2021; 93:12504-12513. [PMID: 34494422 DOI: 10.1021/acs.analchem.1c01284] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
Abstract
Time-resolved fluorescence spectroscopy (TRFS), i.e., measurement of fluorescence decay curves for different excitation and/or emission wavelengths, provides specific and sensitive local information on molecules and on their environment. However, TRFS relies on multiexponential data fitting to derive fluorescence lifetimes from the measured decay curves and the time resolution of the technique is limited by the instrumental response function (IRF). We propose here a multivariate curve resolution (MCR) approach based on data slicing to perform tailored and fit-free analysis of multiexponential fluorescence decay curves. MCR slicing, taking as a basic framework the multivariate curve resolution-alternating least-squares (MCR-ALS) soft-modeling algorithm, relies on a hybrid bilinear/trilinear data decomposition. A key feature of the method is that it enables the recovery of individual components characterized by decay profiles that are only partially describable by monoexponential functions. For TRFS data, not only pure multiexponential tail information but also shorter time delay information can be decomposed, where the signal deviates from the ideal exponential behavior due to the limited time resolution. The accuracy of the proposed approach is validated by analyzing mixtures of three commercial dyes and characterizing the mixture composition, lifetimes, and associated contributions, even in situations where only ternary mixture samples are available. MCR slicing is also applied to the analysis of TRFS data obtained on a photoswitchable fluorescent protein (rsEGFP2). Three fluorescence lifetimes are extracted, along with the profile of the IRF, highlighting that decomposition of complex systems, for which individual isomers are characterized by different exponential decays, can also be achieved.
Collapse
Affiliation(s)
- Olivier Devos
- Univ. Lille, CNRS, UMR 8516 - LASIRE - Laboratory of advanced spectroscopy, interactions, reactivity and environment, Cité scientifique, Bâtiment C5, 59000 Lille, France
| | - Mahdiyeh Ghaffari
- Univ. Lille, CNRS, UMR 8516 - LASIRE - Laboratory of advanced spectroscopy, interactions, reactivity and environment, Cité scientifique, Bâtiment C5, 59000 Lille, France
| | - Raffaele Vitale
- Univ. Lille, CNRS, UMR 8516 - LASIRE - Laboratory of advanced spectroscopy, interactions, reactivity and environment, Cité scientifique, Bâtiment C5, 59000 Lille, France
| | - Anna de Juan
- Chemometrics Group, Dept. of Chemical Engineering and Analytical Chemistry, Universitat de Barcelona, Martí I Franquès, 1, 08028 Barcelona, Spain
| | - Michel Sliwa
- Univ. Lille, CNRS, UMR 8516 - LASIRE - Laboratory of advanced spectroscopy, interactions, reactivity and environment, Cité scientifique, Bâtiment C5, 59000 Lille, France
| | - Cyril Ruckebusch
- Univ. Lille, CNRS, UMR 8516 - LASIRE - Laboratory of advanced spectroscopy, interactions, reactivity and environment, Cité scientifique, Bâtiment C5, 59000 Lille, France
| |
Collapse
|
11
|
Tian L, Hunt B, Bell MAL, Yi J, Smith JT, Ochoa M, Intes X, Durr NJ. Deep Learning in Biomedical Optics. Lasers Surg Med 2021; 53:748-775. [PMID: 34015146 PMCID: PMC8273152 DOI: 10.1002/lsm.23414] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2021] [Revised: 04/02/2021] [Accepted: 04/15/2021] [Indexed: 01/02/2023]
Abstract
This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.
Collapse
Affiliation(s)
- L. Tian
- Department of Electrical and Computer Engineering, Boston University, Boston, MA, USA
| | - B. Hunt
- Thayer School of Engineering, Dartmouth College, Hanover, NH, USA
| | - M. A. L. Bell
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA
| | - J. Yi
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Ophthalmology, Johns Hopkins University, Baltimore, MD, USA
| | - J. T. Smith
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - M. Ochoa
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - X. Intes
- Center for Modeling, Simulation, and Imaging in Medicine, Rensselaer Polytechnic Institute, Troy, New York NY 12180
| | - N. J. Durr
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD, USA
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
12
|
Datta R, Gillette A, Stefely M, Skala MC. Recent innovations in fluorescence lifetime imaging microscopy for biology and medicine. J Biomed Opt 2021; 26:JBO-210093-PER. [PMID: 34247457 PMCID: PMC8271181 DOI: 10.1117/1.jbo.26.7.070603] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/30/2021] [Accepted: 06/11/2021] [Indexed: 05/05/2023]
Abstract
SIGNIFICANCE Fluorescence lifetime imaging microscopy (FLIM) measures the decay rate of fluorophores, thus providing insights into molecular interactions. FLIM is a powerful molecular imaging technique that is widely used in biology and medicine. AIM This perspective highlights some of the major advances in FLIM instrumentation, analysis, and biological and clinical applications that we have found impactful over the last year. APPROACH Innovations in FLIM instrumentation resulted in faster acquisition speeds, rapid imaging over large fields of view, and integration with complementary modalities such as single-molecule microscopy or light-sheet microscopy. There were significant developments in FLIM analysis with machine learning approaches to enhance processing speeds, fit-free techniques to analyze images without a priori knowledge, and open-source analysis resources. The advantages and limitations of these recent instrumentation and analysis techniques are summarized. Finally, applications of FLIM in the last year include label-free imaging in biology, ophthalmology, and intraoperative imaging, FLIM of new fluorescent probes, and lifetime-based Förster resonance energy transfer measurements. CONCLUSIONS A large number of high-quality publications over the last year signifies the growing interest in FLIM and ensures continued technological improvements and expanding applications in biomedical research.
Collapse
Affiliation(s)
- Rupsa Datta
- Morgridge Institute for Research, Madison, Wisconsin, United States
| | - Amani Gillette
- Morgridge Institute for Research, Madison, Wisconsin, United States
- University of Wisconsin, Department of Biomedical Engineering, Madison, Wisconsin, United States
| | - Matthew Stefely
- Morgridge Institute for Research, Madison, Wisconsin, United States
| | - Melissa C. Skala
- Morgridge Institute for Research, Madison, Wisconsin, United States
- University of Wisconsin, Department of Biomedical Engineering, Madison, Wisconsin, United States
- Address all correspondence to Melissa C. Skala,
| |
Collapse
|
13
|
Abstract
A major focus of current biological studies is to fill the knowledge gaps between cell, tissue and organism scales. To this end, a wide array of contemporary optical analytical tools enable multiparameter quantitative imaging of live and fixed cells, three-dimensional (3D) systems, tissues, organs and organisms in the context of their complex spatiotemporal biological and molecular features. In particular, the modalities of luminescence lifetime imaging, comprising fluorescence lifetime imaging (FLI) and phosphorescence lifetime imaging microscopy (PLIM), in synergy with Förster resonance energy transfer (FRET) assays, provide a wealth of information. On the application side, the luminescence lifetime of endogenous molecules inside cells and tissues, overexpressed fluorescent protein fusion biosensor constructs or probes delivered externally provide molecular insights at multiple scales into protein-protein interaction networks, cellular metabolism, dynamics of molecular oxygen and hypoxia, physiologically important ions, and other physical and physiological parameters. Luminescence lifetime imaging offers a unique window into the physiological and structural environment of cells and tissues, enabling a new level of functional and molecular analysis in addition to providing 3D spatially resolved and longitudinal measurements that can range from microscopic to macroscopic scale. We provide an overview of luminescence lifetime imaging and summarize key biological applications from cells and tissues to organisms.
Collapse
Affiliation(s)
- Ruslan I. Dmitriev
- Tissue Engineering and Biomaterials Group, Department of
Human Structure and Repair, Faculty of Medicine and Health Sciences,
Ghent University, Ghent 9000,
Belgium
| | - Xavier Intes
- Department of Biomedical Engineering, Center for
Modeling, Simulation and Imaging for Medicine (CeMSIM),
Rensselaer Polytechnic Institute, Troy, NY
12180-3590, USA
| | - Margarida M. Barroso
- Department of Molecular and Cellular
Physiology, Albany Medical College,
Albany, NY 12208, USA
| |
Collapse
|
14
|
Ghezzi A, Farina A, Bassi A, Valentini G, Labanca I, Acconcia G, Rech I, D'Andrea C. Multispectral compressive fluorescence lifetime imaging microscopy with a SPAD array detector. Opt Lett 2021; 46:1353-1356. [PMID: 33720185 DOI: 10.1364/ol.419381] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Accepted: 02/06/2021] [Indexed: 05/22/2023]
Abstract
Multispectral/hyperspectral fluorescence lifetime imaging microscopy (λFLIM) is a promising tool for studying functional and structural biological processes. The rich information content provided by a multidimensional dataset is often in contrast with the acquisition speed. In this work, we develop and experimentally demonstrate a wide-field λFLIM setup, based on a novel time-resolved 18×1 single-photon avalanche diode array detector working in a single-pixel camera scheme, which parallelizes the spectral detection, reducing measurement time. The proposed system, which implements a single-pixel camera with a compressive sensing scheme, represents an optimal microscopy framework towards the design of λFLIM setups.
Collapse
|
15
|
Smith JT, Aguénounon E, Gioux S, Intes X. Macroscopic fluorescence lifetime topography enhanced via spatial frequency domain imaging. Opt Lett 2020; 45:4232-4235. [PMID: 32735266 PMCID: PMC7935427 DOI: 10.1364/ol.397605] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
We report on a macroscopic fluorescence lifetime imaging (MFLI) topography computational framework based around machine learning with the main goal of retrieving the depth of fluorescent inclusions deeply seated in bio-tissues. This approach leverages the depth-resolved information inherent to time-resolved fluorescence data sets coupled with the retrieval of in situ optical properties as obtained via spatial frequency domain imaging (SFDI). Specifically, a Siamese network architecture is proposed with optical properties (OPs) and time-resolved fluorescence decays as input followed by simultaneous retrieval of lifetime maps and depth profiles. We validate our approach using comprehensive in silico data sets as well as with a phantom experiment. Overall, our results demonstrate that our approach can retrieve the depth of fluorescence inclusions, especially when coupled with optical properties estimation, with high accuracy. We expect the presented computational approach to find great utility in applications such as optical-guided surgery.
Collapse
Affiliation(s)
- Jason T. Smith
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, New York 12180, USA
| | - Enagnon Aguénounon
- University of Strasbourg, ICube Laboratory, 300 Boulevard Sebastien Brant, 67412 Illkirch, France
| | - Sylvain Gioux
- University of Strasbourg, ICube Laboratory, 300 Boulevard Sebastien Brant, 67412 Illkirch, France
| | - Xavier Intes
- Department of Biomedical Engineering, Rensselaer Polytechnic Institute, Troy, New York 12180, USA
- Corresponding author:
| |
Collapse
|