Deep learning-based image processing in optical microscopy

A novel method (RIM-Deep) for enhancing imaging depth and resolution stability of deep cleared tissue in inverted confocal microscopy

The increasing use of tissue clearing techniques underscores the urgent need for cost-effective and simplified deep imaging methods. While traditional inverted confocal microscopes excel in high-resolution imaging of tissue sections and cultured cells, they face limitations in deep imaging of cleared tissues due to refractive index mismatches between the immersion media of objectives and sample container. To overcome these challenges, the RIM-Deep was developed to significantly improve deep imaging capabilities without compromising the normal function of the confocal microscope. This system facilitates deep immunofluorescence imaging of the prefrontal cortex in cleared macaque tissue, extending imaging depth from 2 mm to 5 mm. Applied to an intact and cleared Thy1-EGFP mouse brain, the system allowed for clear axonal visualization at high imaging depth. Moreover, this advancement enables large-scale, deep 3D imaging of intact tissues. In principle, this concept can be extended to any imaging modality, including existing inverted wide-field, confocal, and two-photon microscopy. This would significantly upgrade traditional laboratory configurations and facilitate the study of connectomes in the brain and other tissues.

Applications of Artificial Intelligence, Deep Learning, and Machine Learning to Support the Analysis of Microscopic Images of Cells and Tissues

Artificial intelligence (AI) transforms image data analysis across many biomedical fields, such as cell biology, radiology, pathology, cancer biology, and immunology, with object detection, image feature extraction, classification, and segmentation applications. Advancements in deep learning (DL) research have been a critical factor in advancing computer techniques for biomedical image analysis and data mining. A significant improvement in the accuracy of cell detection and segmentation algorithms has been achieved as a result of the emergence of open-source software and innovative deep neural network architectures. Automated cell segmentation now enables the extraction of quantifiable cellular and spatial features from microscope images of cells and tissues, providing critical insights into cellular organization in various diseases. This review aims to examine the latest AI and DL techniques for cell analysis and data mining in microscopy images, aid the biologists who have less background knowledge in AI and machine learning (ML), and incorporate the ML models into microscopy focus images.

A novel approach in cancer diagnosis: integrating holography microscopic medical imaging and deep learning techniques-challenges and future trends

Medical imaging is pivotal in early disease diagnosis, providing essential insights that enable timely and accurate detection of health anomalies. Traditional imaging techniques, such as Magnetic Resonance Imaging (MRI), Computer Tomography (CT), ultrasound, and Positron Emission Tomography (PET), offer vital insights into three-dimensional structures but frequently fall short of delivering a comprehensive and detailed anatomical analysis, capturing only amplitude details. Three-dimensional holography microscopic medical imaging provides a promising solution by capturing the amplitude (brightness) and phase (structural information) details of biological structures. In this study, we investigate the novel collaborative potential of Deep Learning (DL) and holography microscopic phase imaging for cancer diagnosis. The study comprehensively examines existing literature, analyzes advancements, identifies research gaps, and proposes future research directions in cancer diagnosis through the integrated Quantitative Phase Imaging (QPI) and DL methodology. This novel approach addresses a critical limitation of traditional imaging by capturing detailed structural information, paving the way for more accurate diagnostics. The proposed approach comprises tissue sample collection, holographic image scanning, preprocessing in case of imbalanced datasets, and training on annotated datasets using DL architectures like U-Net and Vision Transformer(ViT's). Furthermore, sophisticated concepts in DL, like the incorporation of Explainable AI (XAI) techniques, are suggested for comprehensive disease diagnosis and identification. The study thoroughly investigates the advantages of integrating holography imaging and DL for precise cancer diagnosis. Additionally, meticulous insights are presented by identifying the challenges associated with this integration methodology.

Quantifying Monomer-Dimer Distribution of Nanoparticles from Uncorrelated Optical Images Using Deep Learning

Nanoparticles embedded in polymer matrices play a critical role in enhancing the properties and functionalities of composite materials. Detecting and quantifying nanoparticles from optical images (fixed samples-in vitro imaging) is crucial for understanding their distribution, aggregation, and interactions, which can lead to advancements in nanotechnology, materials science, and biomedical research. In this article, we propose an ensembled deep learning approach for automatic nanoparticle detection and oligomerization quantification in a polymer matrix for optical images. The majority of prior studies of nanoparticle identification and categorization of fixed samples are based on scanning electron microscopy (SEM) or transmission electron microscopy (TEM) images, which are destructive to biological imaging. However, the proposed study is based on optical images, which are susceptible to noise, low contrast, anisotropic shape, overlapping of the point spread function, plasmon coupling, and resolution limitations. In this study, we fine-tune a deep neural network architecture, YOLOv8, on a carefully annotated data set of correlated optical and SEM images of 80 nm gold nanospheres (AuNSs) of varying oligomerization states. The resultant model features a weighted average accuracy of 80.7% for quantification of AuNSs and determination of their oligomeric state, far surpassing the capabilities of existing manual image processing methods. We also demonstrate its speed and effectiveness in nanoparticle detection and oligomerization within the polymer matrix through tests on high-density uncorrelated optical images. The optical image-based quantification technique will be useful for (live samples-for in vivo imaging) analyzing nanoparticle uptake, oligomerization state, and aggregation kinetics in live cells and identifying stoichiometry of membrane protein and its interactions, nanoparticle-cell interaction, cell signaling imaging, and drug delivery.

Self-Driving Microscopes: AI Meets Super-Resolution Microscopy

The integration of Machine Learning (ML) with super-resolution microscopy represents a transformative advancement in biomedical research. Recent advances in ML, particularly deep learning (DL), have significantly enhanced image processing tasks, such as denoising and reconstruction. This review explores the growing potential of automation in super-resolution microscopy, focusing on how DL can enable autonomous imaging tasks. Overcoming the challenges of automation, particularly in adapting to dynamic biological processes and minimizing manual intervention, is crucial for the future of microscopy. Whilst still in its infancy, automation in super-resolution can revolutionize drug discovery and disease phenotyping leading to similar breakthroughs as have been recognized in this year's Nobel Prizes for Physics and Chemistry.

Effect of shear rate on early Shewanella oneidensis adhesion dynamics monitored by deep learning

Understanding pioneer bacterial adhesion is essential to appreciate bacterial colonization and consider appropriate control strategies. This bacterial entrapment at the wall is known to be controlled by many physical, chemical or biological factors, including hydrodynamic conditions. However, due to the nature of early bacterial adhesion, i.e. a short and dynamic process with low biomass involved, such investigations are challenging. In this context, our study aimed to evaluate the effect of wall shear rate on the early bacterial adhesion dynamics. Firstly, at the population scale by assessing bacterial colonization kinetics and the mechanisms responsible for wall transfer under shear rates using a time-lapse approach. Secondly, at the individual scale, by implementing an automated image processing method based on deep learning to track each individual pioneer bacterium on the wall. Bacterial adhesion experiments are performed on a model bacterium (Shewanella oneidensis MR-1) at different shear rates (0 to1250 s-1) in a microfluidic system mounted under a microscope equipped with a CCD camera. Image processing was performed using a trained neural network (YOLOv8), which allowed information extraction, i.e. bacterial wall residence time and orientation for each adhered bacterium during pioneer colonization (14 min). Collected from over 20,000 bacteria, our results showed that adhered bacteria had a very short residence time at the wall, with over 70 % remaining less than 1 min. Shear rates had a non-proportional effect on pioneer colonization with a bell-shape profile suggesting that intermediate shear rates improved both bacterial wall residence time as well as colonization rate and level. This lack of proportionality highlights the dual effect of wall shear rate on early bacterial colonization; initially increasing it improves bacterial colonization up to a threshold, beyond which it leads to higher bacterial wall detachment. The present study provides quantitative data on the individual dynamics of just adhered bacteria within a population when exposed to different rates of wall shear.

Image processing tools for petabyte-scale light sheet microscopy data

Light sheet microscopy is a powerful technique for high-speed three-dimensional imaging of subcellular dynamics and large biological specimens. However, it often generates datasets ranging from hundreds of gigabytes to petabytes in size for a single experiment. Conventional computational tools process such images far slower than the time to acquire them and often fail outright due to memory limitations. To address these challenges, we present PetaKit5D, a scalable software solution for efficient petabyte-scale light sheet image processing. This software incorporates a suite of commonly used processing tools that are optimized for memory and performance. Notable advancements include rapid image readers and writers, fast and memory-efficient geometric transformations, high-performance Richardson-Lucy deconvolution and scalable Zarr-based stitching. These features outperform state-of-the-art methods by over one order of magnitude, enabling the processing of petabyte-scale image data at the full teravoxel rates of modern imaging cameras. The software opens new avenues for biological discoveries through large-scale imaging experiments.

Accurate Identification of Cancer Cells in Complex Pre-Clinical Models Using a Deep-Learning Neural Network: A Transfection-Free Approach

3D co-cultures are key tools for in vitro biomedical research as they recapitulate more closely the in vivo environment while allowing a tighter control on the culture's composition and experimental conditions. The limited technologies available for the analysis of these models, however, hamper their widespread application. The separation of the contribution of the different cell types, in particular, is a fundamental challenge. In this work, ORACLE (OvaRiAn Cancer ceLl rEcognition) is presented, a deep neural network trained to distinguish between ovarian cancer and healthy cells based on the shape of their nucleus. The extensive validation that are conducted includes multiple cell lines and patient-derived cultures to characterize the effect of all the major potential confounding factors. High accuracy and reliability are maintained throughout the analysis (F1score> 0.9 and Area under the ROC curve -ROC-AUC- score = 0.99) demonstrating ORACLE's effectiveness with this detection and classification task. ORACLE is freely available (https://github.com/MarilisaCortesi/ORACLE/tree/main) and can be used to recognize both ovarian cancer cell lines and primary patient-derived cells. This feature is unique to ORACLE and thus enables for the first time the analysis of in vitro co-cultures comprised solely of patient-derived cells.

Ex vivo nanoscale abluminal mapping of putative cargo receptors at the blood-brain barrier of expanded brain capillaries

Receptor mediated transport of therapeutic antibodies through the blood-brain barrier (BBB) give promise for drug delivery to alleviate brain diseases. We developed a low-cost method to obtain nanoscale localization data of putative cargo receptors. We combine existing ex vivo isolation methods with expansion microscopy (ExM) to analyze receptor localizations in brain microcapillaries. Using this approach, we show how to analyze receptor localizations in endothelial cells of brain microcapillaries in relation to the abluminal marker collagen IV. By choosing the thinnest capillaries, microcapillaries for analysis, we ensure the validity of collagen IV as an abluminal marker. With this tool, we confirm transferrin receptors as well as sortilin to be both luminally and abluminally localized. Furthermore, we identify basigin to be an abluminal receptor. Our methodology can be adapted to analyze different types of isolated brain capillaries and we anticipate that this approach will be very useful for the research community to gain new insight into cargo receptor trafficking in the slim brain endothelial cells to elucidate novel paths for future drug design.

Convolutional neural network transformer (CNNT) for fluorescence microscopy image denoising with improved generalization and fast adaptation

Deep neural networks can improve the quality of fluorescence microscopy images. Previous methods, based on Convolutional Neural Networks (CNNs), require time-consuming training of individual models for each experiment, impairing their applicability and generalization. In this study, we propose a novel imaging-transformer based model, Convolutional Neural Network Transformer (CNNT), that outperforms CNN based networks for image denoising. We train a general CNNT based backbone model from pairwise high-low Signal-to-Noise Ratio (SNR) image volumes, gathered from a single type of fluorescence microscope, an instant Structured Illumination Microscope. Fast adaptation to new microscopes is achieved by fine-tuning the backbone on only 5-10 image volume pairs per new experiment. Results show that the CNNT backbone and fine-tuning scheme significantly reduces training time and improves image quality, outperforming models trained using only CNNs such as 3D-RCAN and Noise2Fast. We show three examples of efficacy of this approach in wide-field, two-photon, and confocal fluorescence microscopy.

Deep learning-based spectroscopic single-molecule localization microscopy

Significance

Spectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.

Aim

Develop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.

Approach

We developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.

Results

For label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.

Conclusions

We demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

Expanding the Horizons of Machine Learning in Nanomaterials to Chiral Nanostructures

Machine learning holds significant research potential in the field of nanotechnology, enabling nanomaterial structure and property predictions, facilitating materials design and discovery, and reducing the need for time-consuming and labor-intensive experiments and simulations. In contrast to their achiral counterparts, the application of machine learning for chiral nanomaterials is still in its infancy, with a limited number of publications to date. This is despite the great potential of machine learning to advance the development of new sustainable chiral materials with high values of optical activity, circularly polarized luminescence, and enantioselectivity, as well as for the analysis of structural chirality by electron microscopy. In this review, an analysis of machine learning methods used for studying achiral nanomaterials is provided, subsequently offering guidance on adapting and extending this work to chiral nanomaterials. An overview of chiral nanomaterials within the framework of synthesis-structure-property-application relationships is presented and insights on how to leverage machine learning for the study of these highly complex relationships are provided. Some key recent publications are reviewed and discussed on the application of machine learning for chiral nanomaterials. Finally, the review captures the key achievements, ongoing challenges, and the prospective outlook for this very important research field.

Convolutional Neural Network Transformer (CNNT) for Fluorescence Microscopy image Denoising with Improved Generalization and Fast Adaptation

Deep neural networks have been applied to improve the image quality of fluorescence microscopy imaging. Previous methods are based on convolutional neural networks (CNNs) which generally require more time-consuming training of separate models for each new imaging experiment, impairing the applicability and generalization. Once the model is trained (typically with tens to hundreds of image pairs) it can then be used to enhance new images that are like the training data. In this study, we proposed a novel imaging-transformer based model, Convolutional Neural Network Transformer (CNNT), to outperform the CNN networks for image denoising. In our scheme we have trained a single CNNT based "backbone model" from pairwise high-low SNR images for one type of fluorescence microscope (instance structured illumination, iSim). Fast adaption to new applications was achieved by fine-tuning the backbone on only 5-10 sample pairs per new experiment. Results show the CNNT backbone and fine-tuning scheme significantly reduces the training time and improves the image quality, outperformed training separate models using CNN approaches such as - RCAN and Noise2Fast. Here we show three examples of the efficacy of this approach on denoising wide-field, two-photon and confocal fluorescence data. In the confocal experiment, which is a 5×5 tiled acquisition, the fine-tuned CNNT model reduces the scan time form one hour to eight minutes, with improved quality.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

Restoration of metabolic functional metrics from label-free, two-photon human tissue images using multiscale deep-learning-based denoising algorithms

Significance

Label-free, two-photon excited fluorescence (TPEF) imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, noise and other artifacts present in these images severely complicate the extraction of biologically useful information.

Aim

We aim to employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-signal-to-noise ratio (SNR), TPEF images.

Approach

TPEF images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used to assess the impact of various denoising models, preprocessing methods, and data on metrics of image quality and the recovery of six metrics of metabolic function from the images relative to ground truth images.

Results

Optimized recovery of the redox ratio and mitochondrial organization is achieved using a novel algorithm based on deep denoising in the wavelet transform domain. This algorithm also leads to significant improvements in peak-SNR (PSNR) and structural similarity index measure (SSIM) for all images. Interestingly, other models yield even higher PSNR and SSIM improvements, but they are not optimal for recovery of metabolic function metrics.

Conclusions

Denoising algorithms can recover diagnostically useful information from low SNR label-free TPEF images and will be useful for the clinical translation of such imaging.

Combinatorial quantification of distinct neural projections from retrograde tracing

Comprehensive quantification of neuronal architectures underlying anatomical brain connectivity remains challenging. We introduce a method to identify distinct axonal projection patterns from a source to a set of target regions and the count of neurons with each pattern. A source region projecting to n targets could have 2n-1 theoretically possible projection types, although only a subset of these types typically exists. By injecting uniquely labeled retrograde tracers in k target regions (k < n), one can experimentally count the cells expressing different color combinations in the source region. The neuronal counts for different color combinations from n-choose-k experiments provide constraints for a model that is robustly solvable using evolutionary algorithms. Here, we demonstrate this method's reliability for 4 targets using simulated triple injection experiments. Furthermore, we illustrate the experimental application of this framework by quantifying the projections of male mouse primary motor cortex to the primary and secondary somatosensory and motor cortices.

Developments in Image Processing Using Deep Learning and Reinforcement Learning

The growth in the volume of data generated, consumed, and stored, which is estimated to exceed 180 zettabytes in 2025, represents a major challenge both for organizations and for society in general. In addition to being larger, datasets are increasingly complex, bringing new theoretical and computational challenges. Alongside this evolution, data science tools have exploded in popularity over the past two decades due to their myriad of applications when dealing with complex data, their high accuracy, flexible customization, and excellent adaptability. When it comes to images, data analysis presents additional challenges because as the quality of an image increases, which is desirable, so does the volume of data to be processed. Although classic machine learning (ML) techniques are still widely used in different research fields and industries, there has been great interest from the scientific community in the development of new artificial intelligence (AI) techniques. The resurgence of neural networks has boosted remarkable advances in areas such as the understanding and processing of images. In this study, we conducted a comprehensive survey regarding advances in AI design and the optimization solutions proposed to deal with image processing challenges. Despite the good results that have been achieved, there are still many challenges to face in this field of study. In this work, we discuss the main and more recent improvements, applications, and developments when targeting image processing applications, and we propose future research directions in this field of constant and fast evolution.

Pair-Variational Autoencoders for Linking and Cross-Reconstruction of Characterization Data from Complementary Structural Characterization Techniques

In materials research, structural characterization often requires multiple complementary techniques to obtain a holistic morphological view of a synthesized material. Depending on the availability and accessibility of the different characterization techniques (e.g., scattering, microscopy, spectroscopy), each research facility or academic research lab may have access to high-throughput capability in one technique but face limitations (sample preparation, resolution, access time) with other technique(s). Furthermore, one type of structural characterization data may be easier to interpret than another (e.g., microscopy images are easier to interpret than small-angle scattering profiles). Thus, it is useful to have machine learning models that can be trained on paired structural characterization data from multiple techniques (easy and difficult to interpret, fast and slow in data collection or sample preparation) so that the model can generate one set of characterization data from the other. In this paper we demonstrate one such machine learning workflow, Pair-Variational Autoencoders (PairVAE), that works with data from small-angle X-ray scattering (SAXS) that present information about bulk morphology and images from scanning electron microscopy (SEM) that present two-dimensional local structural information on the sample. Using paired SAXS and SEM data of newly observed block copolymer assembled morphologies [open access data from Doerk G. S.; et al. Sci. Adv.2023, 9 ( (2), ), eadd3687], we train our PairVAE. After successful training, we demonstrate that the PairVAE can generate SEM images of the block copolymer morphology when it takes as input that sample's corresponding SAXS 2D pattern and vice versa. This method can be extended to other soft material morphologies as well and serves as a valuable tool for easy interpretation of 2D SAXS patterns as well as an engine for generating ensembles of similar microscopy images to create a database for other downstream calculations of structure-property relationships.

Extracting nanoscale membrane morphology from single-molecule localizations

Membrane surface reconstruction at the nanometer scale is required for understanding mechanisms of subcellular shape change. This historically has been the domain of electron microscopy, but extraction of surfaces from specific labels is a difficult task in this imaging modality. Existing methods for extracting surfaces from fluorescence microscopy have poor resolution or require high-quality super-resolution data that are manually cleaned and curated. Here, we present NanoWrap, a new method for extracting surfaces from generalized single-molecule localization microscopy data. This makes it possible to study the shape of specifically labeled membranous structures inside cells. We validate NanoWrap using simulations and demonstrate its reconstruction capabilities on single-molecule localization microscopy data of the endoplasmic reticulum and mitochondria. NanoWrap is implemented in the open-source Python Microscopy Environment.

Computational approaches in rheumatic diseases - Deciphering complex spatio-temporal cell interactions

Inflammatory arthritis, including rheumatoid (RA), and psoriatic (PsA) arthritis, are clinically and immunologically heterogeneous diseases with no identified cure. Chronic inflammation of the synovial tissue ushers loss of function of the joint that severely impacts the patient's quality of life, eventually leading to disability and life-threatening comorbidities. The pathogenesis of synovial inflammation is the consequence of compounded immune and stromal cell interactions influenced by genetic and environmental factors. Deciphering the complexity of the synovial cellular landscape has accelerated primarily due to the utilisation of bulk and single cell RNA sequencing. Particularly the capacity to generate cell-cell interaction networks could reveal evidence of previously unappreciated processes leading to disease. However, there is currently a lack of universal nomenclature as a result of varied experimental and technological approaches that discombobulates the study of synovial inflammation. While spatial transcriptomic analysis that combines anatomical information with transcriptomic data of synovial tissue biopsies promises to provide more insights into disease pathogenesis, in vitro functional assays with single-cell resolution will be required to validate current bioinformatic applications. In order to provide a comprehensive approach and translate experimental data to clinical practice, a combination of clinical and molecular data with machine learning has the potential to enhance patient stratification and identify individuals at risk of arthritis that would benefit from early therapeutic intervention. This review aims to provide a comprehensive understanding of the effect of computational approaches in deciphering synovial inflammation pathogenesis and discuss the impact that further experimental and novel computational tools may have on therapeutic target identification and drug development.

Revealing the Structural Organization of Gamma-irradiated Starch Granules Using Polarization-resolved Second Harmonic Generation Microscopy

Starch is a semi-crystalline macromolecule with the presence of amorphous and crystalline components. The amorphous amylose and crystalline amylopectin regions in starch granules are susceptible to certain physical modifications, such as gamma irradiation. Polarization-resolved second harmonic generation (P-SHG) microscopy in conjunction with SHG-circular dichroism (CD) was used to assess the three-dimensional molecular order and inherent chirality of starch granules and their reaction to different dosages of gamma irradiation. For the first time, the relationship between starch achirality (χ21/χ16 and χ22/χ16) and chirality (χ14/χ16) determining susceptibility tensor ratios has been elucidated. The results showed that changes in the structure and orientation of long-chain amylopectin were supported by the decrease in the SHG anisotropy factor and the χ22/χ16 ratio. Furthermore, SHG-CD illustrated the molecular tilt angle by revealing the arrangement of amylopectin molecules pointing either upward or downward owing to molecular polarity.

Single-shot quantitative phase contrast imaging based on deep learning

Quantitative differential phase-contrast (DPC) imaging is one of the commonly used methods for phase retrieval. However, quantitative DPC imaging requires several pairwise intensity measurements, which makes it difficult to monitor living cells in real-time. In this study, we present a single-shot quantitative DPC imaging method based on the combination of deep learning (DL) and color-encoded illumination. Our goal is to train a model that can generate an isotropic quantitative phase image (i.e., target) directly from a single-shot intensity measurement (i.e., input). The target phase image was reconstructed using a linear-gradient pupil with two-axis measurements, and the model input was the measured color intensities obtained from a radially asymmetric color-encoded illumination pattern. The DL-based model was trained, validated, and tested using thirteen different cell lines. The total number of training, validation, and testing images was 264 (10 cells), 10 (1 cell), and 40 (2 cells), respectively. Our results show that the DL-based phase images are visually similar to the ground-truth phase images and have a high structural similarity index (>0.98). Moreover, the phase difference between the ground-truth and DL-based phase images was smaller than 13%. Our study shows the feasibility of using DL to generate quantitative phase imaging from a single-shot intensity measurement.

Restoration of metabolic functional metrics from label-free, two-photon cervical tissue images using multiscale deep-learning-based denoising algorithms

Label-free, two-photon imaging captures morphological and functional metabolic tissue changes and enables enhanced understanding of numerous diseases. However, this modality suffers from low signal arising from limitations imposed by the maximum permissible dose of illumination and the need for rapid image acquisition to avoid motion artifacts. Recently, deep learning methods have been developed to facilitate the extraction of quantitative information from such images. Here, we employ deep neural architectures in the synthesis of a multiscale denoising algorithm optimized for restoring metrics of metabolic activity from low-SNR, two-photon images. Two-photon excited fluorescence (TPEF) images of reduced nicotinamide adenine dinucleotide (phosphate) (NAD(P)H) and flavoproteins (FAD) from freshly excised human cervical tissues are used. We assess the impact of the specific denoising model, loss function, data transformation, and training dataset on established metrics of image restoration when comparing denoised single frame images with corresponding six frame averages, considered as the ground truth. We further assess the restoration accuracy of six metrics of metabolic function from the denoised images relative to ground truth images. Using a novel algorithm based on deep denoising in the wavelet transform domain, we demonstrate optimal recovery of metabolic function metrics. Our results highlight the promise of denoising algorithms to recover diagnostically useful information from low SNR label-free two-photon images and their potential importance in the clinical translation of such imaging.

Deep Learning Approaches to Osteosarcoma Diagnosis and Classification: A Comparative Methodological Approach

BACKGROUND

Osteosarcoma is the most common primary malignancy of the bone, being most prevalent in childhood and adolescence. Despite recent progress in diagnostic methods, histopathology remains the gold standard for disease staging and therapy decisions. Machine learning and deep learning methods have shown potential for evaluating and classifying histopathological cross-sections.

METHODS

This study used publicly available images of osteosarcoma cross-sections to analyze and compare the performance of state-of-the-art deep neural networks for histopathological evaluation of osteosarcomas.

RESULTS

The classification performance did not necessarily improve when using larger networks on our dataset. In fact, the smallest network combined with the smallest image input size achieved the best overall performance. When trained using 5-fold cross-validation, the MobileNetV2 network achieved 91% overall accuracy.

CONCLUSIONS

The present study highlights the importance of careful selection of network and input image size. Our results indicate that a larger number of parameters is not always better, and the best results can be achieved on smaller and more efficient networks. The identification of an optimal network and training configuration could greatly improve the accuracy of osteosarcoma diagnoses and ultimately lead to better disease outcomes for patients.

Extracting nanoscale membrane morphology from single-molecule localizations

Membrane surface reconstruction at the nanometer scale is required for understanding mechanisms of subcellular shape change. This historically has been the domain of electron microscopy, but extraction of surfaces from specific labels is a difficult task in this imaging modality. Existing methods for extracting surfaces from fluorescence microscopy have poor resolution or require high-quality super-resolution data that is manually cleaned and curated. Here we present a new method for extracting surfaces from generalized single-molecule localization microscopy (SMLM) data. This makes it possible to study the shape of specifically-labelled membraneous structures inside of cells. We validate our method using simulations and demonstrate its reconstruction capabilities on SMLM data of the endoplasmic reticulum and mitochondria. Our method is implemented in the open-source Python Microscopy Environment.

SIGNIFICANCE

We introduce a novel tool for reconstruction of subcellular membrane surfaces from single-molecule localization microscopy data and use it to visualize and quantify local shape and membrane-membrane interactions. We benchmark its performance on simulated data and demonstrate its fidelity to experimental data.

Combinatorial quantification of distinct neural projections from retrograde tracing

Comprehensive quantification of neuronal architectures underlying anatomical brain connectivity remains challenging. We introduce a method to identify the distinct axonal projection patterns from a source to a set of target regions and the count of neurons with each pattern. For a source region projecting to n targets, there are 2n - 1 theoretically possible projection types, although only a subset of these types typically exists. By injecting uniquely labeled retrograde tracers in k regions (k < n), one can experimentally count the cells expressing different combinations of colors in the source region1,2. Such an experiment can be performed for n choose k combinations. The counts of cells with different color combinations from all experiments provide constraints for a system of equations that include 2n - 1 unknown variables, each corresponding to the count of neurons for a projection pattern. Evolutionary algorithms prove to be effective at solving the resultant system of equations, thus allowing the determination of the counts of neurons with each of the possible projection patterns. Numerical analysis of simulated 4 choose 3 retrograde injection experiments using surrogate data demonstrates reliable and precise count estimates for all projection neuron types. We illustrate the experimental application of this framework by quantifying the projections of mouse primary motor cortex to four prominent targets: the primary and secondary somatosensory and motor cortices.

Utilizing deep learning for dermal matrix quality assessment on in vivo line-field confocal optical coherence tomography images

BACKGROUND

Line-field confocal optical coherence tomography (LC-OCT) is an imaging technique providing non-invasive "optical biopsies" with an isotropic spatial resolution of ∼1  μm and deep penetration until the dermis. Analysis of obtained images is classically performed by experts, thus requiring long and fastidious training and giving operator-dependent results. In this study, the objective was to develop a new automated method to score the quality of the dermal matrix precisely, quickly, and directly from in vivo LC-OCT images. Once validated, this new automated method was applied to assess photo-aging-related changes in the quality of the dermal matrix.

MATERIALS AND METHODS

LC-OCT measurements were conducted on the face of 57 healthy Caucasian volunteers. The quality of the dermal matrix was scored by experts trained to evaluate the fibers' state according to four grades. In parallel, these images were used to develop the deep learning model by adapting a MobileNetv3-Small architecture. Once validated, this model was applied to the study of dermal matrix changes on a panel of 36 healthy Caucasian females, divided into three groups according to their age and photo-exposition.

RESULTS

The deep learning model was trained and tested on a set of 15 993 images. Calculated on the test data set, the accuracy score was 0.83. As expected, when applied to different volunteer groups, the model shows greater and deeper alteration of the dermal matrix for old and photoexposed subjects.

CONCLUSIONS

In conclusion, we have developed a new method that automatically scores the quality of the dermal matrix on in vivo LC-OCT images. This accurate model could be used for further investigations, both in the dermatological and cosmetic fields.

Pixel-level multimodal fusion deep networks for predicting subcellular organelle localization from label-free live-cell imaging

Complex intracellular organizations are commonly represented by dividing the metabolic process of cells into different organelles. Therefore, identifying sub-cellular organelle architecture is significant for understanding intracellular structural properties, specific functions, and biological processes in cells. However, the discrimination of these structures in the natural organizational environment and their functional consequences are not clear. In this article, we propose a new pixel-level multimodal fusion (PLMF) deep network which can be used to predict the location of cellular organelle using label-free cell optical microscopy images followed by deep-learning-based automated image denoising. It provides valuable insights that can be of tremendous help in improving the specificity of label-free cell optical microscopy by using the Transformer-Unet network to predict the ground truth imaging which corresponds to different sub-cellular organelle architectures. The new prediction method proposed in this article combines the advantages of a transformer's global prediction and CNN's local detail analytic ability of background features for label-free cell optical microscopy images, so as to improve the prediction accuracy. Our experimental results showed that the PLMF network can achieve over 0.91 Pearson's correlation coefficient (PCC) correlation between estimated and true fractions on lung cancer cell-imaging datasets. In addition, we applied the PLMF network method on the cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new way for the time-resolved study of subcellular components in different cells, especially for cancer cells.

Multiple Parallel Fusion Network for Predicting Protein Subcellular Localization from Stimulated Raman Scattering (SRS) Microscopy Images in Living Cells

Stimulated Raman Scattering Microscopy (SRS) is a powerful tool for label-free detailed recognition and investigation of the cellular and subcellular structures of living cells. Determining subcellular protein localization from the cell level of SRS images is one of the basic goals of cell biology, which can not only provide useful clues for their functions and biological processes but also help to determine the priority and select the appropriate target for drug development. However, the bottleneck in predicting subcellular protein locations of SRS cell imaging lies in modeling complicated relationships concealed beneath the original cell imaging data owing to the spectral overlap information from different protein molecules. In this work, a multiple parallel fusion network, MPFnetwork, is proposed to study the subcellular locations from SRS images. This model used a multiple parallel fusion model to construct feature representations and combined multiple nonlinear decomposing algorithms as the automated subcellular detection method. Our experimental results showed that the MPFnetwork could achieve over 0.93 dice correlation between estimated and true fractions on SRS lung cancer cell datasets. In addition, we applied the MPFnetwork method to cell images for label-free prediction of several different subcellular components simultaneously, rather than using several fluorescent labels. These results open up a new method for the time-resolved study of subcellular components in different cells, especially cancer cells.

Biophysical Reviews: focusing on an issue

This Issue of Biophysical Reviews (Volume 14, Issue 2) presents a new feature known as an 'Issue Focus' - a contiguous thematic block of five articles placed within a regular Issue format. The current 'Issue Focus' is concerned with the recent developments in Costa Rican biophysical science. The regular aspect of this Issue consists of a 'Meet the Editor' piece by Sabrina Leslie, the first instalment of an ongoing Commentary feature known as the 'Editors' Roundup', and five disparate review articles covering a variety of topics.