Development of Deep-Learning-Based Single-Molecule Localization Image Analysis

Artificial Intelligence-Empowered Spectroscopic Single Molecule Localization Microscopy

Spectroscopic single-molecule localization microscopy (SMLM) has revolutionized the visualization and analysis of molecular structures and dynamics at the nanoscale level. The technique of combining high spatial resolution of SMLM with spectral information, enables multicolor super-resolution imaging and provides insights into the local chemical environment of individual molecules. However, spectroscopic SMLM faces significant challenges, including limited spectral resolution and compromised localization precision because of signal splitting and the difficulties in analyzing complex, multidimensional datasets, that limit its application in studying intricate biological systems and materials. The recent integration of artificial intelligence (AI) with spectroscopic SMLM has emerged as a powerful approach for addressing these challenges. Here, it is reviewed how AI-based methods applied to spectroscopic SMLM enhance and expand the capabilities of these applications. Recent advancements in AI-driven data analysis for spectroscopic SMLM, including improved spectral classification, localization precision, and extraction of rich spectral information from unmodified point-spread functions are discussed, further examining their applications in biological studies, materials science, and single-molecule reaction analysis, which highlight how AI provides new insights into molecular behavior and interactions. The AI-empowered approach adds new dimensions of information and provides new opportunities and insights into the nanoscale world of rapidly evolving field of spectroscopic SMLM.

Surpassing the Diffraction Limit in Label-Free Optical Microscopy

Super-resolution optical microscopy has enhanced our ability to visualize biological structures on the nanoscale. Fluorescence-based techniques are today irreplaceable in exploring the structure and dynamics of biological matter with high specificity and resolution. However, the fluorescence labeling concept narrows the range of observed interactions and fundamentally limits the spatiotemporal resolution. In contrast, emerging label-free imaging methods are not inherently limited by speed and have the potential to capture the entirety of complex biological processes and dynamics. While pushing a complex unlabeled microscopy image beyond the diffraction limit to single-molecule resolution and capturing dynamic processes at biomolecular time scales is widely regarded as unachievable, recent experimental strides suggest that elements of this vision might be already in place. These techniques derive signals directly from the sample using inherent optical phenomena, such as elastic and inelastic scattering, thereby enabling the measurement of additional properties, such as molecular mass, orientation, or chemical composition. This perspective aims to identify the cornerstones of future label-free super-resolution imaging techniques, discuss their practical applications and theoretical challenges, and explore directions that promise to enhance our understanding of complex biological systems through innovative optical advancements. Drawing on both traditional and emerging techniques, label-free super-resolution microscopy is evolving to offer detailed and dynamic imaging of living cells, surpassing the capabilities of conventional methods for visualizing biological complexities without the use of labels.

Precision lung cancer screening from CT scans using a VGG16-based convolutional neural network

Objective

The research aims to develop an advanced and precise lung cancer screening model based on Convolutional Neural Networks (CNN).

Methods

Based on the health medical big data platform of Shandong University, we developed a VGG16-Based CNN lung cancer screening model. This model was trained using the Computed Tomography scans data of patients from Pingyi Traditional Chinese Medicine Hospital in Shandong Province, from January to February 2023. Data augmentation techniques, including random resizing, cropping, horizontal flipping, color jitter, random rotation and normalization, were applied to improve model generalization. We used five-fold cross-validation to robustly assess performance. The model was fine-tuned with an SGD optimizer (learning rate 0.001, momentum 0.9, and L2 regularization) and a learning rate scheduler. Dropout layers were added to prevent the model from relying too heavily on specific neurons, enhancing its ability to generalize. Early stopping was implemented when validation loss did not decrease over 10 epochs. In addition, we evaluated the model's performance with Area Under the Curve (AUC), Classification accuracy, Positive Predictive Value (PPV), and Negative Predictive Value (NPV), Sensitivity, Specificity and F1 score. External validation used an independent dataset from the same hospital, covering January to February 2022.

Results

The training and validation loss and accuracy over iterations show that both accuracy metrics peak at over 0.9 by iteration 15, prompting early stopping to prevent overfitting. Based on five-fold cross-validation, the ROC curves for the VGG16-Based CNN model, demonstrate an AUC of 0.963 ± 0.004, highlighting its excellent diagnostic capability. Confusion matrices provide average metrics with a classification accuracy of 0.917 ± 0.004, PPV of 0.868 ± 0.015, NPV of 0.931 ± 0.003, Sensitivity of 0.776 ± 0.01, Specificity of 0.962 ± 0.005 and F1 score of 0.819 ± 0.008, respectively. External validation confirmed the model's robustness across different patient populations and imaging conditions.

Conclusion

The VGG16-Based CNN lung screening model constructed in this study can effectively identify lung tumors, demonstrating reliability and effectiveness in real-world medical settings, and providing strong theoretical and empirical support for its use in lung cancer screening.

Machine Learning for Single-Molecule Localization Microscopy: From Data Analysis to Quantification

Single-molecule localization microscopy (SMLM) is a versatile tool for realizing nanoscale imaging with visible light and providing unprecedented opportunities to observe bioprocesses. The integration of machine learning with SMLM enhances data analysis by improving efficiency and accuracy. This tutorial aims to provide a comprehensive overview of the data analysis process and theoretical aspects of SMLM, while also highlighting the typical applications of machine learning in this field. By leveraging advanced analytical techniques, SMLM is becoming a powerful quantitative analysis tool for biological research.

Simultaneous spectral differentiation of multiple fluorophores in super-resolution imaging using a glass phase plate

By engineering the point-spread function (PSF) of single molecules, different fluorophore species can be imaged simultaneously and distinguished by their unique PSF patterns. Here, we insert a silicon-dioxide phase plate at the Fourier plane of the detection path of a wide-field fluorescence microscope to produce distinguishable PSFs (X-PSFs) at different wavelengths. We demonstrate that the resulting PSFs can be localized spatially and spectrally using a maximum-likelihood estimation algorithm and can be utilized for hyper-spectral super-resolution microscopy of biological samples. We produced superresolution images of fixed U2OS cells using X-PSFs for dSTORM imaging with simultaneous illumination of up to three fluorophore species. The species were distinguished only by the PSF pattern. We achieved ∼21-nm lateral localization precision (FWHM) and ∼17-nm axial precision (FWHM) with an average of 1,800 - 3,500 photons per PSF and a background as high as 130 - 400 photons per pixel. The modified PSF distinguished fluorescent probes with ∼80 nm separation between spectral peaks.

Investigating Uncertainties in Single-Molecule Localization Microscopy Using Experimentally Informed Monte Carlo Simulation

Single-molecule localization microscopy (SMLM) enables the visualization of cellular nanostructures in vitro with sub-20 nm resolution. While substructures can generally be imaged with SMLM, the structural understanding of the images remains elusive. To better understand the link between SMLM images and the underlying structure, we developed a Monte Carlo (MC) simulation based on experimental imaging parameters and geometric information to generate synthetic SMLM images. We chose the nuclear pore complex (NPC), a nanosized channel on the nuclear membrane which gates nucleo-cytoplasmic transport of biomolecules, as a test geometry for testing our MC model. Using the MC model to simulate SMLM images, we first optimized our clustering algorithm to separate >106 molecular localizations of fluorescently labeled NPC proteins into hundreds of individual NPCs in each cell. We then illustrated using our MC model to generate cellular substructures with different angles of labeling to inform our structural understanding through the SMLM images obtained.

Advancing Spectrally-Resolved Single Molecule Localization Microscopy with Deep Learning

Spectrally-resolved single molecule localization microscopy (srSMLM) is a recent technique enriching single molecule localization microscopy with the simultaneous recording of spectra of the single emitters. srSMLM resolution is limited by the number of photons collected per emitters. Sharing a photon budget to record the localization and the spectroscopic information results in a loss of spatial and spectral resolution-or forces the sacrifice of one at the expense of the other. Here, srUnet-a deep-learning Unet-based image processing routine trained to increase the spectral and spatial signals to compensate for the resolution loss inherent in additionally recording the spectral component is reported. Both localization and spectral precision are improved by srUnet-particularly for the low-emitting species. srUnet increases the fraction of localization whose signal can be both spatially and spectrally characterized. It preserves spectral shifts and the linearity of the dispersion of light. It strongly facilitates wavelength assignment in multicolor experiments. srUnet is a simple post-processing add-on boosting srSMLM performance close to conventional SMLM with the potential to turn srSMLM into the new standard for multicolor single molecule imaging.

Recent development of computational cluster analysis methods for single-molecule localization microscopy images

With the development of super-resolution imaging techniques, it is crucial to understand protein structure at the nanoscale in terms of clustering and organization in a cell. However, cluster analysis from single-molecule localization microscopy (SMLM) images remains challenging because the classical computational cluster analysis methods developed for conventional microscopy images do not apply to pointillism SMLM data, necessitating the development of distinct methods for cluster analysis from SMLM images. In this review, we discuss the development of computational cluster analysis methods for SMLM images by categorizing them into classical and machine-learning-based methods. Finally, we address possible future directions for machine learning-based cluster analysis methods for SMLM data.

Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.

Polarity Nano-Mapping of Polymer Film Using Spectrally Resolved Super-Resolution Imaging

With the rapid development of the nanofabrication of polymer materials, the local measurement of the chemical properties of polymer nanostructures has become crucial because they can be highly heterogeneous at the nanoscale. We developed a spectroscopic imaging approach to characterize the nanoscale local polarity of polymer films via spectrally resolved super-resolution microscopy. We demonstrate the capability of the recently developed single-molecule sensing and imaging method to probe the polarity of polymers either inside a polymer matrix or on the external surface of a polymer. The nanoscale polarity sensing capability of our method facilitates the differentiation of various polymer surfaces based on chemical polarities, and it can further differentiate the polarity of functional side chain groups. Moreover, we demonstrate that a two-component polymer mixture can be locally distinguished based on the contrasting polarities of the lateral phase separation, further allowing for the investigation of nanoscale phase separation depending on the composition of the polymer blend film. This approach is anticipated to open the door to further characterizations of various nanocomposite materials.