Automated structure discovery in atomic force microscopy

Analysis of biofilm assembly by large area automated AFM

Biofilms are complex microbial communities critical in medical, industrial, and environmental contexts. Understanding their assembly, structure, genetic regulation, interspecies interactions, and environmental responses is key to developing effective control and mitigation strategies. While atomic force microscopy (AFM) offers critically important high-resolution insights on structural and functional properties at the cellular and even sub-cellular level, its limited scan range and labor-intensive nature restricts the ability to link these smaller scale features to the functional macroscale organization of the films. We begin to address this limitation by introducing an automated large area AFM approach capable of capturing high-resolution images over millimeter-scale areas, aided by machine learning for seamless image stitching, cell detection, and classification. Large area AFM is shown to provide a very detailed view of spatial heterogeneity and cellular morphology during the early stages of biofilm formation which were previously obscured. Using this approach, we examined the organization of Pantoea sp. YR343 on PFOTS-treated glass surfaces. Our findings reveal a preferred cellular orientation among surface-attached cells, forming a distinctive honeycomb pattern. Detailed mapping of flagella interactions suggests that flagellar coordination plays a role in biofilm assembly beyond initial attachment. Additionally, we use large-area AFM to characterize surface modifications on silicon substrates, observing a significant reduction in bacterial density. This highlights the potential of this method for studying surface modifications to better understand and control bacterial adhesion and biofilm formation.

Construction of metal-organic nanostructures and their structural transformations on metal surfaces

Metal-organic nanostructures, composed of organic molecules as building blocks and metal atoms as linkers, exhibit high reversibility and flexibility and open up new vistas for the creation of novel metal-organic nanomaterials and the fabrication of functional molecule-based nanodevices. With the rapid development of emerging surface science and scanning probe microscopy, various metal-organic nanostructures, ranging from zero to two dimensions, have been prepared with atomic precision on well-defined metal surfaces in a bottom-up manner and further visualized at the submolecular (or even atomic) level. In such processes, the metal-organic interactions involved and the synergy and competition of multiple intermolecular interactions have been clearly discriminated as the cause of the diversity and preference of metal-organic nanostructures. Moreover, structural transformations can be controllably directed by subtly tuning such intermolecular interactions. In this perspective, we review recent exciting progress in the construction of metal-organic nanostructures on metal surfaces ranging from zero to two dimensions, which is mainly in terms of the selection of metal types (including sources), in other words, different metal-organic interactions formed. Subsequently, the corresponding structural transformations in response to internal or external conditions are discussed, providing mechanistic insights into precise structural control, e.g., by means of metal/molecule stoichiometric ratios (including through scanning probe microscopy (SPM) manipulations), thermodynamic control, introduction of extrinsic competing counterparts, etc. In addition, some other regulatory factors, such as the functionalization of organic molecules and the choice of substrates and lattices, which also crucially govern the structural transformations, are briefly mentioned in each part. Finally, some potential perspectives for metal-organic nanostructures are evoked.

Direct Imaging of Chirality Transfer Induced by Glycosidic Bond Stereochemistry in Carbohydrate Self-Assemblies

Carbohydrates, essential biological building blocks, exhibit functional mechanisms tied to their intricate stereochemistry. Subtle stereochemical differences, such as those between the anomers maltose and cellobiose, lead to distinct properties due to their differing glycosidic bonds; the former is digestible by humans, while the latter is not. This underscores the importance of precise structural determination of individual carbohydrate molecules for deeper functional insights. However, their structural complexity and conformational flexibility, combined with the high spatial resolution needed, have hindered direct imaging of carbohydrate stereochemistry. Here, we employ noncontact atomic force microscopy integrated with a data-efficient, multifidelity structure search approach accelerated by machine learning integration to determine the precise 3D atomic coordinates of two carbohydrate anomers on Au(111). We observe that the stereochemistry of the glycosidic bond regulates on-surface chiral selection in carbohydrate self-assemblies. The reconstructed models, validated against experimental data, provide reliable atomic-scale structural evidence, uncovering the origin of the on-surface chirality from carbohydrate anomerism. Our study confirms that nc-AFM is a reliable technique for real-space discrimination of carbohydrate stereochemistry at the single-molecule level, providing a pathway for bottom-up investigations into the structure-property relationships of carbohydrates in biological research and materials science.

Advancing High-Throughput Cellular Atomic Force Microscopy with Automation and Artificial Intelligence

Atomic force microscopy (AFM) has reached a significant level of maturity in biology, demonstrated by the diversity of modes for obtaining not only topographical images but also insightful mechanical and adhesion data by performing force measurements on delicate samples with a controlled environment (e.g., liquid, temperature, pH). Numerous studies have applied AFM to describe biological phenomena at the molecular and cellular scales, and even on tissues. Despite these advances, AFM is not established as a diagnostic tool in the biomedical field. This article describes the reasons for this gap, focusing on one of the main weaknesses of bio-AFM: its low data throughput. We review current efforts to improve the automation of AFM measurements in particular on living cells, as well as the developments in automating data analysis. For the latter, artificial intelligence (AI) is progressively employed to classify data to distinguish healthy and diseased cells or tissues. Finally, we propose a roadmap to foster the application of bio-AFM into medical diagnostics.

Imaging Locally Inhomogeneous Properties of Metal Halide Perovskites

Metal halide perovskites (MHPs) are a perfect example of state-of-the-art photovoltaic materials whose compositional and structural diversity, coupled with utilization of low-temperature processing, can undesirably result in spatially inhomogeneous properties that locally vary within the material. This complexity of MHPs requires sensitive imaging characterization methods at the microscopic level to gauge the impact of such inhomogeneities on device performance and to formulate mitigation strategies. This review consolidates properties of MHPs that are susceptible to local variations and highlights appropriate imaging techniques that can be employed to map them. Inhomogeneities in morphology, emission, electrical response, and chemical composition of MHP thin films are specifically considered, and possible microscopic techniques for their visualization are reviewed. For each type of microscopy, a short discussion about spatial resolution, sample requirements, advantages, and limitations is provided, thus leaving the reader with a guide of available imaging characterization tools to evaluate inhomogeneities of their MHPs.

Molecular identification via molecular fingerprint extraction from atomic force microscopy images

Non-Contact Atomic Force Microscopy with CO-functionalized metal tips (referred to as HR-AFM) provides access to the internal structure of individual molecules adsorbed on a surface with totally unprecedented resolution. Previous works have shown that deep learning (DL) models can retrieve the chemical and structural information encoded in a 3D stack of constant-height HR-AFM images, leading to molecular identification. In this work, we overcome their limitations by using a well-established description of the molecular structure in terms of topological fingerprints, the 1024-bit Extended Connectivity Chemical Fingerprints of radius 2 (ECFP4), that were developed for substructure and similarity searching. ECFPs provide local structural information of the molecule, each bit correlating with a particular substructure within the molecule. Our DL model is able to extract this optimized structural descriptor from the 3D HR-AFM stacks and use it, through virtual screening, to identify molecules from their predicted ECFP4 with a retrieval accuracy on theoretical images of 95.4%. Furthermore, this approach, unlike previous DL models, assigns a confidence score, the Tanimoto similarity, to each of the candidate molecules, thus providing information on the reliability of the identification. By construction, the number of times a certain substructure is present in the molecule is lost during the hashing process, necessary to make them useful for machine learning applications. We show that it is possible to complement the fingerprint-based virtual screening with global information provided by another DL model that predicts from the same HR-AFM stacks the chemical formula, boosting the identification accuracy up to a 97.6%. Finally, we perform a limited test with experimental images, obtaining promising results towards the application of this pipeline under real conditions.Scientific contributionPrevious works on molecular identification from AFM images used chemical descriptors that were intuitive for humans but sub-optimal for neural networks. We propose a novel method to extract the ECFP4 from AFM images and identify the molecule via a virtual screening, beating previous state-of-the-art models.

Scanning Probe Microscopy Characterization of Biomolecules enabled by Mass-Selective, Soft-landing Electrospray Ion Beam Deposition

Scanning probe microscopy (SPM), in particular at low temperature (LT) under ultra-high vacuum (UHV) conditions, offers the possibility of real-space imaging with resolution reaching the atomic level. However, its potential for the analysis of complex biological molecules has been hampered by requirements imposed by sample preparation. Transferring molecules onto surfaces in UHV is typically accomplished by thermal sublimation in vacuum. This approach however is limited by the thermal stability of the molecules, i. e. not possible for biological molecules with low vapour pressure. Bypassing this limitation, electrospray ionisation offers an alternative method to transfer molecules from solution to the gas-phase as intact molecular ions. In soft-landing electrospray ion beam deposition (ESIBD), these molecular ions are subsequently mass-selected and gently landed on surfaces which permits large and thermally fragile molecules to be analyzed by LT-UHV SPM. In this concept, we discuss how ESIBD+SPM prepares samples of complex biological molecules at a surface, offering controls of the molecular structural integrity, three-dimensional shape, and purity. These achievements unlock the analytical potential of SPM which is showcased by imaging proteins, peptides, DNA, glycans, and conjugates of these molecules, revealing details of their connectivity, conformation, and interaction that could not be accessed by any other technique.

How scanning probe microscopy can be supported by artificial intelligence and quantum computing?

The impact of Artificial Intelligence (AI) is rapidly expanding, revolutionizing both science and society. It is applied to practically all areas of life, science, and technology, including materials science, which continuously requires novel tools for effective materials characterization. One of the widely used techniques is scanning probe microscopy (SPM). SPM has fundamentally changed materials engineering, biology, and chemistry by providing tools for atomic-precision surface mapping. Despite its many advantages, it also has some drawbacks, such as long scanning times or the possibility of damaging soft-surface materials. In this paper, we focus on the potential for supporting SPM-based measurements, with an emphasis on the application of AI-based algorithms, especially Machine Learning-based algorithms, as well as quantum computing (QC). It has been found that AI can be helpful in automating experimental processes in routine operations, algorithmically searching for optimal sample regions, and elucidating structure-property relationships. Thus, it contributes to increasing the efficiency and accuracy of optical nanoscopy scanning probes. Moreover, the combination of AI-based algorithms and QC may have enormous potential to enhance the practical application of SPM. The limitations of the AI-QC-based approach were also discussed. Finally, we outline a research path for improving AI-QC-powered SPM. RESEARCH HIGHLIGHTS: Artificial intelligence and quantum computing as support for scanning probe microscopy. The analysis indicates a research gap in the field of scanning probe microscopy. The research aims to shed light into ai-qc-powered scanning probe microscopy.

Automated Bio-AFM Generation of Large Mechanome Data Set and Their Analysis by Machine Learning to Classify Cancerous Cell Lines

Mechanobiological measurements have the potential to discriminate healthy cells from pathological cells. However, a technology frequently used to measure these properties, i.e., atomic force microscopy (AFM), suffers from its low output and lack of standardization. In this work, we have optimized AFM mechanical measurement on cell populations and developed a technology combining cell patterning and AFM automation that has the potential to record data on hundreds of cells (956 cells measured for publication). On each cell, 16 force curves (FCs) and seven features/FC, constituting the mechanome, were calculated. All of the FCs were then classified using machine learning tools with a statistical approach based on a fuzzy logic algorithm, trained to discriminate between nonmalignant and cancerous cells (training base, up to 120 cells/cell line). The proof of concept was first made on prostate nonmalignant (RWPE-1) and cancerous cell lines (PC3-GFP), then on nonmalignant (Hs 895.Sk) and cancerous (Hs 895.T) skin fibroblast cell lines, and demonstrated the ability of our method to classify correctly 73% of the cells (194 cells in the database/cell line) despite the very high degree of similarity of the whole set of measurements (79-100% similarity).

Benchmarking Unsupervised Clustering Algorithms for Atomic Force Microscopy Data on Polyhydroxyalkanoate Films

Surface of polyhydroxyalkanoate (PHA) films of varying monomer compositions are analyzed using atomic force microscopy (AFM) and unsupervised machine learning (ML) algorithms to investigate and classify films based on global attributes such as the scan size, film thickness, and monomer type. The experiment provides benchmarked results for 12 of the most widely used clustering algorithms via a hybrid investigation approach while highlighting the impact of using the Fourier transform (FT) on high-dimensional vectorized data for classification on various pools of data. Our findings indicate that the use of a one-dimensional (1D) FT of vectorized data produces the most accurate outcome. The experiment also provides insights into case-by-case investigations of algorithm performances and the impact of various data pools. Lastly, we show an early version of our tool aimed at investigating surfaces using ML approaches and discuss the results of our current experiment to configure future improvements.

Automated Structure Discovery for Scanning Tunneling Microscopy

Scanning tunneling microscopy (STM) with a functionalized tip apex reveals the geometric and electronic structures of a sample within the same experiment. However, the complex nature of the signal makes images difficult to interpret and has so far limited most research to planar samples with a known chemical composition. Here, we present automated structure discovery for STM (ASD-STM), a machine learning tool for predicting the atomic structure directly from an STM image, by building upon successful methods for structure discovery in noncontact atomic force microscopy (nc-AFM). We apply the method on various organic molecules and achieve good accuracy on structure predictions and chemical identification on a qualitative level while highlighting future development requirements for ASD-STM. This method is directly applicable to experimental STM images of organic molecules, making structure discovery available for a wider scanning probe microscopy audience outside of nc-AFM. This work also allows more advanced machine learning methods to be developed for STM structure discovery.

On machine learning analysis of atomic force microscopy images for image classification, sample surface recognition

Atomic force microscopy (AFM or SPM) imaging is one of the best matches with machine learning (ML) analysis among microscopy techniques. The digital format of AFM images allows for direct utilization in ML algorithms without the need for additional processing. Additionally, AFM enables the simultaneous imaging of distributions of over a dozen different physicochemical properties of sample surfaces, a process known as multidimensional imaging. While this wealth of information can be challenging to analyze using traditional methods, ML provides a seamless approach to this task. However, the relatively slow speed of AFM imaging poses a challenge in applying deep learning methods broadly used in image recognition. This prospective is focused on ML recognition/classification when using a relatively small number of AFM images, aka small database. We discuss ML methods other than popular deep-learning neural networks. The described approach has already been successfully used to analyze and classify the surfaces of biological cells. It can be applied to recognize medical images, specific material processing, in forensic studies, even to identify the authenticity of arts. A general template for ML analysis specific to AFM is suggested, with a specific example of the identification of cell phenotype. Special attention is given to the analysis of the statistical significance of the obtained results, an important feature that is often overlooked in papers dealing with machine learning. A simple method for finding statistical significance is also described.

Precise Surface Profiling at the Nanoscale Enabled by Deep Learning

Surface topography, or height profile, is a critical property for various micro- and nanostructured materials and devices, as well as biological systems. At the nanoscale, atomic force microscopy (AFM) is the tool of choice for surface profiling due to its capability to noninvasively map the topography of almost all types of samples. However, this method suffers from one drawback: the convolution of the nanoprobe's shape in the height profile of the samples, which is especially severe for sharp protrusion features. Here, we report a deep learning (DL) approach to overcome this limit. Adopting an image-to-image translation methodology, we use data sets of tip-convoluted and deconvoluted image pairs to train an encoder-decoder based deep convolutional neural network. The trained network successfully removes the tip convolution from AFM topographic images of various nanocorrugated surfaces and recovers the true, precise 3D height profiles of these samples.

Structure Discovery in Atomic Force Microscopy Imaging of Ice

The interaction of water with surfaces is crucially important in a wide range of natural and technological settings. In particular, at low temperatures, unveiling the atomistic structure of adsorbed water clusters would provide valuable data for understanding the ice nucleation process. Using high-resolution atomic force microscopy (AFM) and scanning tunneling microscopy, several studies have demonstrated the presence of water pentamers, hexamers, and heptamers (and of their combinations) on a variety of metallic surfaces, as well as the initial stages of 2D ice growth on an insulating surface. However, in all of these cases, the observed structures were completely flat, providing a relatively straightforward path to interpretation. Here, we present high-resolution AFM measurements of several water clusters on Au(111) and Cu(111), whose understanding presents significant challenges due to both their highly 3D configuration and their large size. For each of them, we use a combination of machine learning, atomistic modeling with neural network potentials, and statistical sampling to propose an underlying atomic structure, finally comparing its AFM simulated images to the experimental ones. These results provide insights into the early phases of ice formation, which is a ubiquitous phenomenon ranging from biology to astrophysics.

Machine learning-enabled autonomous operation for atomic force microscopes

The use of scientific instruments generally requires prior knowledge and skill on the part of operators, and thus, the obtained results often vary with different operators. The autonomous operation of instruments producing reproducible and reliable results with little or no operator-to-operator variation could be of considerable benefit. Here, we demonstrate the autonomous operation of an atomic force microscope using a machine learning-based object detection technique. The developed atomic force microscope was able to autonomously perform instrument initialization, surface imaging, and image analysis. Two cameras were employed, and a machine-learning algorithm of region-based convolutional neural networks was implemented, to detect and recognize objects of interest and to perform self-calibration, alignment, and operation of each part of the instrument, as well as the analysis of obtained images. Our machine learning-based approach could be generalized to apply to various types of scanning probe microscopes and other scientific instruments.

Direct observation of glycans bonded to proteins and lipids at the single-molecule level

Proteins and lipids decorated with glycans are found throughout biological entities, playing roles in biological functions and dysfunctions. Current analytical strategies for these glycan-decorated biomolecules, termed glycoconjugates, rely on ensemble-averaged methods that do not provide a full view of positions and structures of glycans attached at individual sites in a given molecule, especially for glycoproteins. We show single-molecule analysis of glycoconjugates by direct imaging of individual glycoconjugate molecules using low-temperature scanning tunneling microscopy. Intact glycoconjugate ions from electrospray are soft-landed on a surface for their direct single-molecule imaging. The submolecular imaging resolution corroborated by quantum mechanical modeling unveils whole structures and attachment sites of glycans in glycopeptides, glycolipids, N-glycoproteins, and O-glycoproteins densely decorated with glycans.

Visualizing Chiral Interactions in Carbohydrates Adsorbed on Au(111) by High-Resolution STM Imaging

Carbohydrates are the most abundant organic material on Earth and the structural "material of choice" in many living systems. Nevertheless, design and engineering of synthetic carbohydrate materials presently lag behind that for protein and nucleic acids. Bottom-up engineering of carbohydrate materials demands an atomic-level understanding of their molecular structures and interactions in condensed phases. Here, high-resolution scanning tunneling microscopy (STM) is used to visualize at submolecular resolution the three-dimensional structure of cellulose oligomers assembled on Au(1111) and the interactions that drive their assembly. The STM imaging, supported by ab initio calculations, reveals the orientation of all glycosidic bonds and pyranose rings in the oligomers, as well as details of intermolecular interactions between the oligomers. By comparing the assembly of D- and L-oligomers, these interactions are shown to be enantioselective, capable of driving spontaneous enantioseparation of cellulose chains from its unnatural enantiomer and promoting the formation of engineered carbohydrate assemblies in the condensed phases.

Development of Single Molecule Techniques for Sensing and Manipulation of CRISPR and Polymerase Enzymes

Clustered regularly interspaced short palindromic repeats (CRISPR) and polymerases are powerful enzymes and their diverse applications in genomics, proteomics, and transcriptomics have revolutionized the biotechnology industry today. CRISPR has been widely adopted for genomic editing applications and Polymerases can efficiently amplify genomic transcripts via polymerase chain reaction (PCR). Further investigations into these enzymes can reveal specific details about their mechanisms that greatly expand their use. Single-molecule techniques are an effective way to probe enzymatic mechanisms because they may resolve intermediary conformations and states with greater detail than ensemble or bulk biosensing techniques. This review discusses various techniques for sensing and manipulation of single biomolecules that can help facilitate and expedite these discoveries. Each platform is categorized as optical, mechanical, or electronic. The methods, operating principles, outputs, and utility of each technique are briefly introduced, followed by a discussion of their applications to monitor and control CRISPR and Polymerases at the single molecule level, and closing with a brief overview of their limitations and future prospects.

Machine Learning for Optical Scanning Probe Nanoscopy

The ability to perform nanometer-scale optical imaging and spectroscopy is key to deciphering the low-energy effects in quantum materials, as well as vibrational fingerprints in planetary and extraterrestrial particles, catalytic substances, and aqueous biological samples. These tasks can be accomplished by the scattering-type scanning near-field optical microscopy (s-SNOM) technique that has recently spread to many research fields and enabled notable discoveries. Herein, it is shown that the s-SNOM, together with scanning probe research in general, can benefit in many ways from artificial-intelligence (AI) and machine-learning (ML) algorithms. Augmented with AI- and ML-enhanced data acquisition and analysis, scanning probe optical nanoscopy is poised to become more efficient, accurate, and intelligent.

Applying a Deep-Learning-Based Keypoint Detection in Analyzing Surface Nanostructures

Scanning tunneling microscopy (STM) imaging has been routinely applied in studying surface nanostructures owing to its capability of acquiring high-resolution molecule-level images of surface nanostructures. However, the image analysis still heavily relies on manual analysis, which is often laborious and lacks uniform criteria. Recently, machine learning has emerged as a powerful tool in material science research for the automatic analysis and processing of image data. In this paper, we propose a method for analyzing molecular STM images using computer vision techniques. We develop a lightweight deep learning framework based on the YOLO algorithm by labeling molecules with its keypoints. Our framework achieves high efficiency while maintaining accuracy, enabling the recognitions of molecules and further statistical analysis. In addition, the usefulness of this model is exemplified by exploring the length of polyphenylene chains fabricated from on-surface synthesis. We foresee that computer vision methods will be frequently used in analyzing image data in the field of surface chemistry.

Machine learning-aided atomic structure identification of interfacial ionic hydrates from AFM images

Relevant to broad applied fields and natural processes, interfacial ionic hydrates have been widely studied by using ultrahigh-resolution atomic force microscopy (AFM). However, the complex relationship between the AFM signal and the investigated system makes it difficult to determine the atomic structure of such a complex system from AFM images alone. Using machine learning, we achieved precise identification of the atomic structures of interfacial water/ionic hydrates based on AFM images, including the position of each atom and the orientations of water molecules. Furthermore, it was found that structure prediction of ionic hydrates can be achieved cost-effectively by transfer learning using neural network trained with easily available interfacial water data. Thus, this work provides an efficient and economical methodology that not only opens up avenues to determine atomic structures of more complex systems from AFM images, but may also help to interpret other scientific studies involving sophisticated experimental results.

Molecular Identification from AFM Images Using the IUPAC Nomenclature and Attribute Multimodal Recurrent Neural Networks

Spectroscopic methods─like nuclear magnetic resonance, mass spectrometry, X-ray diffraction, and UV/visible spectroscopies─applied to molecular ensembles have so far been the workhorse for molecular identification. Here, we propose a radically different chemical characterization approach, based on the ability of noncontact atomic force microscopy with metal tips functionalized with a CO molecule at the tip apex (referred as HR-AFM) to resolve the internal structure of individual molecules. Our work demonstrates that a stack of constant-height HR-AFM images carries enough chemical information for a complete identification (structure and composition) of quasiplanar organic molecules, and that this information can be retrieved using machine learning techniques that are able to disentangle the contribution of chemical composition, bond topology, and internal torsion of the molecule to the HR-AFM contrast. In particular, we exploit multimodal recurrent neural networks (M-RNN) that combine convolutional neural networks for image analysis and recurrent neural networks to deal with language processing, to formulate the molecular identification as an imaging captioning problem. The algorithm is trained using a data set─which contains almost 700,000 molecules and 165 million theoretical AFM images─to produce as final output the IUPAC name of the imaged molecule. Our extensive test with theoretical images and a few experimental ones shows the potential of deep learning algorithms in the automatic identification of molecular compounds by AFM. This achievement supports the development of on-surface synthesis and overcomes some limitations of spectroscopic methods in traditional solution-based synthesis.

Correction of AFM data artifacts using a convolutional neural network trained with synthetically generated data

AFM microscopy from its nature produces outputs with certain distortions, inaccuracies and errors given by its physical principle. These distortions are more or less well studied and documented. Based on the nature of the individual distortions, different reconstruction and compensation filters have been developed to post-process the scanned images. This article presents an approach based on machine learning - the involved convolutional neural network learns from pairs of distorted images and the ground truth image and then it is able to process pairs of images of interest and produce a filtered image with the artifacts removed or at least suppressed. What is important in our approach is that the neural network is trained purely on synthetic data generated by a simulator of the inputs, based on an analytical description of the physical phenomena causing the distortions. The generator produces training samples involving various combinations of the distortions. The resulting trained network seems to be able to autonomously recognize the distortions present in the testing image (no knowledge of the distortions or any other human knowledge is provided at the test time) and apply the appropriate corrections. The experimental results show that not only is the new approach better or at least on par with conventional post-processing methods, but more importantly, it does not require any operator's input and works completely autonomously. The source codes of the training set generator and of the convolutional neural net model are made public, as well as an evaluation dataset of real captured AFM images.

DNA Origami Nanostructure Detection and Yield Estimation Using Deep Learning

DNA origami is a milestone in DNA nanotechnology. It is robust and efficient in constructing arbitrary two- and three-dimensional nanostructures. The shape and size of origami structures vary. To characterize them, an atomic force microscope, a transmission electron microscope, and other microscopes are utilized. However, the identification of various origami nanostructures heavily depends on the experience of researchers. In this study, we used the deep learning method (improved Yolox) to detect multiple DNA origami structures and estimate their yield. We designed a feature enhancement fusion network with the attention mechanism, and related parameters were researched. Experiments conducted to verify the proposed method showed that the detection accuracy was higher than that of other methods. This method can detect and estimate the DNA origami yield in complex environments, and the detection speed is in the millisecond range.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

Machine Learning-Assisted Identification of Copolymer Microstructures Based on Microscopic Images

The microstructure of polymer materials is an important bridge between their molecular structure and macroproperties, which is of great significance to be effectively identified. With the increasing refinement of polymer material design, the microstructure of different polymer materials gradually converges, which is difficult to distinguish. In this study, the machine learning method is applied to recognize the microstructure. A highly accurate and interpretable model based on small experimental data sets has been completed by the methods of transfer learning and feature visualization, making the result of the model that can be explained from the perspective of physical chemistry. This work provides an idea for identifying microstructure and will help further promote intelligent polymer research and development.

Chemical bond imaging using torsional and flexural higher eigenmodes of qPlus sensors

Non-contact atomic force microscopy (AFM) with CO-functionalized tips allows visualization of the chemical structure of adsorbed molecules and identify individual inter- and intramolecular bonds. This technique enables in-depth studies of on-surface reactions and self-assembly processes. Herein, we analyze the suitability of qPlus sensors, which are commonly used for such studies, for the application of modern multifrequency AFM techniques. Two different qPlus sensors were tested for submolecular resolution imaging via actuating torsional and flexural higher eigenmodes and via bimodal AFM. The torsional eigenmode of one of our sensors is perfectly suited for performing lateral force microscopy (LFM) with single bond resolution. The obtained LFM images agree well with images from the literature, which were scanned with customized qPlus sensors that were specifically designed for LFM. The advantage of using a torsional eigenmode is that the same molecule can be imaged either with a vertically or laterally oscillating tip without replacing the sensor simply by actuating a different eigenmode. Submolecular resolution is also achieved by actuating the 2^nd flexural eigenmode of our second sensor. In this case, we observe particular contrast features that only appear in the AFM images of the 2^nd flexural eigenmode but not for the fundamental eigenmode. With complementary laser Doppler vibrometry measurements and AFM simulations we can rationalize that these contrast features are caused by a diagonal (i.e. in-phase vertical and lateral) oscillation of the AFM tip.

QUAM-AFM: A Free Database for Molecular Identification by Atomic Force Microscopy

This paper introduces Quasar Science Resources-Autonomous University of Madrid atomic force microscopy image data set (QUAM-AFM), the largest data set of simulated atomic force microscopy (AFM) images generated from a selection of 685,513 molecules that span the most relevant bonding structures and chemical species in organic chemistry. QUAM-AFM contains, for each molecule, 24 3D image stacks, each consisting of constant-height images simulated for 10 tip-sample distances with a different combination of AFM operational parameters, resulting in a total of 165 million images with a resolution of 256 × 256 pixels. The 3D stacks are especially appropriate to tackle the goal of the chemical identification within AFM experiments by using deep learning techniques. The data provided for each molecule include, besides a set of AFM images, ball-and-stick depictions, IUPAC names, chemical formulas, atomic coordinates, and map of atom heights. In order to simplify the use of the collection as a source of information, we have developed a graphical user interface that allows the search for structures by CID number, IUPAC name, or chemical formula.

Visualization and identification of single meteoritic organic molecules by atomic force microscopy

Using high-resolution atomic force microscopy (AFM) with CO-functionalized tips, we atomically resolved individual molecules from Murchison meteorite samples. We analyzed powdered Murchison meteorite material directly, as well as processed extracts that we prepared to facilitate characterization by AFM. From the untreated Murchison sample, we resolved very few molecules, as the sample contained mostly small molecules that could not be identified by AFM. By contrast, using a procedure based on several trituration and extraction steps with organic solvents, we isolated a fraction enriched in larger organic compounds. The treatment increased the fraction of molecules that could be resolved by AFM, allowing us to identify organic constituents and molecular moieties, such as polycyclic aromatic hydrocarbons and aliphatic chains. The AFM measurements are complemented by high-resolution mass spectrometry analysis of Murchison fractions. We provide a proof of principle that AFM can be used to image and identify individual organic molecules from meteorites and propose a method for extracting and preparing meteorite samples for their investigation by AFM. We discuss the challenges and prospects of this approach to study extraterrestrial samples based on single-molecule identification.

Electrostatic Discovery Atomic Force Microscopy

While offering high resolution atomic and electronic structure, scanning probe microscopy techniques have found greater challenges in providing reliable electrostatic characterization on the same scale. In this work, we offer electrostatic discovery atomic force microscopy, a machine learning based method which provides immediate maps of the electrostatic potential directly from atomic force microscopy images with functionalized tips. We apply this to characterize the electrostatic properties of a variety of molecular systems and compare directly to reference simulations, demonstrating good agreement. This approach offers reliable atomic scale electrostatic maps on any system with minimal computational overhead.

Biosensing, Characterization of Biosensors, and Improved Drug Delivery Approaches Using Atomic Force Microscopy: A Review

Since its invention, atomic force microscopy (AFM) has come forth as a powerful member of the “scanning probe microscopy” (SPM) family and an unparallel platform for high-resolution imaging and characterization for inorganic and organic samples, especially biomolecules, biosensors, proteins, DNA, and live cells. AFM characterizes any sample by measuring interaction force between the AFM cantilever tip (the probe) and the sample surface, and it is advantageous over other SPM and electron micron microscopy techniques as it can visualize and characterize samples in liquid, ambient air, and vacuum. Therefore, it permits visualization of three-dimensional surface profiles of biological specimens in the near-physiological environment without sacrificing their native structures and functions and without using laborious sample preparation protocols such as freeze-drying, staining, metal coating, staining, or labeling. Biosensors are devices comprising a biological or biologically extracted material (assimilated in a physicochemical transducer) that are utilized to yield electronic signal proportional to the specific analyte concentration. These devices utilize particular biochemical reactions moderated by isolated tissues, enzymes, organelles, and immune system for detecting chemical compounds via thermal, optical, or electrical signals. Other than performing high-resolution imaging and nanomechanical characterization (e.g., determining Young’s modulus, adhesion, and deformation) of biosensors, AFM cantilever (with a ligand functionalized tip) can be transformed into a biosensor (microcantilever-based biosensors) to probe interactions with a particular receptors of choice on live cells at a single-molecule level (using AFM-based single-molecule force spectroscopy techniques) and determine interaction forces and binding kinetics of ligand receptor interactions. Targeted drug delivery systems or vehicles composed of nanoparticles are crucial in novel therapeutics. These systems leverage the idea of targeted delivery of the drug to the desired locations to reduce side effects. AFM is becoming an extremely useful tool in figuring out the topographical and nanomechanical properties of these nanoparticles and other drug delivery carriers. AFM also helps determine binding probabilities and interaction forces of these drug delivery carriers with the targeted receptors and choose the better agent for drug delivery vehicle by introducing competitive binding. In this review, we summarize contributions made by us and other researchers so far that showcase AFM as biosensors, to characterize other sensors, to improve drug delivery approaches, and to discuss future possibilities.

Accelerating AFM Characterization via Deep-Learning-Based Image Super-Resolution

Atomic force microscopy (AFM) is one of the most popular imaging and characterizing methods applicable to a wide range of nanoscale material systems. However, high-resolution imaging using AFM generally suffers from a low scanning yield due to its method of raster scanning. Here, a systematic method of data acquisition and preparation combined with a deep-learning-based image super-resolution, enabling rapid AFM characterization with accuracy, is proposed. Its application to measuring the geometrical and mechanical properties of structured DNA assemblies reveals that around a tenfold reduction in AFM imaging time can be achieved without significant loss of accuracy. Through a transfer learning strategy, it can be efficiently customized for a specific target sample on demand.

Hydrogen bonded trimesic acid networks on Cu(111) reveal how basic chemical properties are imprinted in HR-AFM images

High resolution non-contact atomic force microscopy measurements characterize assemblies of trimesic acid molecules on Cu(111) and the link group interactions, providing the first fingerprints utilizing CO-based probes for this widely studied paradigm for hydrogen bond driven molecular self assembly. The enhanced submolecular resolution offered by this technique uniquely reveals key aspects of the competing interactions. Accurate comparison between full-density-based modeled images and experiment allows to identify key structural elements in the assembly in terms of the electron-withdrawing character of the carboxylic groups, interactions of those groups with Cu atoms in the surface, and the valence electron density in the intermolecular region of the hydrogen bonds. This study of trimesic acid assemblies on Cu(111) combining high resolution atomic force microscopy measurements with theory and simulation forges clear connections between fundamental chemical properties of molecules and key features imprinted in force images with submolecular resolution.

Advances in Atomic Force Microscopy: Imaging of Two- and Three-Dimensional Interfacial Water

Interfacial water is closely related to many core scientific and technological issues, covering a broad range of fields, such as material science, geochemistry, electrochemistry and biology. The understanding of the structure and dynamics of interfacial water is the basis of dealing with a series of issues in science and technology. In recent years, atomic force microscopy (AFM) with ultrahigh resolution has become a very powerful option for the understanding of the complex structural and dynamic properties of interfacial water on solid surfaces. In this perspective, we provide an overview of the application of AFM in the study of two dimensional (2D) or three dimensional (3D) interfacial water, and present the prospect and challenges of the AFM-related techniques in experiments and simulations, in order to gain a better understanding of the physicochemical properties of interfacial water.

Manifold dynamic non-covalent interactions for steering molecular assembly and cyclization

Deciphering rich non-covalent interactions that govern many chemical and biological processes is crucial for the design of drugs and controlling molecular assemblies and their chemical transformations. However, real-space characterization of these weak interactions in complex molecular architectures at the single bond level has been a longstanding challenge. Here, we employed bond-resolved scanning probe microscopy combined with an exhaustive structural search algorithm and quantum chemistry calculations to elucidate multiple non-covalent interactions that control the cohesive molecular clustering of well-designed precursor molecules and their chemical reactions. The presence of two flexible bromo-triphenyl moieties in the precursor leads to the assembly of distinct non-planar dimer and trimer clusters by manifold non-covalent interactions, including hydrogen bonding, halogen bonding, C-H⋯π and lone pair⋯π interactions. The dynamic nature of weak interactions allows for transforming dimers into energetically more favourable trimers as molecular density increases. The formation of trimers also facilitates thermally-triggered intermolecular Ullmann coupling reactions, while the disassembly of dimers favours intramolecular cyclization, as evidenced by bond-resolved imaging of metalorganic intermediates and final products. The richness of manifold non-covalent interactions offers unprecedented opportunities for controlling the assembly of complex molecular architectures and steering on-surface synthesis of quantum nanostructures.

The role of convolutional neural networks in scanning probe microscopy: a review

Progress in computing capabilities has enhanced science in many ways. In recent years, various branches of machine learning have been the key facilitators in forging new paths, ranging from categorizing big data to instrumental control, from materials design through image analysis. Deep learning has the ability to identify abstract characteristics embedded within a data set, subsequently using that association to categorize, identify, and isolate subsets of the data. Scanning probe microscopy measures multimodal surface properties, combining morphology with electronic, mechanical, and other characteristics. In this review, we focus on a subset of deep learning algorithms, that is, convolutional neural networks, and how it is transforming the acquisition and analysis of scanning probe data.

Reducing molecular simulation time for AFM images based on super-resolution methods

Atomic force microscopy (AFM) has been an important tool for nanoscale imaging and characterization with atomic and subatomic resolution. Theoretical investigations are getting highly important for the interpretation of AFM images. Researchers have used molecular simulation to examine the AFM imaging mechanism. With a recent flurry of researches applying machine learning to AFM, AFM images obtained from molecular simulation have also been used as training data. However, the simulation is incredibly time consuming. In this paper, we apply super-resolution methods, including compressed sensing and deep learning methods, to reconstruct simulated images and to reduce simulation time. Several molecular simulation energy maps under different conditions are presented to demonstrate the performance of reconstruction algorithms. Through the analysis of reconstructed results, we find that both presented algorithms could complete the reconstruction with good quality and greatly reduce simulation time. Moreover, the super-resolution methods can be used to speed up the generation of training data and vary simulation resolution for AFM machine learning.

Machine Vision Automated Chiral Molecule Detection and Classification in Molecular Imaging

Scanning probe microscopy (SPM) is recognized as an essential characterization tool in a broad range of applications, allowing for real-space atomic imaging of solid surfaces, nanomaterials, and molecular systems. Recently, the imaging of chiral molecular nanostructures via SPM has become a matter of increased scientific and technological interest due to their imminent use as functional platforms in a wide scope of applications, including nonlinear chiroptics, enantioselective catalysis, and enantiospecific sensing. Due to the time-consuming and error-prone image analysis process, a highly efficient analytic framework capable of identifying complex chiral patterns in SPM images is needed. Here, we adopted a state-of-the-art machine vision algorithm to develop a one-image-one-system deep learning framework for the analysis of SPM images. To demonstrate its accuracy and versatility, we employed it to determine the chirality of the molecules comprising two supramolecular self-assemblies with two distinct chiral organization patterns. Our framework accurately detected the position and labeled the chirality of each molecule. This framework underpins the tremendous potential of machine learning algorithms for the automated recognition of complex SPM image patterns in a wide range of research disciplines.

A Deep Learning Approach for Molecular Classification Based on AFM Images

In spite of the unprecedented resolution provided by non-contact atomic force microscopy (AFM) with CO-functionalized and advances in the interpretation of the observed contrast, the unambiguous identification of molecular systems solely based on AFM images, without any prior information, remains an open problem. This work presents a first step towards the automatic classification of AFM experimental images by a deep learning model trained essentially with a theoretically generated dataset. We analyze the limitations of two standard models for pattern recognition when applied to AFM image classification and develop a model with the optimal depth to provide accurate results and to retain the ability to generalize. We show that a variational autoencoder (VAE) provides a very efficient way to incorporate, from very few experimental images, characteristic features into the training set that assure a high accuracy in the classification of both theoretical and experimental images.

Predicting hydration layers on surfaces using deep learning

Characterisation of the nanoscale interface formed between minerals and water is essential to the understanding of natural processes, such as biomineralization, and to develop new technologies where function is dominated by the mineral-water interface. Atomic force microscopy offers the potential to characterize solid-liquid interfaces in high-resolution, with several experimental and theoretical studies offering molecular scale resolution by linking measurements directly to water density on the surface. However, the theoretical techniques used to interpret such results are computationally intensive and development of the approach has been limited by interpretation challenges. In this work, we develop a deep learning architecture to learn the solid-liquid interface of polymorphs of calcium carbonate, allowing for the rapid predictions of density profiles with reasonable accuracy.

Enabling autonomous scanning probe microscopy imaging of single molecules with deep learning

Scanning probe microscopies allow investigating surfaces at the nanoscale, in real space and with unparalleled signal-to-noise ratio. However, these microscopies are not used as much as it would be expected considering their potential. The main limitations preventing a broader use are the need of experienced users, the difficulty in data analysis and the time-consuming nature of experiments that require continuous user supervision. In this work, we addressed the latter and developed an algorithm that controlled the operation of an Atomic Force Microscope (AFM) that, without the need of user intervention, allowed acquiring multiple high-resolution images of different molecules. We used DNA on mica as a model sample to test our control algorithm, which made use of two deep learning techniques that so far have not been used for real time SPM automation. One was an object detector, YOLOv3, which provided the location of molecules in the captured images. The second was a Siamese network that could identify the same molecule in different images. This allowed both performing a series of images on selected molecules while incrementing the resolution, as well as keeping track of molecules already imaged at high resolution, avoiding loops where the same molecule would be imaged an unlimited number of times. Overall, our implementation of deep learning techniques brings SPM a step closer to full autonomous operation.

Automated Searching and Identification of Self-Organized Nanostructures

Currently, researchers spend significant time manually searching through large volumes of data produced during scanning probe imaging to identify specific patterns and motifs formed via self-assembly and self-organization. Here, we use a combination of Monte Carlo simulations, general statistics, and machine learning to automatically distinguish several spatially correlated patterns in a mixed, highly varied data set of real AFM images of self-organized nanoparticles. We do this regardless of feature-scale and without the need for manually labeled training data. Provided that the structures of interest can be simulated, the strategy and protocols we describe can be easily adapted to other self-organized systems and data sets.

Protruding hydrogen atoms as markers for the molecular orientation of a metallocene

A distinct dumbbell shape is observed as the dominant contrast feature in the experimental data when imaging 1,1'-ferrocene dicarboxylic acid (FDCA) molecules on bulk and thin film CaF₂(111) surfaces with non-contact atomic force microscopy (NC-AFM). We use NC-AFM image calculations with the probe particle model to interpret this distinct shape by repulsive interactions between the NC-AFM tip and the top hydrogen atoms of the cyclopentadienyl (Cp) rings. Simulated NC-AFM images show an excellent agreement with experimental constant-height NC-AFM data of FDCA molecules at several tip-sample distances. By measuring this distinct dumbbell shape together with the molecular orientation, a strategy is proposed to determine the conformation of the ferrocene moiety, herein on CaF₂(111) surfaces, by using the protruding hydrogen atoms as markers.

Detecting stable adsorbates of (1S)-camphor on Cu(111) with Bayesian optimization

Identifying the atomic structure of organic-inorganic interfaces is challenging with current research tools. Interpreting the structure of complex molecular adsorbates from microscopy images can be difficult, and using atomistic simulations to find the most stable structures is limited to partial exploration of the potential energy surface due to the high-dimensional phase space. In this study, we present the recently developed Bayesian Optimization Structure Search (BOSS) method as an efficient solution for identifying the structure of non-planar adsorbates. We apply BOSS with density-functional theory simulations to detect the stable adsorbate structures of (1S)-camphor on the Cu(111) surface. We identify the optimal structure among eight unique types of stable adsorbates, in which camphor chemisorbs via oxygen (global minimum) or physisorbs via hydrocarbons to the Cu(111) surface. This study demonstrates that new cross-disciplinary tools, such as BOSS, facilitate the description of complex surface structures and their properties, and ultimately allow us to tune the functionality of advanced materials.