AdaptiveBandit: A Multi-armed Bandit Framework for Adaptive Sampling in Molecular Simulations

Ensemble Adaptive Sampling Scheme: Identifying an Optimal Sampling Strategy via Policy Ranking

Efficient sampling in biomolecular simulations is critical for accurately capturing the complex dynamic behaviors of biological systems. Adaptive sampling techniques aim to improve efficiency by focusing computational resources on the most relevant regions of the phase space. In this work, we present a framework for identifying the optimal sampling policy through metric-driven ranking. Our approach systematically evaluates the policy ensemble and ranks the policies based on their ability to explore the conformational space effectively. Through a series of biomolecular simulation case studies, we demonstrate that the choice of a different adaptive sampling policy at each round significantly outperforms single policy sampling, leading to faster convergence and improved sampling performance. This approach takes an ensemble of adaptive sampling policies and identifies the optimal policy for the next round based on current data. Beyond presenting this ensemble view of adaptive sampling, we also propose two sampling algorithms that approximate this ranking framework on the fly. The modularity of this framework allows incorporation of any adaptive sampling policy, making it versatile and suitable as a comprehensive adaptive sampling scheme.

Comparative Analysis of Reinforcement Learning Algorithms for Finding Reaction Pathways: Insights from a Large Benchmark Data Set

The identification of kinetically feasible reaction pathways that connect a reactant to its product, including numerous intermediates and transition states, is crucial for predicting chemical reactions and elucidating reaction mechanisms. However, as molecular systems become increasingly complex or larger, the number of local minimum structures and transition states grows, which makes this task challenging, even with advanced computational approaches. We introduced a reinforcement learning algorithm to efficiently identify a kinetically feasible reaction pathway between a given local minimum structure for the reactant and a given one for the product, starting from the reactant. The performance of the algorithm was validated using a benchmark data set of large-scale chemical reaction path networks. Several search policies were proposed, using metrics based on energetic or structural similarity to the product's goal structure, for each local minimum structure candidate found during the search. The performances of baseline greedy, random, and uniform search policies varied substantially depending on the system. In contrast, exploration-exploitation balanced policies such as Thompson sampling, probability of improvement, and expected improvement consistently demonstrated stable and high performance. Furthermore, we characterized the search mechanisms that depend on different policies in detail. This study also addressed potential avenues for further research, such as hierarchical reinforcement learning and multiobjective optimization, which could deepen the problem setting explored in this study.

Adaptive CVgen: Leveraging reinforcement learning for advanced sampling in protein folding and chemical reactions

Enhanced sampling techniques have traditionally encountered two significant challenges: identifying suitable reaction coordinates and addressing the exploration-exploitation dilemma, particularly the difficulty of escaping local energy minima. Here, we introduce Adaptive CVgen, a universal adaptive sampling framework designed to tackle these issues. Our approach utilizes a set of collective variables (CVs) to comprehensively cover the system's potential evolutionary phase space, generating diverse reaction coordinates to address the first challenge. Moreover, we integrate reinforcement learning strategies to dynamically adjust the generated reaction coordinates, thereby effectively balancing the exploration-exploitation dilemma. We apply this framework to sample the conformational space of six proteins transitioning from completely disordered states to folded states, as well as to model the chemical synthesis process of C60, achieving conformations that perfectly match the standard C60 structure. The results demonstrate Adaptive CVgen's effectiveness in exploring new conformations and escaping local minima, achieving both sampling efficiency and exploration accuracy. This framework holds potential for extending to various related challenges, including protein folding dynamics, drug targeting, and complex chemical reactions, thereby opening promising avenues for application in these fields.

Rare-Event Sampling using a Reinforcement Learning-Based Weighted Ensemble Method

Despite the power of path sampling strategies in enabling simulations of rare events, such strategies have not reached their full potential. A common challenge that remains is the identification of a progress coordinate that captures the slow relevant motions of a rare event. Here we have developed a weighted ensemble (WE) path sampling strategy that exploits reinforcement learning to automatically identify an effective progress coordinate among a set of potential coordinates during a simulation. We apply our WE strategy with reinforcement learning to three benchmark systems: (i) an egg carton-shaped toy potential, (ii) an S-shaped toy potential, and (iii) a dimer of the HIV-1 capsid protein (C-terminal domain). To enable rapid testing of the latter system at the atomic level, we employed discrete-state synthetic molecular dynamics trajectories using a generative, fine-grained Markov state model that was based on extensive conventional simulations. Our results demonstrate that using concepts from reinforcement learning with a weighted ensemble of trajectories automatically identifies relevant progress co-ordinates among multiple candidates at a given time during a simulation. Due to the rigorous weighting of trajectories, the simulations maintain rigorous kinetics.

Enhanced Sampling of Biomolecular Slow Conformational Transitions Using Adaptive Sampling and Machine Learning

Biomolecular simulations often suffer from the "time scale problem", hindering the study of rare events occurring over extended time scales. Enhanced sampling techniques aim to alleviate this issue by accelerating conformational transitions, yet they typically necessitate well-defined collective variables (CVs), posing a significant challenge. Machine learning offers promising solutions but typically requires rich training data encompassing the entire free energy surface (FES). In this work, we introduce an automated iterative pipeline designed to mitigate these limitations. Our protocol first utilizes a CV-free count-based adaptive sampling method to generate a data set rich in rare events. From this data set, slow modes are identified using Koopman-reweighted time-lagged independent component analysis (KTICA), which are subsequently leveraged by on-the-fly probability enhanced sampling (OPES) to efficiently explore the FES. The effectiveness of our pipeline is demonstrated and further compared with the common Markov State Model (MSM) approach on two model systems with increasing complexity: alanine dipeptide (Ala2) and deca-alanine (Ala10), underscoring its applicability across diverse biomolecular simulations.

Biomolecular simulations at the exascale: From drug design to organelles and beyond

The rapid advancement in computational power available for research offers to bring not only quantitative improvements, but also qualitative changes in the field of biomolecular simulation. Here, we review the state of biomolecular dynamics simulations at the threshold to exascale resources becoming available. Both developments in parallel and distributed computing will be discussed, providing a perspective on the state of the art of both. A main focus will be on obtaining binding and conformational free energies, with an outlook to macromolecular complexes and (sub)cellular assemblies.

Structural insights into i-motif DNA structures in sequences from the insulin-linked polymorphic region

The insulin-linked polymorphic region is a variable number of tandem repeats region of DNA in the promoter of the insulin gene that regulates transcription of insulin. This region is known to form the alternative DNA structures, i-motifs and G-quadruplexes. Individuals have different sequence variants of tandem repeats and although previous work investigated the effects of some variants on G-quadruplex formation, there is not a clear picture of the relationship between the sequence diversity, the DNA structures formed, and the functional effects on insulin gene expression. Here we show that different sequence variants of the insulin linked polymorphic region form different DNA structures in vitro. Additionally, reporter genes in cellulo indicate that insulin expression may change depending on which DNA structures form. We report the crystal structure and dynamics of an intramolecular i-motif, which reveal sequences within the loop regions forming additional stabilising interactions that are critical to formation of stable i-motif structures. The outcomes of this work reveal the detail in formation of stable i-motif DNA structures, with potential for rational based drug design for compounds to target i-motif DNA.

Multi-armed bandit algorithm for sequential experiments of molecular properties with dynamic feature selection

Sequential optimization is one of the promising approaches in identifying the optimal candidate(s) (molecules, reactants, drugs, etc.) with desired properties (reaction yield, selectivity, efficacy, etc.) from a large set of potential candidates, while minimizing the number of experiments required. However, the high dimensionality of the feature space (e.g., molecular descriptors) makes it often difficult to utilize the relevant features during the process of updating the set of candidates to be examined. In this article, we developed a new sequential optimization algorithm for molecular problems based on reinforcement learning, multi-armed linear bandit framework, and online, dynamic feature selections in which relevant molecular descriptors are updated along with the experiments. We also designed a stopping condition aimed to guarantee the reliability of the chosen candidate from the dataset pool. The developed algorithm was examined by comparing with Bayesian optimization (BO), using two synthetic datasets and two real datasets in which one dataset includes hydration free energy of molecules and another one includes a free energy difference between enantiomer products in chemical reaction. We found that the dynamic feature selection in representing the desired properties along the experiments provides a better performance (e.g., time required to find the best candidate and stop the experiment) as the overall trend and that our multi-armed linear bandit approach with a dynamic feature selection scheme outperforms the standard BO with fixed feature variables. The comparison of our algorithm to BO with dynamic feature selection is also addressed.

Identification of a Cryptic Pocket in Methionine Aminopeptidase-II Using Adaptive Bandit Molecular Dynamics Simulations and Markov State Models

Methionine aminopeptidase-II (MetAP-II) is a metalloprotease, primarily responsible for the cotranslational removal of the N-terminal initiator methionine from the nascent polypeptide chain during protein synthesis. MetAP-II has been implicated in angiogenesis and endothelial cell proliferation and is therefore considered a validated target for cancer therapeutics. However, there is no effective drug available against MetAP-II. In this study, we employ Adaptive Bandit molecular dynamics simulations to investigate the structural dynamics of the apo and ligand-bound MetAP-II. Our results focus on the dynamic behavior of the disordered loop that is not resolved in most of the crystal structures. Further analysis of the conformational flexibility of the disordered loop reveals a hidden cryptic pocket that is predicted to be potentially druggable. The network analysis indicates that the disordered loop region has a direct signaling route to the active site. These findings highlight a new way to target MetAP-II by designing inhibitors for the allosteric site within this disordered loop region.

Deciphering the Coevolutionary Dynamics of L2 β-Lactamases via Deep Learning

L2 β-lactamases, serine-based class A β-lactamases expressed by Stenotrophomonas maltophilia, play a pivotal role in antimicrobial resistance (AMR). However, limited studies have been conducted on these important enzymes. To understand the coevolutionary dynamics of L2 β-lactamase, innovative computational methodologies, including adaptive sampling molecular dynamics simulations, and deep learning methods (convolutional variational autoencoders and BindSiteS-CNN) explored conformational changes and correlations within the L2 β-lactamase family together with other representative class A enzymes including SME-1 and KPC-2. This work also investigated the potential role of hydrophobic nodes and binding site residues in facilitating the functional mechanisms. The convergence of analytical approaches utilized in this effort yielded comprehensive insights into the dynamic behavior of the β-lactamases, specifically from an evolutionary standpoint. In addition, this analysis presents a promising approach for understanding how the class A β-lactamases evolve in response to environmental pressure and establishes a theoretical foundation for forthcoming endeavors in drug development aimed at combating AMR.

Adaptive Sampling Methods for Molecular Dynamics in the Era of Machine Learning

Molecular dynamics (MD) simulations are fundamental computational tools for the study of proteins and their free energy landscapes. However, sampling protein conformational changes through MD simulations is challenging due to the relatively long time scales of these processes. Many enhanced sampling approaches have emerged to tackle this problem, including biased sampling and path-sampling methods. In this Perspective, we focus on adaptive sampling algorithms. These techniques differ from other approaches because the thermodynamic ensemble is preserved and the sampling is enhanced solely by restarting MD trajectories at particularly chosen seeds rather than introducing biasing forces. We begin our treatment with an overview of theoretically transparent methods, where we discuss principles and guidelines for adaptive sampling. Then, we present a brief summary of select methods that have been applied to realistic systems in the past. Finally, we discuss recent advances in adaptive sampling methodology powered by deep learning techniques, as well as their shortcomings.

Machine learning coarse-grained potentials of protein thermodynamics

A generalized understanding of protein dynamics is an unsolved scientific problem, the solution of which is critical to the interpretation of the structure-function relationships that govern essential biological processes. Here, we approach this problem by constructing coarse-grained molecular potentials based on artificial neural networks and grounded in statistical mechanics. For training, we build a unique dataset of unbiased all-atom molecular dynamics simulations of approximately 9 ms for twelve different proteins with multiple secondary structure arrangements. The coarse-grained models are capable of accelerating the dynamics by more than three orders of magnitude while preserving the thermodynamics of the systems. Coarse-grained simulations identify relevant structural states in the ensemble with comparable energetics to the all-atom systems. Furthermore, we show that a single coarse-grained potential can integrate all twelve proteins and can capture experimental structural features of mutated proteins. These results indicate that machine learning coarse-grained potentials could provide a feasible approach to simulate and understand protein dynamics.

Binding-and-Folding Recognition of an Intrinsically Disordered Protein Using Online Learning Molecular Dynamics

Intrinsically disordered proteins participate in many biological processes by folding upon binding to other proteins. However, coupled folding and binding processes are not well understood from an atomistic point of view. One of the main questions is whether folding occurs prior to or after binding. Here we use a novel, unbiased, high-throughput adaptive sampling approach to reconstruct the binding and folding between the disordered transactivation domain of c-Myb and the KIX domain of the CREB-binding protein. The reconstructed long-term dynamical process highlights the binding of a short stretch of amino acids on c-Myb as a folded α-helix. Leucine residues, especially Leu298-Leu302, establish initial native contacts that prime the binding and folding of the rest of the peptide, with a mixture of conformational selection on the N-terminal region with an induced fit of the C-terminal.

Thirty years of molecular dynamics simulations on posttranslational modifications of proteins

Posttranslational modifications (PTMs) are an integral component to how cells respond to perturbation. While experimental advances have enabled improved PTM identification capabilities, the same throughput for characterizing how structural changes caused by PTMs equate to altered physiological function has not been maintained. In this Perspective, we cover the history of computational modeling and molecular dynamics simulations which have characterized the structural implications of PTMs. We distinguish results from different molecular dynamics studies based upon the timescales simulated and analysis approaches used for PTM characterization. Lastly, we offer insights into how opportunities for modern research efforts on in silico PTM characterization may proceed given current state-of-the-art computing capabilities and methodological advancements.

Multiagent Reinforcement Learning-Based Adaptive Sampling for Conformational Dynamics of Proteins

Machine learning is increasingly applied to improve the efficiency and accuracy of molecular dynamics (MD) simulations. Although the growth of distributed computer clusters has allowed researchers to obtain higher amounts of data, unbiased MD simulations have difficulty sampling rare states, even under massively parallel adaptive sampling schemes. To address this issue, several algorithms inspired by reinforcement learning (RL) have arisen to promote exploration of the slow collective variables (CVs) of complex systems. Nonetheless, most of these algorithms are not well-suited to leverage the information gained by simultaneously sampling a system from different initial states (e.g., a protein in different conformations associated with distinct functional states). To fill this gap, we propose two algorithms inspired by multiagent RL that extend the functionality of closely related techniques (REAP and TSLC) to situations where the sampling can be accelerated by learning from different regions of the energy landscape through coordinated agents. Essentially, the algorithms work by remembering which agent discovered each conformation and sharing this information with others at the action-space discretization step. A stakes function is introduced to modulate how different agents sense rewards from discovered states of the system. The consequences are three-fold: (i) agents learn to prioritize CVs using only relevant data, (ii) redundant exploration is reduced, and (iii) agents that obtain higher stakes are assigned more actions. We compare our algorithm with other adaptive sampling techniques (least counts, REAP, TSLC, and AdaptiveBandit) to show and rationalize the gain in performance.

Additive eigenvectors as optimal reaction coordinates, conditioned trajectories, and time-reversible description of stochastic processes

A fundamental way to analyze complex multidimensional stochastic dynamics is to describe it as diffusion on a free energy landscape—free energy as a function of reaction coordinates (RCs). For such a description to be quantitatively accurate, the RC should be chosen in an optimal way. The committor function is a primary example of an optimal RC for the description of equilibrium reaction dynamics between two states. Here, additive eigenvectors (addevs) are considered as optimal RCs to address the limitations of the committor. An addev master equation for a Markov chain is derived. A stationary solution of the equation describes a sub-ensemble of trajectories conditioned on having the same optimal RC for the forward and time-reversed dynamics in the sub-ensemble. A collection of such sub-ensembles of trajectories, called stochastic eigenmodes, can be used to describe/approximate the stochastic dynamics. A non-stationary solution describes the evolution of the probability distribution. However, in contrast to the standard master equation, it provides a time-reversible description of stochastic dynamics. It can be integrated forward and backward in time. The developed framework is illustrated on two model systems—unidirectional random walk and diffusion.

Integration of machine learning with computational structural biology of plants

Computational structural biology of proteins has developed rapidly in recent decades with the development of new computational tools and the advancement of computing hardware. However, while these techniques have widely been used to make advancements in human medicine, these methods have seen less utilization in the plant sciences. In the last several years, machine learning methods have gained popularity in computational structural biology. These methods have enabled the development of new tools which are able to address the major challenges that have hampered the wide adoption of the computational structural biology of plants. This perspective examines the remaining challenges in computational structural biology and how the development of machine learning techniques enables more in-depth computational structural biology of plants.

Adsorption Sites on Pd Nanoparticles Unraveled by Machine-Learning Potential with Adaptive Sampling

Catalytic properties of noble-metal nanoparticles (NPs) are largely determined by their surface morphology. The latter is probed by surface-sensitive spectroscopic techniques in different spectra regions. A fast and precise computational approach enabling the prediction of surface-adsorbate interaction would help the reliable description and interpretation of experimental data. In this work, we applied Machine Learning (ML) algorithms for the task of adsorption-energy approximation for CO on Pd nanoclusters. Due to a high dependency of binding energy from the nature of the adsorbing site and its local coordination, we tested several structural descriptors for the ML algorithm, including mean Pd-C distances, coordination numbers (CN) and generalized coordination numbers (GCN), radial distribution functions (RDF), and angular distribution functions (ADF). To avoid overtraining and to probe the most relevant positions above the metal surface, we utilized the adaptive sampling methodology for guiding the ab initio Density Functional Theory (DFT) calculations. The support vector machines (SVM) and Extra Trees algorithms provided the best approximation quality and mean absolute error in energy prediction up to 0.12 eV. Based on the developed potential, we constructed an energy-surface 3D map for the whole Pd₅₅ nanocluster and extended it to new geometries, Pd₇₉, and Pd₈₅, not implemented in the training sample. The methodology can be easily extended to adsorption energies onto mono- and bimetallic NPs at an affordable computational cost and accuracy.

Nonparametric Analysis of Nonequilibrium Simulations

We extend the nonparametric framework of reaction coordinate optimization to nonequilibrium ensembles of (short) trajectories. For example, we show how, starting from such an ensemble, one can obtain an equilibrium free-energy profile along the committor, which can be used to determine important properties of the dynamics exactly. A new adaptive sampling approach, the transition-state ensemble enrichment, is suggested, which samples the configuration space by "growing" committor segments toward each other starting from the boundary states. This framework is suggested as a general tool, alternative to the Markov state models, for a rigorous and accurate analysis of simulations of large biomolecular systems, as it has the following attractive properties. It is immune to the curse of dimensionality, does not require system-specific information, can approximate arbitrary reaction coordinates with high accuracy, and has sensitive and rigorous criteria to test optimality and convergence. The approaches are illustrated on a 50-dimensional model system and a realistic protein folding trajectory.

AI-driven multiscale simulations illuminate mechanisms of SARS-CoV-2 spike dynamics

We develop a generalizable AI-driven workflow that leverages heterogeneous HPC resources to explore the time-dependent dynamics of molecular systems. We use this workflow to investigate the mechanisms of infectivity of the SARS-CoV-2 spike protein, the main viral infection machinery. Our workflow enables more efficient investigation of spike dynamics in a variety of complex environments, including within a complete SARS-CoV-2 viral envelope simulation, which contains 305 million atoms and shows strong scaling on ORNL Summit using NAMD. We present several novel scientific discoveries, including the elucidation of the spike's full glycan shield, the role of spike glycans in modulating the infectivity of the virus, and the characterization of the flexible interactions between the spike and the human ACE2 receptor. We also demonstrate how AI can accelerate conformational sampling across different systems and pave the way for the future application of such methods to additional studies in SARS-CoV-2 and other molecular systems.

Artificial intelligence techniques for integrative structural biology of intrinsically disordered proteins

We outline recent developments in artificial intelligence (AI) and machine learning (ML) techniques for integrative structural biology of intrinsically disordered proteins (IDP) ensembles. IDPs challenge the traditional protein structure-function paradigm by adapting their conformations in response to specific binding partners leading them to mediate diverse, and often complex cellular functions such as biological signaling, self-organization and compartmentalization. Obtaining mechanistic insights into their function can therefore be challenging for traditional structural determination techniques. Often, scientists have to rely on piecemeal evidence drawn from diverse experimental techniques to characterize their functional mechanisms. Multiscale simulations can help bridge critical knowledge gaps about IDP structure-function relationships-however, these techniques also face challenges in resolving emergent phenomena within IDP conformational ensembles. We posit that scalable statistical inference techniques can effectively integrate information gleaned from multiple experimental techniques as well as from simulations, thus providing access to atomistic details of these emergent phenomena.

AI-Driven Multiscale Simulations Illuminate Mechanisms of SARS-CoV-2 Spike Dynamics

We develop a generalizable AI-driven workflow that leverages heterogeneous HPC resources to explore the time-dependent dynamics of molecular systems. We use this workflow to investigate the mechanisms of infectivity of the SARS-CoV-2 spike protein, the main viral infection machinery. Our workflow enables more efficient investigation of spike dynamics in a variety of complex environments, including within a complete SARS-CoV-2 viral envelope simulation, which contains 305 million atoms and shows strong scaling on ORNL Summit using NAMD. We present several novel scientific discoveries, including the elucidation of the spike's full glycan shield, the role of spike glycans in modulating the infectivity of the virus, and the characterization of the flexible interactions between the spike and the human ACE2 receptor. We also demonstrate how AI can accelerate conformational sampling across different systems and pave the way for the future application of such methods to additional studies in SARS-CoV-2 and other molecular systems.