Energy landscapes for machine learning

Feature-aware unsupervised lesion segmentation for brain tumor images using fast data density functional transform

We demonstrate that isomorphically mapping gray-level medical image matrices onto energy spaces underlying the framework of fast data density functional transform (fDDFT) can achieve the unsupervised recognition of lesion morphology. By introducing the architecture of geometric deep learning and metrics of graph neural networks, gridized density functionals of the fDDFT establish an unsupervised feature-aware mechanism with global convolutional kernels to extract the most likely lesion boundaries and produce lesion segmentation. An AutoEncoder-assisted module reduces the computational complexity from [Formula: see text] to [Formula: see text], thus efficiently speeding up global convolutional operations. We validate their performance utilizing various open-access datasets and discuss limitations. The inference time of each object in large three-dimensional datasets is 1.76 s on average. The proposed gridized density functionals have activation capability synergized with gradient ascent operations, hence can be modularized and embedded in pipelines of modern deep neural networks. Algorithms of geometric stability and similarity convergence also raise the accuracy of unsupervised recognition and segmentation of lesion images. Their performance achieves the standard requirement for conventional deep neural networks; the median dice score is higher than 0.75. The experiment shows that the synergy of fDDFT and a naïve neural network improves the training and inference time by 58% and 51%, respectively, and the dice score raises to 0.9415. This advantage facilitates fast computational modeling in interdisciplinary applications and clinical investigation.

Thermodynamics of the Ising Model Encoded in Restricted Boltzmann Machines

The restricted Boltzmann machine (RBM) is a two-layer energy-based model that uses its hidden-visible connections to learn the underlying distribution of visible units, whose interactions are often complicated by high-order correlations. Previous studies on the Ising model of small system sizes have shown that RBMs are able to accurately learn the Boltzmann distribution and reconstruct thermal quantities at temperatures away from the critical point Tc. How the RBM encodes the Boltzmann distribution and captures the phase transition are, however, not well explained. In this work, we perform RBM learning of the 2d and 3d Ising model and carefully examine how the RBM extracts useful probabilistic and physical information from Ising configurations. We find several indicators derived from the weight matrix that could characterize the Ising phase transition. We verify that the hidden encoding of a visible state tends to have an equal number of positive and negative units, whose sequence is randomly assigned during training and can be inferred by analyzing the weight matrix. We also explore the physical meaning of the visible energy and loss function (pseudo-likelihood) of the RBM and show that they could be harnessed to predict the critical point or estimate physical quantities such as entropy.

The Loss Surface of Deep Linear Networks Viewed Through the Algebraic Geometry Lens

By using the viewpoint of modern computational algebraic geometry, we explore properties of the optimization landscapes of deep linear neural network models. After providing clarification on the various definitions of "flat" minima, we show that the geometrically flat minima, which are merely artifacts of residual continuous symmetries of the deep linear networks, can be straightforwardly removed by a generalized L₂-regularization. Then, we establish upper bounds on the number of isolated stationary points of these networks with the help of algebraic geometry. Combining these upper bounds with a method in numerical algebraic geometry, we find all stationary points for modest depth and matrix size. We demonstrate that, in the presence of the non-zero regularization, deep linear networks can indeed possess local minima which are not global minima. Finally, we show that even though the number of stationary points increases as the number of neurons (regularization parameters) increases (decreases), higher index saddles are surprisingly rare.

Systematic Comparison of Genetic Algorithm and Basin Hopping Approaches to the Global Optimization of Si(111) Surface Reconstructions

We present a systematic study of two widely used material structure prediction methods, the Genetic Algorithm and Basin Hopping approaches to global optimization, in a search for the 3 × 3, 5 × 5, and 7 × 7 reconstructions of the Si(111) surface. The Si(111) 7 × 7 reconstruction is the largest and most complex surface reconstruction known, and finding it is a very exacting test for global optimization methods. In this paper, we introduce a modification to previous Genetic Algorithm work on structure search for periodic systems, to allow the efficient search for surface reconstructions, and present a rigorous study of the effect of the different parameters of the algorithm. We also perform a detailed comparison with the recently improved Basin Hopping algorithm using Delocalized Internal Coordinates. Both algorithms succeeded in either resolving the 3 × 3, 5 × 5, and 7 × 7 DAS surface reconstructions or getting “sufficiently close”, i.e., identifying structures that only differ for the positions of a few atoms as well as thermally accessible structures within k_BT/unit area of the global minimum, with T = 300 K. Overall, the Genetic Algorithm is more robust with respect to parameter choice and in success rate, while the Basin Hopping method occasionally exhibits some advantages in speed of convergence. In line with previous studies, the results confirm that robustness, success, and speed of convergence of either approach are strongly influenced by how much the trial moves tend to preserve favorable bonding patterns once these appear.

A Local Optima Network View of Real Function Fitness Landscapes

The local optima network model has proved useful in the past in connection with combinatorial optimization problems. Here we examine its extension to the real continuous function domain. Through a sampling process, the model builds a weighted directed graph which captures the function's minima basin structure and its interconnection and which can be easily manipulated with the help of complex networks metrics. We show that the model provides a complementary view of function spaces that is easier to analyze and visualize, especially at higher dimensions. In particular, we show that function hardness as represented by algorithm performance is strongly related to several graph properties of the corresponding local optima network, opening the way for a classification of problem difficulty according to the corresponding graph structure and with possible extensions in the design of better metaheuristic approaches.

On the capacity and superposition of minima in neural network loss function landscapes

Minima of the loss function landscape (LFL) of a neural network are locally optimal sets of weights that extract and process information from the input data to make outcome predictions. In underparameterised networks, the capacity of the weights may be insufficient to fit all the relevant information. We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input information differently. This effect can be exploited using a meta-network in which the predictive power from multiple minima of the LFL is combined to produce a better classifier. With this approach, we can increase the area under the receiver operating characteristic curve by around 20 % for a complex learning problem. We propose a theoretical basis for combining minima and show how a meta-network can be trained to select the representative that is used for classification of a specific data item. Finally, we present an analysis of symmetry-equivalent solutions to machine learning problems, which provides a systematic means to improve the efficiency of this approach.

Self-assembly of two-dimensional, amorphous materials on a liquid substrate

Recent experimental utilization of liquid substrate in the production of two-dimensional crystals, such as graphene, together with a general interest in amorphous materials, raises the following question: is it beneficial to use a liquid substrate to optimize amorphous material production? Inspired by epitaxial growth, we use a two-dimensional coarse-grained model of interacting particles to show that introducing a motion for the substrate atoms improves the self-assembly process of particles that move on top of the substrate. We find that a specific amount of substrate liquidity (for a given sample temperature) is needed to achieve optimal self-assembly. Our results illustrate the opportunities that the combination of different degrees of freedom provides to the self-assembly processes.

Characterising the area under the curve loss function landscape

One of the most common metrics to evaluate neural network classifiers is the area under the receiver operating characteristic curve (AUC). However, optimisation of the AUC as the loss function during network training is not a standard procedure. Here we compare minimising the cross-entropy (CE) loss and optimising the AUC directly. In particular, we analyse the loss function landscape (LFL) of approximate AUC (appAUC) loss functions to discover the organisation of this solution space. We discuss various surrogates for AUC approximation and show their differences. We find that the characteristics of the appAUC landscape are significantly different from the CE landscape. The approximate AUC loss function improves testing AUC, and the appAUC landscape has substantially more minima, but these minima are less robust, with larger average Hessian eigenvalues. We provide a theoretical foundation to explain these results. To generalise our results, we lastly provide an overview of how the LFL can help to guide loss function analysis and selection.

Machine-Learned Free Energy Surfaces for Capillary Condensation and Evaporation in Mesopores

Using molecular simulations, we study the processes of capillary condensation and capillary evaporation in model mesopores. To determine the phase transition pathway, as well as the corresponding free energy profile, we carry out enhanced sampling molecular simulations using entropy as a reaction coordinate to map the onset of order during the condensation process and of disorder during the evaporation process. The structural analysis shows the role played by intermediate states, characterized by the onset of capillary liquid bridges and bubbles. We also analyze the dependence of the free energy barrier on the pore width. Furthermore, we propose a method to build a machine learning model for the prediction of the free energy surfaces underlying capillary phase transition processes in mesopores.

Combining Machine Learning and Computational Chemistry for Predictive Insights Into Chemical Systems

Machine learning models are poised to make a transformative impact on chemical sciences by dramatically accelerating computational algorithms and amplifying insights available from computational chemistry methods. However, achieving this requires a confluence and coaction of expertise in computer science and physical sciences. This Review is written for new and experienced researchers working at the intersection of both fields. We first provide concise tutorials of computational chemistry and machine learning methods, showing how insights involving both can be achieved. We follow with a critical review of noteworthy applications that demonstrate how computational chemistry and machine learning can be used together to provide insightful (and useful) predictions in molecular and materials modeling, retrosyntheses, catalysis, and drug design.

Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses

Despite the fact that the loss functions of deep neural networks are highly nonconvex, gradient-based optimization algorithms converge to approximately the same performance from many random initial points. One thread of work has focused on explaining this phenomenon by numerically characterizing the local curvature near critical points of the loss function, where the gradients are near zero. Such studies have reported that neural network losses enjoy a no-bad-local-minima property, in disagreement with more recent theoretical results. We report here that the methods used to find these putative critical points suffer from a bad local minima problem of their own: they often converge to or pass through regions where the gradient norm has a stationary point. We call these gradient-flat regions, since they arise when the gradient is approximately in the kernel of the Hessian, such that the loss is locally approximately linear, or flat, in the direction of the gradient. We describe how the presence of these regions necessitates care in both interpreting past results that claimed to find critical points of neural network losses and in designing second-order methods for optimizing neural networks.

Crystal Structure Prediction for Benzene Using Basin-Hopping Global Optimization

Organic molecules can be stable in distinct crystalline forms, known as polymorphs, which have significant consequences for industrial applications. Here, we predict the polymorphs of crystalline benzene computationally for an accurate anisotropic model parametrized to reproduce electronic structure calculations. We adapt the basin-hopping global optimization procedure to the case of crystalline unit cells, simultaneously optimizing the molecular coordinates and unit cell parameters to locate multiple low-energy structures from a variety of crystal space groups. We rapidly locate all the well-established experimental polymorphs of benzene, each of which corresponds to a single local energy minimum of the model. Our results show that basin-hopping can be both an efficient and effective tool for polymorphic crystal structure prediction, requiring no a priori experimental knowledge of cell parameters or symmetry.

Application of Optimization Algorithms in Clusters

The structural characterization of clusters or nanoparticles is essential to rationalize their size and composition-dependent properties. As experiments alone could not provide complete picture of cluster structures, so independent theoretical investigations are needed to find out a detail description of the geometric arrangement and corresponding properties of the clusters. The potential energy surfaces (PES) are explored to find several minima with an ultimate goal of locating the global minima (GM) for the clusters. Optimization algorithms, such as genetic algorithm (GA), basin hopping method and its variants, self-consistent basin-to-deformed-basin mapping, heuristic algorithm combined with the surface and interior operators (HA-SIO), fast annealing evolutionary algorithm (FAEA), random tunneling algorithm (RTA), and dynamic lattice searching (DLS) have been developed to solve the geometrical isomers in pure elemental clusters. Various model or empirical potentials (EPs) as Lennard-Jones (LJ), Born-Mayer, Gupta, Sutton-Chen, and Murrell-Mottram potentials are used to describe the bonding in different type of clusters. Due to existence of a large number of homotops in nanoalloys, genetic algorithm, basin-hopping algorithm, modified adaptive immune optimization algorithm (AIOA), evolutionary algorithm (EA), kick method and Knowledge Led Master Code (KLMC) are also used. In this review the optimization algorithms, computational techniques and accuracy of results obtained by using these mechanisms for different types of clusters will be discussed.

Visualizing free-energy landscapes for four hard disks

We present a simple model system with four hard disks moving in a circular region for which free-energy landscapes can be directly calculated and visualized in two and three dimensions. We construct several energy landscapes for our system, and we explore the strengths and limitations of each in terms of understanding system dynamics, in particular the relationship between state transitions and free-energy barriers. We also demonstrate the importance of distinguishing between system dynamics in real space and those in landscape coordinates, and we show that care must be taken to appropriately combine dynamics with barrier properties to understand the transition rates. This simple model provides an intuitive way to understand free-energy landscapes, and it illustrates the benefits that free-energy landscapes can have over potential energy landscapes.

Mortality prediction model for the triage of COVID-19, pneumonia, and mechanically ventilated ICU patients: A retrospective study

Rationale

Prediction of patients at risk for mortality can help triage patients and assist in resource allocation.

Objectives

Develop and evaluate a machine learning-based algorithm which accurately predicts mortality in COVID-19, pneumonia, and mechanically ventilated patients.

Methods

Retrospective study of 53,001 total ICU patients, including 9166 patients with pneumonia and 25,895 mechanically ventilated patients, performed on the MIMIC dataset. An additional retrospective analysis was performed on a community hospital dataset containing 114 patients positive for SARS-COV-2 by PCR test. The outcome of interest was in-hospital patient mortality.

Results

When trained and tested on the MIMIC dataset, the XGBoost predictor obtained area under the receiver operating characteristic (AUROC) values of 0.82, 0.81, 0.77, and 0.75 for mortality prediction on mechanically ventilated patients at 12-, 24-, 48-, and 72- hour windows, respectively, and AUROCs of 0.87, 0.78, 0.77, and 0.734 for mortality prediction on pneumonia patients at 12-, 24-, 48-, and 72- hour windows, respectively. The predictor outperformed the qSOFA, MEWS and CURB-65 risk scores at all prediction windows. When tested on the community hospital dataset, the predictor obtained AUROCs of 0.91, 0.90, 0.86, and 0.87 for mortality prediction on COVID-19 patients at 12-, 24-, 48-, and 72- hour windows, respectively, outperforming the qSOFA, MEWS and CURB-65 risk scores at all prediction windows.

Conclusions

This machine learning-based algorithm is a useful predictive tool for anticipating patient mortality at clinically useful timepoints, and is capable of accurate mortality prediction for mechanically ventilated patients as well as those diagnosed with pneumonia and COVID-19.

•

Mortality predictions have not previously been evaluated for COVID-19 patients.

•

Machine learning may be a useful predictive tool for anticipating patient mortality.

•

Prediction can be estimated at clinically useful windows up to 72 h in advance.

Archetypal landscapes for deep neural networks

The predictive capabilities of deep neural networks (DNNs) continue to evolve to increasingly impressive levels. However, it is still unclear how training procedures for DNNs succeed in finding parameters that produce good results for such high-dimensional and nonconvex loss functions. In particular, we wish to understand why simple optimization schemes, such as stochastic gradient descent, do not end up trapped in local minima with high loss values that would not yield useful predictions. We explain the optimizability of DNNs by characterizing the local minima and transition states of the loss-function landscape (LFL) along with their connectivity. We show that the LFL of a DNN in the shallow network or data-abundant limit is funneled, and thus easy to optimize. Crucially, in the opposite low-data/deep limit, although the number of minima increases, the landscape is characterized by many minima with similar loss values separated by low barriers. This organization is different from the hierarchical landscapes of structural glass formers and explains why minimization procedures commonly employed by the machine-learning community can navigate the LFL successfully and reach low-lying solutions.

Perspective: new insights from loss function landscapes of neural networks

We investigate the structure of the loss function landscape for neural networks subject to dataset mislabelling, increased training set diversity, and reduced node connectivity, using various techniques developed for energy landscape exploration. The benchmarking models are classification problems for atomic geometry optimisation and hand-written digit prediction. We consider the effect of varying the size of the atomic configuration space used to generate initial geometries and find that the number of stationary points increases rapidly with the size of the training configuration space. We introduce a measure of node locality to limit network connectivity and perturb permutational weight symmetry, and examine how this parameter affects the resulting landscapes. We find that highly-reduced systems have low capacity and exhibit landscapes with very few minima. On the other hand, small amounts of reduced connectivity can enhance network expressibility and can yield more complex landscapes. Investigating the effect of deliberate classification errors in the training data, we find that the variance in testing AUC, computed over a sample of minima, grows significantly with the training error, providing new insight into the role of the variance-bias trade-off when training under noise. Finally, we illustrate how the number of local minima for networks with two and three hidden layers, but a comparable number of variable edge weights, increases significantly with the number of layers, and as the number of training data decreases. This work helps shed further light on neural network loss landscapes and provides guidance for future work on neural network training and optimisation.

Statistical reprogramming of macroscopic self-assembly with dynamic boundaries

Self-assembly is a ubiquitous process that can generate complex and functional structures via local interactions among a large set of simpler components. The ability to program the self-assembly pathway of component sets elucidates fundamental physics and enables alternative competitive fabrication technologies. Reprogrammability offers further opportunities for tuning structural and material properties but requires reversible selection from multistable self-assembling patterns, which remains a challenge. Here, we show statistical reprogramming of two-dimensional (2D), noncompact self-assembled structures by the dynamic confinement of orbitally shaken and magnetically repulsive millimeter-scale particles. Under a constant shaking regime, we control the rate of radius change of an assembly arena via moving hard boundaries and select among a finite set of self-assembled patterns repeatably and reversibly. By temporarily trapping particles in topologically identified stable states, we also demonstrate 2D reprogrammable stiffness and three-dimensional (3D) magnetic clutching of the self-assembled structures. Our reprogrammable system has prospective implications for the design of granular materials in a multitude of physical scales where out-of-equilibrium self-assembly can be realized with different numbers or types of particles. Our dynamic boundary regulation may also enable robust bottom-up control strategies for novel robotic assembly applications by designing more complex spatiotemporal interactions using mobile robots.

Geometry of Energy Landscapes and the Optimizability of Deep Neural Networks

Deep neural networks are workhorse models in machine learning with multiple layers of nonlinear functions composed in series. Their loss function is highly nonconvex, yet empirically even gradient descent minimization is sufficient to arrive at accurate and predictive models. It is hitherto unknown why deep neural networks are easily optimizable. We analyze the energy landscape of a spin glass model of deep neural networks using random matrix theory and algebraic geometry. We analytically show that the multilayered structure holds the key to optimizability: Fixing the number of parameters and increasing network depth, the number of stationary points in the loss function decreases, minima become more clustered in parameter space, and the trade-off between the depth and width of minima becomes less severe. Our analytical results are numerically verified through comparison with neural networks trained on a set of classical benchmark datasets. Our model uncovers generic design principles of machine learning models.

Learning dynamical information from static protein and sequencing data

Many complex processes, from protein folding to neuronal network dynamics, can be described as stochastic exploration of a high-dimensional energy landscape. Although efficient algorithms for cluster detection in high-dimensional spaces have been developed over the last two decades, considerably less is known about the reliable inference of state transition dynamics in such settings. Here we introduce a flexible and robust numerical framework to infer Markovian transition networks directly from time-independent data sampled from stationary equilibrium distributions. We demonstrate the practical potential of the inference scheme by reconstructing the network dynamics for several protein-folding transitions, gene-regulatory network motifs, and HIV evolution pathways. The predicted network topologies and relative transition time scales agree well with direct estimates from time-dependent molecular dynamics data, stochastic simulations, and phylogenetic trees, respectively. Owing to its generic structure, the framework introduced here will be applicable to high-throughput RNA and protein-sequencing datasets, and future cryo-electron microscopy (cryo-EM) data.

From ab initio data to high-dimensional potential energy surfaces: A critical overview and assessment of the development of permutationally invariant polynomial potential energy surfaces for single molecules

The representation of high-dimensional potential energy surfaces by way of the many-body expansion and permutationally invariant polynomials has become a well-established tool for improving the resolution and extending the scope of molecular simulations. The high level of accuracy that can be attained by these potential energy functions (PEFs) is due in large part to their specificity: for each term in the many-body expansion, a species-specific training set must be generated at the desired level of theory and a number of fits attempted in order to obtain a robust and reliable PEF. In this work, we attempt to characterize the numerical aspects of the fitting problem, addressing questions which are of simultaneous practical and fundamental importance. These include concrete illustrations of the nonconvexity of the problem, the ill-conditionedness of the linear system to be solved and possible need for regularization, the sensitivity of the solutions to the characteristics of the training set, and limitations of the approach with respect to accuracy and the types of molecules that can be treated. In addition, we introduce a general approach to the generation of training set configurations based on the familiar harmonic approximation and evaluate the possible benefits to the use of quasirandom sequences for sampling configuration space in this context. Using sulfate as a case study, the findings are largely generalizable and expected to ultimately facilitate the efficient development of PIP-based many-body PEFs for general systems via automation.

The Structure of Adamantane Clusters: Atomistic vs. Coarse-Grained Predictions From Global Optimization

Candidate structures for the global minima of adamantane clusters, (C₁₀H₁₆)_N, are presented. Based on a rigid model for individual molecules with atom-atom pairwise interactions that include Lennard-Jones and Coulomb contributions, low-energy structures were obtained up to N = 42 using the basin-hopping method. The results indicate that adamantane clusters initially grow accordingly with an icosahedral packing scheme, followed above N = 14 by a structural transition toward face-centered cubic structures. The special stabilities obtained at N = 13, 19, and 38 are consistent with these two structural families, and agree with recent mass spectrometry measurements on cationic adamantane clusters. Coarse-graining the intermolecular potential by averaging over all possible orientations only partially confirm the all-atom results, the magic numbers at 13 and 38 being preserved. However, the details near the structural transition are not captured well, because despite their high symmetry the adamantane molecules are still rather anisotropic.

Autonomous Molecular Design: Then and Now

The success of deep machine learning in processing of large amounts of data, for example, in image or voice recognition and generation, raises the possibilities that these tools can also be applied for solving complex problems in materials science. In this forum article, we focus on molecular design that aims to answer the question on how we can predict and synthesize molecules with tailored physical, chemical, or biological properties. A potential answer to this question could be found by using intelligent systems that integrate physical models and computational machine learning techniques with automated synthesis and characterization tools. Such systems learn through every single experiment in an analogy to a human scientific expert. While the general idea of an autonomous system for molecular synthesis and characterization has been around for a while, its implementations for the materials sciences are sparse. Here we provide an overview of the developments in chemistry automation and the applications of machine learning techniques in the chemical and pharmaceutical industries with a focus on the novel capabilities that deep learning brings in.

Jamming transition as a paradigm to understand the loss landscape of deep neural networks

Deep learning has been immensely successful at a variety of tasks, ranging from classification to artificial intelligence. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased, remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in fully connected deep networks a phase transition delimits the over- and underparametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss (the spectrum of the Hessian) are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime. Interestingly, we observe that the ability of fully connected networks to fit random data is independent of their depth, an independence that appears to also hold for real data. We also study a quantity Δ which characterizes how well (Δ<0) or badly (Δ>0) a datum is learned. At the critical point it is power-law distributed on several decades, P_{+}(Δ)∼Δ^{θ} for Δ>0 and P_{-}(Δ)∼(-Δ)^{-γ} for Δ<0, with exponents that depend on the choice of activation function. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.

Exploring Energy Landscapes

Recent advances in the potential energy landscapes approach are highlighted, including both theoretical and computational contributions. Treating the high dimensionality of molecular and condensed matter systems of contemporary interest is important for understanding how emergent properties are encoded in the landscape and for calculating these properties while faithfully representing barriers between different morphologies. The pathways characterized in full dimensionality, which are used to construct kinetic transition networks, may prove useful in guiding such calculations. The energy landscape perspective has also produced new procedures for structure prediction and analysis of thermodynamic properties. Basin-hopping global optimization, with alternative acceptance criteria and generalizations to multiple metric spaces, has been used to treat systems ranging from biomolecules to nanoalloy clusters and condensed matter. This review also illustrates how all this methodology, developed in the context of chemical physics, can be transferred to landscapes defined by cost functions associated with machine learning.

Loss surface of XOR artificial neural networks

Training an artificial neural network involves an optimization process over the landscape defined by the cost (loss) as a function of the network parameters. We explore these landscapes using optimization tools developed for potential energy landscapes in molecular science. The number of local minima and transition states (saddle points of index one), as well as the ratio of transition states to minima, grow rapidly with the number of nodes in the network. There is also a strong dependence on the regularization parameter, with the landscape becoming more convex (fewer minima) as the regularization term increases. We demonstrate that in our formulation, stationary points for networks with N_{h} hidden nodes, including the minimal network required to fit the XOR data, are also stationary points for networks with N_{h}+1 hidden nodes when all the weights involving the additional node are zero. Hence, smaller networks trained on XOR data are embedded in the landscapes of larger networks. Our results clarify certain aspects of the classification and sensitivity (to perturbations in the input data) of minima and saddle points for this system, and may provide insight into dropout and network compression.

A metadynamic approach to understand the recognition mechanism of the histone H3 tail with the ATRXADD domain

The binding affinity between the histone 3 (H3) tail and the ADD domain of ATRX (ATRX_ADD) increases with the subsequent addition of methyl groups on lysine 9 on H3. To improve our understanding of how the difference in methylation state affects binding between H3 and the ATRX_ADD, we adopted a metadynamic approach to explore the recognition mechanism between the two proteins and identify the key intermolecular interactions that mediate this protein-peptide interaction (PPI). The non-methylated H3 peptide is recognized only by the PHD finger of ATRX_ADD while mono-, di-, and trimethylated H3 is recognized by both the PHD and GATA-like zinc finger of the domain. Furthermore, water molecules play an important role in orienting the lysine 9 anchor towards the GATA-like zinc finger, which results in stabilizing the lysine 9 binding pocket on ATRX_ADD. We compared our computational results against experimentally determined NMR and X-ray structures by demonstrating the RMSD, order parameter S² and hydration number of the complex. The metadynamics data provide new insight into roles of water-bridges and the mechanisms through which K9 hydration stabilizes the H3K9me3:ATRX_ADD PPI, providing context for the high affinity demonstrated between this protein and peptide.

Machine Learning of Biomolecular Reaction Coordinates

We present a systematic approach to reduce the dimensionality of a complex molecular system. Starting with a data set of molecular coordinates (obtained from experiment or simulation) and an associated set of metastable conformational states (obtained from clustering the data), a supervised machine learning model is trained to assign unknown molecular structures to the set of metastable states. In this way, the model learns to determine the features of the molecular coordinates that are most important to discriminate the states. Using a new algorithm that exploits this feature importance via an iterative exclusion principle, we identify the essential internal coordinates (such as specific interatomic distances or dihedral angles) of the system, which are shown to represent versatile reaction coordinates that account for the dynamics of the slow degrees of freedom and explain the mechanism of the underlying processes. Moreover, these coordinates give rise to a free energy landscape that may reveal previously hidden intermediate states of the system.

Machine learning landscapes and predictions for patient outcomes

The theory and computational tools developed to interpret and explore energy landscapes in molecular science are applied to the landscapes defined by local minima for neural networks. These machine learning landscapes correspond to fits of training data, where the inputs are vital signs and laboratory measurements for a database of patients, and the objective is to predict a clinical outcome. In this contribution, we test the predictions obtained by fitting to single measurements, and then to combinations of between 2 and 10 different patient medical data items. The effect of including measurements over different time intervals from the 48 h period in question is analysed, and the most recent values are found to be the most important. We also compare results obtained for neural networks as a function of the number of hidden nodes, and for different values of a regularization parameter. The predictions are compared with an alternative convex fitting function, and a strong correlation is observed. The dependence of these results on the patients randomly selected for training and testing decreases systematically with the size of the database available. The machine learning landscapes defined by neural network fits in this investigation have single-funnel character, which probably explains why it is relatively straightforward to obtain the global minimum solution, or a fit that behaves similarly to this optimal parameterization.

Properties of kinetic transition networks for atomic clusters and glassy solids

Small-world and scale-free properties are analysed for kinetic transition networks of clusters and glassy systems.