Optimization on multifractal loss landscapes explains a diverse range of geometrical and dynamical properties of deep learning

Gradient descent and its variants are foundational in solving optimization problems across many disciplines. In deep learning, these optimizers demonstrate a remarkable ability to dynamically navigate complex loss landscapes, ultimately converging to solutions that generalize well. To elucidate the mechanism underlying this ability, we introduce a theoretical framework that models the complexities of loss landscapes as multifractal. Our model unifies and explains a broad range of realistic geometrical signatures of loss landscapes, including clustered degenerate minima, multiscale structure, and rich optimization dynamics in deep neural networks, such as the edge of stability, non-stationary anomalous diffusion, and the extended edge of chaos without requiring fine-tuning parameters. We further develop a fractional diffusion theory to illustrate how these optimization dynamics, coupled with multifractal structure, effectively guide optimizers toward smooth solution spaces housing flatter minima, thus enhancing generalization. Our findings suggest that the complexities of loss landscapes do not hinder optimization; rather, they facilitate the process. This perspective not only has important implications for understanding deep learning but also extends potential applicability to other disciplines where optimization unfolds on complex landscapes.

Molecular Origin of Internal Friction in Intrinsically Disordered Proteins

Protein folding and dynamics are controlled by an interplay of thermal and viscosity effects. The effect of viscous drag through the solvent molecules is described by the classic Kramers theory in the high friction limit, which considers the dampening of the reactant molecules in the solution and quantifies the dependence of the reaction rate on the frictional drag. In addition to the external energy dissipation originating from the surrounding solvent molecules, there is an additional mode of internal energy dissipative force operative within the polypeptide chain reflecting the internal resistance of the chain to its conformational alterations. This dry, solvent-independent intrinsic frictional drag is termed internal friction. In the case of natively folded proteins, the physical origin of internal friction is primarily attributed to the intrachain interactions or other nonnative interactions in their compact states. However, the molecular origin of internal friction in intrinsically disordered proteins (IDPs) remains elusive.In this Account, we address this fundamental issue: what are the principal drivers of viscosity-independent (dry) friction in highly solvated, expanded, conformationally flexible, rapidly fluctuating IDPs that do not possess persistent intrachain interactions? IDPs exhibit diffusive conformational dynamics that is predominantly dominated by the segmental motion of the backbone arising due to the dihedral rotations in the Ramachandran Φ-Ψ space. The physical origin of friction in a complex biopolymeric system such as IDPs can be described by classic polymer models, namely, Rouse/Zimm models with internal friction. These one-dimensional models do not invoke torsional fluctuation components. Kuhn's classic description includes the connection between internal friction and microscopic dihedral hopping. Based on our time-resolved fluorescence anisotropy results, we describe that the sequence-dependent, collective, short-range backbone dihedral rotations govern localized internal friction in an archetypal IDP, namely, α-synuclein. The highly sensitive, residue-specific fluorescence depolarization kinetics offers a potent methodology to characterize and quantify the directional decorrelation engendered due to the short-range dihedral relaxation of the polypeptide backbone in the dihedral space. We utilized this characteristic relaxation time scale as our dynamic readout to quantify the site-specific frictional component. Our linear viscosity-dependent model of torsional relaxation time scale furnished a finite nonzero time constant at the zero solvent viscosity representing the solvent-independent internal friction. These results unveil the effect of the degree of dihedral restraining parameter on the internal friction component by showing that a restrained proline residue imparts higher torsional stiffness in the chain segments and, therefore, exhibits higher internal friction. This Account sheds light on the molecular underpinning of the sequence-specific internal friction in IDPs and will be of interest to unmask the role of internal friction in a diverse range of biomolecular processes involving binding-induced folding, allosteric interaction, protein misfolding and aggregation, and biomolecular condensation via phase separation.

Observing the base-by-base search for native structure along transition paths during the folding of single nucleic acid hairpins

Biomolecular folding involves searching among myriad possibilities for the native conformation, but the elementary steps expected from theory for this search have never been detected directly. We probed the dynamics of folding at high resolution using optical tweezers, measuring individual trajectories as nucleic acid hairpins passed through the high-energy transition states that dominate kinetics and define folding mechanisms. We observed brief but ubiquitous pauses in the transition states, with a dwell time distribution that matched microscopic theories of folding quantitatively. The sequence dependence suggested that pauses were dominated by microbarriers from nonnative conformations during the search by each nucleotide residue for the native base-pairing conformation. Furthermore, the pauses were position dependent, revealing subtle local variations in energy-landscape roughness and allowing the diffusion coefficient describing the microscopic dynamics within the barrier to be found without reconstructing the shape of the energy landscape. These results show how high-resolution measurements can elucidate key microscopic events during folding to test fundamental theories of folding.

Effect of Memory and Inertial Contribution on Transition-Time Distributions: Theory and Simulations

Transition paths refer to the time taken by molecules to cross a barrier separating two molecular conformations. In this work, we study how memory, as well as inertial contribution in the dynamics along a reaction coordinate, can affect the distribution of the transition-path time. We use a simple model of dynamics governed by a generalized Langevin equation with a power-law memory along with the inertial term, which was neglected in previous studies, where memory effects were explored only in the overdamped limit. We derive an approximate expression for the transit-time distribution and discuss our results for the short- and long-time limits and also compare it with known results in the high friction (overdamped) limit as well as in the Markovian limit. We have developed a numerical algorithm to test our theoretical results against extensive numerical simulations.

Conformational Dynamics in Extended RGD-Containing Peptides

RGD is a prolific example of a tripeptide used in biomaterials for cell adhesion, but the potency of free or surface-bound RGD tripeptide is orders-of-magnitude less than the RGD domain within natural proteins. We designed a set of peptides with varying lengths, composed of fragments of fibronectin protein whose central three residues are RGD, in order to vary their conformational behavior without changing the binding site's chemical environment. With these peptides, we measure the conformational dynamics and transient structure of the active site. Our studies reveal how flanking residues affect conformational behavior and integrin binding. We find that disorder of the binding site is important to the potency of RGD peptides and that transient hydrogen bonding near the RGD site affects both the energy landscape roughness of the peptides and peptide binding. This phenomenon is independent of longer-range folding interactions and helps explain why short binding sequences, including RGD itself, do not fully replicate the integrin-targeting properties of extracellular matrix proteins. Our studies reinforce that peptide binding is a holistic event and fragments larger than those directly involved in binding should be considered in the design of peptide epitopes for functional biomaterials.

Non-Markov bond model for dynamic force spectroscopy

Single-molecule force spectroscopy data are conventionally analyzed using a schematic model, wherein a molecular bond is represented as a virtual particle diffusing in a one-dimensional free-energy landscape. However, this simple and efficient approach is unable to account for the "anomalous" bond-breaking kinetics increasingly observed in force spectroscopy experiments and simulations, e.g., in the form of non-exponential distributions of bond lifetimes under constant load. Here, we show that such characteristic traits arise naturally in a rigorous extension of the one-dimensional theory that accounts for the transient dynamics of a generic set of coupled degrees of freedom. These "hidden modes" affect the reaction dynamics in various ways, depending on their relaxation spectrum and the loading protocol, giving rise, in particular, to apparent static and dynamic disorder. In two complementary asymptotic limits, we are able to find exact analytical expressions for pertinent experimental observables, such as the mean rupture force and the rupture-force distribution. Intriguingly, our asymptotic results become unconditionally exact at high loading rates, thus providing us with a microscopically consistent theory of rapid force spectroscopy that avoids the usual Markov assumption.

Quantifying residue-specific conformational dynamics of a highly reactive 29-mer peptide

Understanding structural transitions within macromolecules remains an important challenge in biochemistry, with important implications for drug development and medicine. Insight into molecular behavior often requires residue-specific dynamics measurement at micromolar concentrations. We studied MP01-Gen4, a library peptide selected to rapidly undergo bioconjugation, by using electron paramagnetic resonance (EPR) to measure conformational dynamics. We mapped the dynamics of MP01-Gen4 with residue-specificity and identified the regions involved in a structural transformation related to the conjugation reaction. Upon reaction, the conformational dynamics of residues near the termini slow significantly more than central residues, indicating that the reaction induces a structural transition far from the reaction site. Arrhenius analysis demonstrates a nearly threefold decrease in the activation energy of conformational diffusion upon reaction (8.0 kBT to 3.4 kBT), which occurs across the entire peptide, independently of residue position. This novel approach to EPR spectral analysis provides insight into the positional extent of disorder and the nature of the energy landscape of a highly reactive, intrinsically disordered library peptide before and after conjugation.

Generalized Langevin Equation as a Model for Barrier Crossing Dynamics in Biomolecular Folding

Conformational memory in single-molecule dynamics has attracted recent attention and, in particular, has been invoked as a possible explanation of some of the intriguing properties of transition paths observed in single-molecule force spectroscopy (SMFS) studies. Here we study one candidate for a non-Markovian model that can account for conformational memory, the generalized Langevin equation with a friction force that depends not only on the instantaneous velocity but also on the velocities in the past. The memory in this model is determined by a time-dependent friction memory kernel. We propose a method for extracting this kernel directly from an experimental signal and illustrate its feasibility by applying it to a generalized Rouse model of a SMFS experiment, where the memory kernel is known exactly. Using the same model, we further study how memory affects various statistical properties of transition paths observed in SMFS experiments and evaluate the performance of recent approximate analytical theories of non-Markovian dynamics of barrier crossing. We argue that the same type of analysis can be applied to recent single-molecule observations of transition paths in protein and DNA folding.

Formation of α-helical and β-sheet structures in membrane-bound human IAPP monomer and the resulting membrane deformation

Upon adsorption on membrane, human IAPP monomer takes conformational changes from coils to α-helices and β-sheets. The helices inserted and β on surface cause different types of membrane deformation, implying two distinct aggregation mechanisms.

Dynamics of Disordered Proteins under Confinement: Memory Effects and Internal Friction

Many proteins are disordered under physiological conditions. How efficiently they can search for their cellular targets and how fast they can fold upon target binding is determined by their intrinsic dynamics, which have thus attracted much recent attention. Experiments and molecular simulations show that the inherent reconfiguration timescale for unfolded proteins has a solvent friction component and an internal friction component, and the microscopic origin of the latter, along with its proper mathematical description, has been a topic of considerable debate. Internal friction varies across different proteins of comparable length and increases with decreasing denaturant concentration, showing that it depends on how compact the protein is. Here we report on a systematic atomistic simulation study of how confinement, which induces a more compact unfolded state, affects dynamics and friction in disordered peptides. We find that the average reconfiguration timescales increase exponentially as the peptide's spatial dimensions are reduced; at the same time, confinement broadens the spectrum of relaxation timescales exhibited by the peptide. There are two important implications of this broadening: First, it limits applicability of the common Rouse and Zimm models with internal friction, as those models attempt to capture internal friction effects by introducing a single internal friction timescale. Second, the long-tailed distribution of relaxation times leads to anomalous diffusion effects in the dynamics of intramolecular distances. Analysis and interpretation of experimental signals from various measurements that probe intramolecular protein dynamics (such as single-molecule fluorescence correlation spectroscopy and single-molecule force spectroscopy) rely on the assumption of diffusive dynamics along the distances being probed; hence, our results suggest the need for more general models allowing for anomalous diffusion effects.

Transition path times reveal memory effects and anomalous diffusion in the dynamics of protein folding

Recent single-molecule experiments probed transition paths of biomolecular folding and, in particular, measured the time biomolecules spend while crossing their free energy barriers. A surprising finding from these studies is that the transition barriers crossed by transition paths, as inferred from experimentally observed transition path times, are often lower than the independently determined free energy barriers. Here we explore memory effects leading to anomalous diffusion as a possible origin of this discrepancy. Our analysis of several molecular dynamics trajectories shows that the dynamics of common reaction coordinates used to describe protein folding is subdiffusive, at least at sufficiently short times. We capture this effect using a one-dimensional fractional Brownian motion (FBM) model, in which the system undergoes a subdiffusive process in the presence of a potential of mean force, and show that this model yields much broader distributions of transition path times with stretched exponential long-time tails. Without any adjustable parameters, these distributions agree well with the transition path times computed directly from protein trajectories. We further discuss how the FBM model can be tested experimentally.

Single-Molecule Test for Markovianity of the Dynamics along a Reaction Coordinate

In an effort to answer the much-debated question of whether the time evolution of common experimental observables can be described as one-dimensional diffusion in the potential of mean force, we propose a simple criterion that allows one to test whether the Markov assumption is applicable to a single-molecule trajectory x( t). This test does not involve fitting of the data to any presupposed model and can be applied to experimental data with relatively low temporal resolution.

Equilibrium folding dynamics of meACP in water, heavy water, and low concentration of urea

Many proteins fold in apparent two-state behavior, as partially folded intermediates only transiently accumulate and easily escape detection. Besides a native form and a mainly unfolded form, we captured a partially unfolded form of an acyl carrier protein from Micromonospora echinospora (meACP) in the folding/unfolding equilibrium using chemical exchange saturation transfer NMR experiments. The C-terminal region of the partially unfolded form is mainly folded and the N-terminal is unfolded. Furthermore, to understand how the folding process of meACP is influenced by solvent environments, we compared the folding dynamics of meACP in D₂O, H₂O and low concentration of urea. As the environment becomes more denaturing from D₂O to H₂O and then to urea, the unfolded state becomes increasingly populated, and the folding rate decreases. Adding a small amount of urea, which does not change solvent viscosity, has little effects on the unfolding rates, while changing H₂O to D₂O reduces the unfolding rates possibly due to the increase of solvent viscosity. The quantified solvent effects on the protein folding Gibbs energy and activation energy suggest that the transition state of folding may have a similar structure to the native state of the protein.

Unravelling the complexity of signalling networks in cancer: A review of the increasing role for computational modelling

Cancer induction is a highly complex process involving hundreds of different inducers but whose eventual outcome is the same. Clearly, it is essential to understand how signalling pathways and networks generated by these inducers interact to regulate cell behaviour and create the cancer phenotype. While enormous strides have been made in identifying key networking profiles, the amount of data generated far exceeds our ability to understand how it all "fits together". The number of potential interactions is astronomically large and requires novel approaches and extreme computation methods to dissect them out. However, such methodologies have high intrinsic mathematical and conceptual content which is difficult to follow. This review explains how computation modelling is progressively finding solutions and also revealing unexpected and unpredictable nano-scale molecular behaviours extremely relevant to how signalling and networking are coherently integrated. It is divided into linked sections illustrated by numerous figures from the literature describing different approaches and offering visual portrayals of networking and major conceptual advances in the field. First, the problem of signalling complexity and data collection is illustrated for only a small selection of known oncogenes. Next, new concepts from biophysics, molecular behaviours, kinetics, organisation at the nano level and predictive models are presented. These areas include: visual representations of networking, Energy Landscapes and energy transfer/dissemination (entropy); diffusion, percolation; molecular crowding; protein allostery; quinary structure and fractal distributions; energy management, metabolism and re-examination of the Warburg effect. The importance of unravelling complex network interactions is then illustrated for some widely-used drugs in cancer therapy whose interactions are very extensive. Finally, use of computational modelling to develop micro- and nano- functional models ("bottom-up" research) is highlighted. The review concludes that computational modelling is an essential part of cancer research and is vital to understanding network formation and molecular behaviours that are associated with it. Its role is increasingly essential because it is unravelling the huge complexity of cancer induction otherwise unattainable by any other approach.

Direct measurement of sequence-dependent transition path times and conformational diffusion in DNA duplex formation

The conformational diffusion coefficient, D, sets the timescale for microscopic structural changes during folding transitions in biomolecules like nucleic acids and proteins. D encodes significant information about the folding dynamics such as the roughness of the energy landscape governing the folding and the level of internal friction in the molecule, but it is challenging to measure. The most sensitive measure of D is the time required to cross the energy barrier that dominates folding kinetics, known as the transition path time. To investigate the sequence dependence of D in DNA duplex formation, we measured individual transition paths from equilibrium folding trajectories of single DNA hairpins held under tension in high-resolution optical tweezers. Studying hairpins with the same helix length but with G:C base-pair content varying from 0 to 100%, we determined both the average time to cross the transition paths, τ_tp, and the distribution of individual transit times, P_TP(t). We then estimated D from both τ_tp and P_TP(t) from theories assuming one-dimensional diffusive motion over a harmonic barrier. τ_tp decreased roughly linearly with the G:C content of the hairpin helix, being 50% longer for hairpins with only A:T base pairs than for those with only G:C base pairs. Conversely, D increased linearly with helix G:C content, roughly doubling as the G:C content increased from 0 to 100%. These results reveal that G:C base pairs form faster than A:T base pairs because of faster conformational diffusion, possibly reflecting lower torsional barriers, and demonstrate the power of transition path measurements for elucidating the microscopic determinants of folding.

Dynamics of single-stranded DNA tethered to a solid

Tethering is used to deliver specific biological and industrial functions. For example, single-stranded DNA (ssDNA) is tethered to polymerases and long sequences of double-stranded DNA (dsDNA) during replication, and to solids in DNA microarrays. However, tethering ssDNA to a large object limits not only the available ssDNA conformations, but also the range of time-scales over which the mechanical responses of ssDNA are important. In this work we examine the effect of tethering by measurement of the mechanical response of ssDNA that is tethered at each end to two separate atomic force microscope cantilevers in aqueous solution. Thermal motion of the cantilevers drives the ends of the ssDNA chain at frequencies near 2 kHz. The presence of a tethered molecule makes a large difference to the asymmetric cross-correlation of two cantilevers, which enables resolution of the mechanical properties in our experiments. By analysis of the correlated motion of the cantilevers we extract the friction and stiffness of the ssDNA. We find that the measured friction is much larger than the friction that is usually associated with the unencumbered motion of ssDNA. We also find that the measured relaxation time, ∼30 μs, is much greater than prior measurements of the free-molecule relaxation time. We attribute the difference to the loss of conformational possibilities as a result of constraining the ends of the ssDNA.

Using Kinetic Network Models To Probe Non-Native Salt-Bridge Effects on α-Helix Folding

Salt-bridge interactions play an important role in stabilizing many protein structures, and have been shown to be designable features for protein design. In this work, we study the effects of non-native salt bridges on the folding of a soluble alanine-based peptide (Fs peptide) using extensive all-atom molecular dynamics simulations performed on the Folding@home distributed computing platform. Using Markov State Models, we show how non-native salt-bridges affect the folding kinetics of Fs peptide by perturbing specific conformational states. Furthermore, we present methods for the automatic detection and analysis of such states. These results provide insight into helix folding mechanisms and useful information to guide simulation-based computational protein design.