Emergent behaviour and neural dynamics in artificial agents tracking odour plumes

Sensing flow gradients is necessary for learning autonomous underwater navigation

Aquatic animals are much better at underwater navigation than robotic vehicles. Robots face major challenges in deep water because of their limited access to global positioning signals and flow maps. These limitations, and the changing nature of water currents, support the use of reinforcement learning approaches, where the navigator learns through trial-and-error interactions with the flow environment. But is it feasible to learn underwater navigation in the agent's Umwelt, without any land references? Here, we tasked an artificial swimmer with learning to reach a specific destination in unsteady flows by relying solely on egocentric observations, collected through on-board flow sensors in the agent's body frame, with no reference to a geocentric inertial frame. We found that while sensing local flow velocities is sufficient for geocentric navigation, successful egocentric navigation requires additional information of local flow gradients. Importantly, egocentric navigation strategies obey rotational symmetry and are more robust in unfamiliar conditions and flows not experienced during training. Our work expands underwater robot-centric learning, helps explain why aquatic organisms have arrays of flow sensors that detect gradients, and provides physics-based guidelines for transfer learning of learned policies to unfamiliar and diverse flow environments.

Design Principles From Natural Olfaction for Electronic Noses

Natural olfactory systems possess remarkable sensitivity and precision beyond what is currently achievable by engineered gas sensors. Unlike their artificial counterparts, noses are capable of distinguishing scents associated with mixtures of volatile molecules in complex, typically fluctuating environments and can adapt to changes. This perspective examines the multifaceted biological principles that provide olfactory systems their discriminatory prowess, and how these ideas can be ported to the design of electronic noses for substantial improvements in performance across metrics such as sensitivity and ability to speciate chemical mixtures. The topics examined herein include the fluid dynamics of odorants in natural channels; specificity and kinetics of odorant interactions with olfactory receptors and mucus linings; complex signal processing that spatiotemporally encodes physicochemical properties of odorants; active sampling techniques, like biological sniffing and nose repositioning; biological priming; and molecular chaperoning. Each of these components of natural olfactory systems are systmatically investigated, as to how they have been or can be applied to electronic noses. While not all artificial sensors can employ these strategies simultaneously, integrating a subset of bioinspired principles can address issues like sensitivity, drift, and poor selectivity, offering advancements in many sectors such as environmental monitoring, industrial safety, and disease diagnostics.

A vector-based strategy for olfactory navigation in Drosophila

Odors serve as essential cues for navigation. Although tracking an odor plume has been modeled as a reflexive process, it remains unclear whether animals can use memories of their past odor encounters to infer the spatial structure of their chemical environment or their location within it. Here we developed a virtual-reality olfactory paradigm that allows head-fixed Drosophila to navigate structured chemical landscapes, offering insight into how memory mechanisms shape their navigational strategies. We found that flies track an appetitive odor corridor by following its boundary, alternating between rapid counterturns to exit the plume and directed returns to its edge. Using a combination of behavioral modeling, functional calcium imaging, and neural perturbations, we demonstrate that this 'edge-tracking' strategy relies on vector-based computations within the Drosophila central complex in which flies store and dynamically update memories of the direction to return them to the plume's boundary. Consistent with this, we find that FC2 neurons within the fan-shaped body, which encode a fly's navigational goal, signal the direction back to the odor boundary when flies are outside the plume. Together, our studies suggest that flies leverage the plume's boundary as a dynamic landmark to guide their navigation, analogous to the memory-based strategies other insects use for long-distance migration or homing to their nests. Plume tracking thus uses components of a conserved navigational toolkit, enabling flies to use memory mechanisms to navigate through a complex shifting chemical landscape.

TiDHy: Timescale Demixing via Hypernetworks to learn simultaneous dynamics from mixed observations

Neural activity and behavior arise from multiple concurrent time-varying systems, including neuromodulation, neural state, and history; however, most current approaches model these data as one set of dynamics with a single timescale. Here we develop Timescale Demixing via Hypernetworks (TiDHy) as a new computational method to model spatiotemporal data, decomposing them into multiple simultaneous latent dynamical systems that may span orders-of-magnitude different timescales. Specifically, we train a hypernetwork to dynamically reweigh linear combinations of latent dynamics. This approach enables accurate data reconstruction, converges to true latent dynamics, and captures multiple timescales of variation. We first demonstrate that TiDHy can demix dynamics and timescales from synthetic data comprising multiple independent switching linear dynamical systems, even when the observations are mixed. Next, with a simulated locomotion behavior dataset, we show that TiDHy accurately captures both the fast dynamics of movement kinematics and the slow dynamics of changing terrains. Finally, in an open-source multi-animal social behavior dataset, we show that the keypoint trajectory dynamics extracted with TiDHy can be used to accurately identify social behaviors of multiple mice. Taken together, TiDHy is a powerful new algorithm for demixing simultaneous latent dynamical systems with applications to diverse computational domains.

Q-learning with temporal memory to navigate turbulence

We consider the problem of olfactory searches in a turbulent environment. We focus on agents that respond solely to odor stimuli, with no access to spatial perception nor prior information about the odor. We ask whether navigation to a target can be learned robustly within a sequential decision making framework. We develop a reinforcement learning algorithm using a small set of interpretable olfactory states and train it with realistic turbulent odor cues. By introducing a temporal memory, we demonstrate that two salient features of odor traces, discretized in few olfactory states, are sufficient to learn navigation in a realistic odor plume. Performance is dictated by the sparse nature of turbulent odors. An optimal memory exists which ignores blanks within the plume and activates a recovery strategy outside the plume. We obtain the best performance by letting agents learn their recovery strategy and show that it is mostly casting cross wind, similar to behavior observed in flying insects. The optimal strategy is robust to substantial changes in the odor plumes, suggesting minor parameter tuning may be sufficient to adapt to different environments.

I2Bot: an open-source tool for multi-modal and embodied simulation of insect navigation

Achieving a comprehensive understanding of animal intelligence demands an integrative approach that acknowledges the interplay between an organism's brain, body and environment. Insects, despite their limited computational resources, demonstrate remarkable abilities in navigation. Existing computational models often fall short in faithfully replicating the morphology of real insects and their interactions with the environment, hindering validation and practical application in robotics. To address these gaps, we present I2Bot, a novel simulation tool based on the morphological characteristics of real insects. This tool empowers robotic models with dynamic sensory capabilities, realistic modelling of insect morphology, physical dynamics and sensory capacity. By integrating gait controllers and computational models into I2Bot, we have implemented classical embodied navigation behaviours and revealed some fundamental navigation principles. By open-sourcing I2Bot, we aim to accelerate the understanding of insect intelligence and foster advances in the development of autonomous robotic systems.

Neural circuits for goal-directed navigation across species

Across species, navigation is crucial for finding both resources and shelter. In vertebrates, the hippocampus supports memory-guided goal-directed navigation, whereas in arthropods the central complex supports similar functions. A growing literature is revealing similarities and differences in the organization and function of these brain regions. We review current knowledge about how each structure supports goal-directed navigation by building internal representations of the position or orientation of an animal in space, and of the location or direction of potential goals. We describe input pathways to each structure - medial and lateral entorhinal cortex in vertebrates, and columnar and tangential neurons in insects - that primarily encode spatial and non-spatial information, respectively. Finally, we highlight similarities and differences in spatial encoding across clades and suggest experimental approaches to compare coding principles and behavioral capabilities across species. Such a comparative approach can provide new insights into the neural basis of spatial navigation and neural computation.

How do mammals convert dynamic odor information into neural maps for landscape navigation?

Odors are transported by seemingly chaotic plumes, whose spatiotemporal structure contains rich information about space, with olfaction serving as a gateway for obtaining and processing this spatial information. Beyond tracking odors, olfaction provides localization and chemical communication cues for detecting conspecifics and predators, and linking external environments to internal cognitive maps. In this Essay, we discuss recent physiological, behavioral, and methodological advancements in mammalian olfactory research to present our current understanding of how olfaction can be used to navigate the environment. We also examine potential neural mechanisms that might convert dynamic olfactory inputs into environmental maps along this axis. Finally, we consider technological applications of odor dynamics for developing bio-inspired sensor technologies, robotics, and computational models. By shedding light on the principles underlying the processing of odor dynamics, olfactory research will pave the way for innovative solutions that bridge the gap between biology and technology, enriching our understanding of the natural world.

Wind gates olfaction-driven search states in free flight

For organisms tracking a chemical cue to its source, the motion of their surrounding fluid provides crucial information for success. Swimming and flying animals engaged in olfaction-driven search often start by turning into the direction of an oncoming wind or water current. However, it is unclear how organisms adjust their strategies when directional cues are absent or unreliable, as is often the case in nature. Here, we use the genetic toolkit of Drosophila melanogaster to develop an optogenetic paradigm to deliver temporally precise "virtual" olfactory experiences for free-flying animals in either laminar wind or still air. We first confirm that in laminar wind flies turn upwind. Furthermore, we show that they achieve this using a rapid (∼100 ms) turn, implying that flies estimate the ambient wind direction prior to "surging" upwind. In still air, flies adopt a remarkably stereotyped "sink and circle" search state characterized by ∼60° turns at 3-4 Hz, biased in a consistent direction. Together, our results show that Drosophila melanogaster assesses the presence and direction of ambient wind prior to deploying a distinct search strategy. In both laminar wind and still air, immediately after odor onset, flies decelerate and often perform a rapid turn. Both maneuvers are consistent with predictions from recent control theoretic analyses for how insects may estimate properties of wind while in flight. We suggest that flies may use their deceleration and "anemometric" turn as active sensing maneuvers to rapidly gauge properties of their wind environment before initiating a proximal or upwind search routine.

Fish-inspired tracking of underwater turbulent plumes

Autonomous ocean-exploring vehicles have begun to take advantage of onboard sensor measurements of water properties such as salinity and temperature to locate oceanic features in real time. Such targeted sampling strategies enable more rapid study of ocean environments by actively steering towards areas of high scientific value. Inspired by the ability of aquatic animals to navigate via flow sensing, this work investigates hydrodynamic cues for accomplishing targeted sampling using a palm-sized robotic swimmer. As proof-of-concept analogy for tracking hydrothermal vent plumes in the ocean, the robot is tasked with locating the center of turbulent jet flows in a 13,000-liter water tank using data from onboard pressure sensors. To learn a navigation strategy, we first implemented RL on a simulated version of the robot navigating in proximity to turbulent jets. After training, the RL algorithm discovered an effective strategy for locating the jets by following transverse velocity gradients sensed by pressure sensors located on opposite sides of the robot. When implemented on the physical robot, this gradient following strategy enabled the robot to successfully locate the turbulent plumes at more than twice the rate of random searching. Additionally, we found that navigation performance improved as the distance between the pressure sensors increased, which can inform the design of distributed flow sensors in ocean robots. Our results demonstrate the effectiveness and limits of flow-based navigation for autonomously locating hydrodynamic features of interest.

Reinforcement learning with thermal fluctuations at the nanoscale

Reinforcement Learning offers a framework to learn to choose actions in order to control a system. However, at small scales Brownian fluctuations limit the control of nanomachine actuation or nanonavigation and of the molecular machinery of life. We analyze this regime using the general framework of Markov decision processes. We show that at the nanoscale, while optimal control actions should bring an improvement proportional to the small ratio of the applied force times a length scale over the temperature, the learned improvement is smaller and proportional to the square of this small ratio. Consequently, the efficiency of learning, which compares the learning improvement to the theoretical optimal improvement, drops to zero. Nevertheless, these limitations can be circumvented by using actions learned at a lower temperature. These results are illustrated with simulations of the control of the shape of small particle clusters.

A virtual rodent predicts the structure of neural activity across behaviours

Animals have exquisite control of their bodies, allowing them to perform a diverse range of behaviours. How such control is implemented by the brain, however, remains unclear. Advancing our understanding requires models that can relate principles of control to the structure of neural activity in behaving animals. Here, to facilitate this, we built a 'virtual rodent', in which an artificial neural network actuates a biomechanically realistic model of the rat1 in a physics simulator2. We used deep reinforcement learning3-5 to train the virtual agent to imitate the behaviour of freely moving rats, thus allowing us to compare neural activity recorded in real rats to the network activity of a virtual rodent mimicking their behaviour. We found that neural activity in the sensorimotor striatum and motor cortex was better predicted by the virtual rodent's network activity than by any features of the real rat's movements, consistent with both regions implementing inverse dynamics6. Furthermore, the network's latent variability predicted the structure of neural variability across behaviours and afforded robustness in a way consistent with the minimal intervention principle of optimal feedback control7. These results demonstrate how physical simulation of biomechanically realistic virtual animals can help interpret the structure of neural activity across behaviour and relate it to theoretical principles of motor control.

Inductive biases of neural network modularity in spatial navigation

The brain may have evolved a modular architecture for daily tasks, with circuits featuring functionally specialized modules that match the task structure. We hypothesize that this architecture enables better learning and generalization than architectures with less specialized modules. To test this, we trained reinforcement learning agents with various neural architectures on a naturalistic navigation task. We found that the modular agent, with an architecture that segregates computations of state representation, value, and action into specialized modules, achieved better learning and generalization. Its learned state representation combines prediction and observation, weighted by their relative uncertainty, akin to recursive Bayesian estimation. This agent's behavior also resembles macaques' behavior more closely. Our results shed light on the possible rationale for the brain's modularity and suggest that artificial systems can use this insight from neuroscience to improve learning and generalization in natural tasks.

Wind Gates Olfaction Driven Search States in Free Flight

For organisms tracking a chemical cue to its source, the motion of their surrounding fluid provides crucial information for success. Swimming and flying animals engaged in olfaction driven search often start by turning into the direction of an oncoming wind or water current. However, it is unclear how organisms adjust their strategies when directional cues are absent or unreliable, as is often the case in nature. Here, we use the genetic toolkit of Drosophila melanogaster to develop an optogenetic paradigm to deliver temporally precise "virtual" olfactory experiences for free-flying animals in either laminar wind or still air. We first confirm that in laminar wind flies turn upwind. Furthermore, we show that they achieve this using a rapid (∼100 ms) turn, implying that flies estimate the ambient wind direction prior to "surging" upwind. In still air, flies adopt remarkably stereotyped "sink and circle" search state characterized by ∼60°turns at 3-4 Hz, biased in a consistent direction. Together, our results show that Drosophila melanogaster assess the presence and direction of ambient wind prior to deploying a distinct search strategy. In both laminar wind and still air, immediately after odor onset, flies decelerate and often perform a rapid turn. Both maneuvers are consistent with predictions from recent control theoretic analyses for how insects may estimate properties of wind while in flight. We suggest that flies may use their deceleration and "anemometric" turn as active sensing maneuvers to rapidly gauge properties of their wind environment before initiating a proximal or upwind search routine.

Learning optimal integration of spatial and temporal information in noisy chemotaxis

We investigate the boundary between chemotaxis driven by spatial estimation of gradients and chemotaxis driven by temporal estimation. While it is well known that spatial chemotaxis becomes disadvantageous for small organisms at high noise levels, it is unclear whether there is a discontinuous switch of optimal strategies or a continuous transition exists. Here, we employ deep reinforcement learning to study the possible integration of spatial and temporal information in an a priori unconstrained manner. We parameterize such a combined chemotactic policy by a recurrent neural network and evaluate it using a minimal theoretical model of a chemotactic cell. By comparing with constrained variants of the policy, we show that it converges to purely temporal and spatial strategies at small and large cell sizes, respectively. We find that the transition between the regimes is continuous, with the combined strategy outperforming in the transition region both the constrained variants as well as models that explicitly integrate spatial and temporal information. Finally, by utilizing the attribution method of integrated gradients, we show that the policy relies on a nontrivial combination of spatially and temporally derived gradient information in a ratio that varies dynamically during the chemotactic trajectories.

Emergence of belief-like representations through reinforcement learning

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming "beliefs"-optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN's learned representation encodes belief information, but only when the RNN's capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark

The olfactory search POMDP (partially observable Markov decision process) is a sequential decision-making problem designed to mimic the task faced by insects searching for a source of odor in turbulence, and its solutions have applications to sniffer robots. As exact solutions are out of reach, the challenge consists in finding the best possible approximate solutions while keeping the computational cost reasonable. We provide a quantitative benchmarking of a solver based on deep reinforcement learning against traditional POMDP approximate solvers. We show that deep reinforcement learning is a competitive alternative to standard methods, in particular to generate lightweight policies suitable for robots.