Microswimmers learning chemotaxis with genetic algorithms

Microbial electrotaxis: rewiring environmental microbiomes

Electric fields in sediments and soils are critical yet overlooked drivers of microbial ecology. This review examines the importance of electrotaxis in shaping microbial community dynamics and ecology models, surpassing traditional frameworks centered on chemotaxis. We analyze evidence that electric field gradients influence microbial community structure, function, and biogeochemical cycles in natural environments. Current mechanistic models, primarily based on eukaryotic systems, insufficiently explain bacterial electrotactic responses, necessitating new conceptual frameworks that integrate electrochemical and biological perspectives. We also evaluate its applications in environmental and microbiome engineering, with future research recommendations and methodologies in electrotaxis research. This synthesis aims to establish electrotaxis as an essential consideration in microbial ecology, presenting both challenges and opportunities for advancing our understanding of microbial ecosystems.

SwarmRL: building the future of smart active systems

This work introduces SwarmRL, a Python package designed to study intelligent active particles. SwarmRL provides an easy-to-use interface for developing models to control microscopic colloids using classical control and deep reinforcement learning approaches. These models may be deployed in simulations or real-world environments under a common framework. We explain the structure of the software and its key features and demonstrate how it can be used to accelerate research. With SwarmRL, we aim to streamline research into micro-robotic control while bridging the gap between experimental and simulation-driven sciences. SwarmRL is available open-source on GitHub at https://github.com/SwarmRL/SwarmRL .

Reinforcement learning selects multimodal locomotion strategies for bioinspired microswimmers

Natural microswimmers exhibit multimodal locomotion strategies to achieve versatile navigation tasks such as finding nutrient sources, avoiding danger and migrating to new habitats. These multimodal locomotion strategies typically involve complex coordination of cell actuators (i.e., flagella) to generate translation, rotation and combined motions. Yet, it is generally difficult to establish a simple relationship between actuation and locomotion strategies due to the complex hydrodynamic coupling between the swimmer and the surrounding fluid. While many bioinspired microswimmers have been engendered, it remains challenging for these artificial swimmers to generate effective locomotion strategies for different functional tasks similar to their biological counterparts. Here, we explore a reinforcement learning (RL) method to enable a bioinspired microswimmer to select locomotion strategies based on different functional tasks. We illustrate this approach using a bioinspired model swimmer derived from Chlamydomonas reinhardtii, which consists of a body sphere and two flagella spheres. We first demonstrate that this RL-powered bioinspired swimmer can select effective locomotion strategies that maximize displacement or minimize energy input by setting corresponding learning goals. We further illustrate how RL can enable the bioinspired swimmer to achieve multi-directional navigation via multimodal locomotion strategies that coordinately switch between forward and steering gaits. Our approach opens a new avenue to designing bioinspired microswimmers with multimodal locomotion capabilities.

Optimal Navigation in Microfluidics via the Optimization of a Discrete Loss

Optimal path planning and control of microscopic devices navigating in fluid environments is essential for applications ranging from targeted drug delivery to environmental monitoring. These tasks are challenging due to the complexity of microdevice-flow interactions. We introduce a closed-loop control method that optimizes a discrete loss (ODIL) in terms of dynamics and path objectives. In comparison with reinforcement learning, ODIL is more robust, up to 3 orders faster, and excels in high-dimensional action and state spaces, making it a powerful tool for navigating complex flow environments.

Field-Directed Motion, Cargo Capture, and Closed-Loop Controlled Navigation of Microellipsoids

Microrobots have the potential for diverse applications, including targeted drug delivery and minimally invasive surgery. Despite advancements in microrobot design and actuation strategies, achieving precise control over their motion remains challenging due to the dominance of viscous drag, system disturbances, physicochemical heterogeneities, and stochastic Brownian forces. Here, a precise control over the interfacial motion of model microellipsoids is demonstrated using time-varying rotating magnetic fields. The impacts of microellipsoid aspect ratio, field characteristics, and magnetic properties of the medium and the particle on the motion are investigated. The role of mobile micro-vortices generated is highlighted by rotating microellipsoids in capturing, transporting, and releasing cargo objects. Furthermore, an approach is presented for controlled navigation through mazes based on real-time particle and obstacle sensing, path planning, and magnetic field actuation without human intervention. The study introduces a mechanism of directing motion of microparticles using rotating magnetic fields, and a control scheme for precise navigation and delivery of micron-sized cargo using simple microellipsoids as microbots.

Reinforcement learning of biomimetic navigation: a model problem for sperm chemotaxis

Motile biological cells can respond to local environmental cues and exhibit various navigation strategies to search for specific targets. These navigation strategies usually involve tuning of key biophysical parameters of the cells, such that the cells can modulate their trajectories to move in response to the detected signals. Here we introduce a reinforcement learning approach to modulate key biophysical parameters and realize navigation strategies reminiscent to those developed by biological cells. We present this approach using sperm chemotaxis toward an egg as a paradigm. By modulating the trajectory curvature of a sperm cell model, the navigation strategies informed by reinforcement learning are capable to resemble sperm chemotaxis observed in experiments. This approach provides an alternative method to capture biologically relevant navigation strategies, which may inform the necessary parameter modulations required for obtaining specific navigation strategies and guide the design of biomimetic micro-robotics.

Machine learning assisted sorting of active microswimmers

Active matter systems, being in a non-equilibrium state, exhibit complex behaviors, such as self-organization, giving rise to emergent phenomena. There are many examples of active particles with biological origins, including bacteria and spermatozoa, or with artificial origins, such as self-propelled swimmers and Janus particles. The ability to manipulate active particles is vital for their effective application, e.g., separating motile spermatozoa from nonmotile and dead ones, to increase fertilization chance. In this study, we proposed a mechanism-an apparatus-to sort and demix active particles based on their motility values (Péclet number). Initially, using Brownian simulations, we demonstrated the feasibility of sorting self-propelled particles. Following this, we employed machine learning methods, supplemented with data from comprehensive simulations that we conducted for this study, to model the complex behavior of active particles. This enabled us to sort them based on their Péclet number. Finally, we evaluated the performance of the developed models and showed their effectiveness in demixing and sorting the active particles. Our findings can find applications in various fields, including physics, biology, and biomedical science, where the sorting and manipulation of active particles play a pivotal role.

Learning optimal integration of spatial and temporal information in noisy chemotaxis

We investigate the boundary between chemotaxis driven by spatial estimation of gradients and chemotaxis driven by temporal estimation. While it is well known that spatial chemotaxis becomes disadvantageous for small organisms at high noise levels, it is unclear whether there is a discontinuous switch of optimal strategies or a continuous transition exists. Here, we employ deep reinforcement learning to study the possible integration of spatial and temporal information in an a priori unconstrained manner. We parameterize such a combined chemotactic policy by a recurrent neural network and evaluate it using a minimal theoretical model of a chemotactic cell. By comparing with constrained variants of the policy, we show that it converges to purely temporal and spatial strategies at small and large cell sizes, respectively. We find that the transition between the regimes is continuous, with the combined strategy outperforming in the transition region both the constrained variants as well as models that explicitly integrate spatial and temporal information. Finally, by utilizing the attribution method of integrated gradients, we show that the policy relies on a nontrivial combination of spatially and temporally derived gradient information in a ratio that varies dynamically during the chemotactic trajectories.

Smart active particles learn and transcend bacterial foraging strategies

Throughout evolution, bacteria and other microorganisms have learned efficient foraging strategies that exploit characteristic properties of their unknown environment. While much research has been devoted to the exploration of statistical models describing the dynamics of foraging bacteria and other (micro-) organisms, little is known, regarding the question of how good the learned strategies actually are. This knowledge gap is largely caused by the absence of methods allowing to systematically develop alternative foraging strategies to compare with. In the present work, we use deep reinforcement learning to show that a smart run-and-tumble agent, which strives to find nutrients for its survival, learns motion patterns that are remarkably similar to the trajectories of chemotactic bacteria. Strikingly, despite this similarity, we also find interesting differences between the learned tumble rate distribution and the one that is commonly assumed for the run and tumble model. We find that these differences equip the agent with significant advantages regarding its foraging and survival capabilities. Our results uncover a generic route to use deep reinforcement learning for discovering search and collection strategies that exploit characteristic but initially unknown features of the environment. These results can be used, e.g., to program future microswimmers, nanorobots, and smart active particles for tasks like searching for cancer cells, micro-waste collection, or environmental remediation.

Learning how to find targets in the micro-world: the case of intermittent active Brownian particles

Finding the best strategy to minimize the time needed to find a given target is a crucial task both in nature and in reaching decisive technological advances. By considering learning agents able to switch their dynamics between standard and active Brownian motion, here we focus on developing effective target-search behavioral policies for microswimmers navigating a homogeneous environment and searching for targets of unknown position. We exploit projective simulation, a reinforcement learning algorithm, to acquire an efficient stochastic policy represented by the probability of switching the phase, i.e. the navigation mode, in response to the type and the duration of the current phase. Our findings reveal that the target-search efficiency increases with the particle's self-propulsion during the active phase and that, while the optimal duration of the passive case decreases monotonically with the activity, the optimal duration of the active phase displays a non-monotonic behavior.

AI-enhanced biomedical micro/nanorobots in microfluidics

Human beings encompass sophisticated microcirculation and microenvironments, incorporating a broad spectrum of microfluidic systems that adopt fundamental roles in orchestrating physiological mechanisms. In vitro recapitulation of human microenvironments based on lab-on-a-chip technology represents a critical paradigm to better understand the intricate mechanisms. Moreover, the advent of micro/nanorobotics provides brand new perspectives and dynamic tools for elucidating the complex process in microfluidics. Currently, artificial intelligence (AI) has endowed micro/nanorobots (MNRs) with unprecedented benefits, such as material synthesis, optimal design, fabrication, and swarm behavior. Using advanced AI algorithms, the motion control, environment perception, and swarm intelligence of MNRs in microfluidics are significantly enhanced. This emerging interdisciplinary research trend holds great potential to propel biomedical research to the forefront and make valuable contributions to human health. Herein, we initially introduce the AI algorithms integral to the development of MNRs. We briefly revisit the components, designs, and fabrication techniques adopted by robots in microfluidics with an emphasis on the application of AI. Then, we review the latest research pertinent to AI-enhanced MNRs, focusing on their motion control, sensing abilities, and intricate collective behavior in microfluidics. Furthermore, we spotlight biomedical domains that are already witnessing or will undergo game-changing evolution based on AI-enhanced MNRs. Finally, we identify the current challenges that hinder the practical use of the pioneering interdisciplinary technology.

Bioelectric control of locomotor gaits in the walking ciliate Euplotes

Diverse animal species exhibit highly stereotyped behavioral actions and locomotor sequences as they explore their natural environments. In many such cases, the neural basis of behavior is well established, where dedicated neural circuitry contributes to the initiation and regulation of certain response sequences. At the microscopic scale, single-celled eukaryotes (protists) also exhibit remarkably complex behaviors and yet are completely devoid of nervous systems. Here, to address the question of how single cells control behavior, we study locomotor patterning in the exemplary hypotrich ciliate Euplotes, a highly polarized cell, which actuates a large number of leg-like appendages called cirri (each a bundle of ∼25-50 cilia) to swim in fluids or walk on surfaces. As it navigates its surroundings, a walking Euplotes cell is routinely observed to perform side-stepping reactions, one of the most sophisticated maneuvers ever observed in a single-celled organism. These are spontaneous and stereotyped reorientation events involving a transient and fast backward motion followed by a turn. Combining high-speed imaging with simultaneous time-resolved electrophysiological recordings, we show that this complex coordinated motion sequence is tightly regulated by rapid membrane depolarization events, which orchestrate the activity of different cirri on the cell. Using machine learning and computer vision methods, we map detailed measurements of cirri dynamics to the cell's membrane bioelectrical activity, revealing a differential response in the front and back cirri. We integrate these measurements with a minimal model to understand how Euplotes-a unicellular organism-manipulates its membrane potential to achieve real-time control over its motor apparatus.

Chemotaxis of an elastic flagellated microrobot

Machine learning algorithms offer a tool to boost mobility and flexibility of a synthetic microswimmer, hence may help us design truly smart microrobots. In this work, we design a two-gait microrobot swimming in circular or helical trajectory. It utilizes the coupling between flagellum elasticity and resistive force to change the characteristics of swimming trajectory. Leveraging a deep reinforcement learning (DRL) approach, we show that the microrobot can self-learn chemotactic motion autonomously (without heuristics) using only several current and historical chemoattractant concentration and curvature information. The learned strategy is more efficient than a human-devised shortsighted strategy and can be further greatly improved in a stochastic environment. Furthermore, in the helical trajectory case, if additional heuristic information of direction is supplemented to evaluate the strategy during the learning process, then a highly efficient strategy can be discovered by the DRL. The microrobot can quickly align the helix vector to the gradient direction using just several smart sequential gait switchings. The success for the efficient strategies depends on how much historical information is provided and also the steering angle step size of the microrobot. Our results provide useful guidance for the design and smart maneuver of synthetic spermlike microswimmers.

Learning to cooperate for low-Reynolds-number swimming: a model problem for gait coordination

Biological microswimmers can coordinate their motions to exploit their fluid environment-and each other-to achieve global advantages in their locomotory performance. These cooperative locomotion require delicate adjustments of both individual swimming gaits and spatial arrangements of the swimmers. Here we probe the emergence of such cooperative behaviors among artificial microswimmers endowed with artificial intelligence. We present the first use of a deep reinforcement learning approach to empower the cooperative locomotion of a pair of reconfigurable microswimmers. The AI-advised cooperative policy comprises two stages: an approach stage where the swimmers get in close proximity to fully exploit hydrodynamic interactions, followed a synchronization stage where the swimmers synchronize their locomotory gaits to maximize their overall net propulsion. The synchronized motions allow the swimmer pair to move together coherently with an enhanced locomotion performance unattainable by a single swimmer alone. Our work constitutes a first step toward uncovering intriguing cooperative behaviors of smart artificial microswimmers, demonstrating the vast potential of reinforcement learning towards intelligent autonomous manipulations of multiple microswimmers for their future biomedical and environmental applications.

Biodegradable Microrobots and Their Biomedical Applications: A Review

During recent years, microrobots have drawn extensive attention owing to their good controllability and great potential in biomedicine. Powered by external physical fields or chemical reactions, these untethered microdevices are promising candidates for in vivo complex tasks, such as targeted delivery, imaging and sensing, tissue engineering, hyperthermia, and assisted fertilization, among others. However, in clinical use, the biodegradability of microrobots is significant for avoiding toxic residue in the human body. The selection of biodegradable materials and the corresponding in vivo environment needed for degradation are increasingly receiving attention in this regard. This review aims at analyzing different types of biodegradable microrobots by critically discussing their advantages and limitations. The chemical degradation mechanisms behind biodegradable microrobots and their typical applications are also thoroughly investigated. Furthermore, we examine their feasibility and deal with the in vivo suitability of different biodegradable microrobots in terms of their degradation mechanisms; pathological environments; and corresponding biomedical applications, especially targeted delivery. Ultimately, we highlight the prevailing obstacles and perspective solutions, ranging from their manufacturing methods, control of movement, and degradation rate to insufficient and limited in vivo tests, that could be of benefit to forthcoming clinical applications.

Searching for a source without gradients: how good is infotaxis and how to beat it

Infotaxis is a popular search algorithm designed to track a source of odour in a turbulent environment using information provided by odour detections. To exemplify its capabilities, the source-tracking task was framed as a partially observable Markov decision process consisting in finding, as fast as possible, a stationary target hidden in a two-dimensional grid using stochastic partial observations of the target location. Here, we provide an extended review of infotaxis, together with a toolkit for devising better strategies. We first characterize the performance of infotaxis in domains from one dimension to four dimensions. Our results show that, while being suboptimal, infotaxis is reliable (the probability of not reaching the source approaches zero), efficient (the mean search time scales as expected for the optimal strategy) and safe (the tail of the distribution of search times decays faster than any power law, though subexponentially). We then present three possible ways of beating infotaxis, all inspired by methods used in artificial intelligence: tree search, heuristic approximation of the value function, and deep reinforcement learning. The latter is able to find, without any prior human knowledge, the (near) optimal strategy. Altogether, our results provide evidence that the margin of improvement of infotaxis towards the optimal strategy gets smaller as the dimensionality increases.

Hunting active Brownian particles: Learning optimal behavior

We numerically study active Brownian particles that can respond to environmental cues through a small set of actions (switching their motility and turning left or right with respect to some direction) which are motivated by recent experiments with colloidal self-propelled Janus particles. We employ reinforcement learning to find optimal mappings between the state of particles and these actions. Specifically, we first consider a predator-prey situation in which prey particles try to avoid a predator. Using as reward the squared distance from the predator, we discuss the merits of three state-action sets and show that turning away from the predator is the most successful strategy. We then remove the predator and employ as collective reward the local concentration of signaling molecules exuded by all particles and show that aligning with the concentration gradient leads to chemotactic collapse into a single cluster. Our results illustrate a promising route to obtain local interaction rules and design collective states in active matter.

Stabilizing viscous extensional flows using reinforcement learning

The four-roll mill, wherein four identical cylinders undergo rotation of identical magnitude but alternate signs, was originally proposed by G. I. Taylor to create local extensional flows and study their ability to deform small liquid drops. Since an extensional flow has an unstable eigendirection, a drop located at the flow stagnation point will have a tendency to escape. This unstable dynamics can, however, be stabilized using, e.g., a modulation of the rotation rates of the cylinders. Here we use reinforcement learning, a branch of machine learning devoted to the optimal selection of actions based on cumulative rewards, in order to devise a stabilization algorithm for the four-roll mill flow. The flow is modelled as the linear superposition of four two-dimensional rotlets and the drop is treated as a rigid spherical particle smaller than all other length scales in the problem. Unlike previous attempts to devise control, we take a probabilistic approach whereby speed adjustments are drawn from a probability density function whose shape is improved over time via a form of gradient ascent know as actor-critic method. With enough training, our algorithm is able to precisely control the drop and keep it close to the stagnation point for as long as needed. We explore the impact of the physical and learning parameters on the effectiveness of the control and demonstrate the robustness of the algorithm against thermal noise. We finally show that reinforcement learning can provide a control algorithm effective for all initial positions and that can be adapted to limit the magnitude of the flow extension near the position of the drop.