Evolution of Biological Image Stabilization

Moving in an Uncertain World: Robust and Adaptive Control of Locomotion from Organisms to Machine Intelligence

Whether walking, running, slithering, or flying, organisms display a remarkable ability to move through complex and uncertain environments. In particular, animals have evolved to cope with a host of uncertainties-both of internal and external origin-to maintain adequate performance in an ever-changing world. In this review, we present mathematical methods in engineering to highlight emerging principles of robust and adaptive control of organismal locomotion. Specifically, by drawing on the mathematical framework of control theory, we decompose the robust and adaptive hierarchical structure of locomotor control. We show how this decomposition along the robust-adaptive axis provides testable hypotheses to classify behavioral outcomes to perturbations. With a focus on studies in non-human animals, we contextualize recent findings along the robust-adaptive axis by emphasizing two broad classes of behaviors: (1) compensation to appendage loss and (2) image stabilization and fixation. Next, we attempt to map robust and adaptive control of locomotion across some animal groups and existing bio-inspired robots. Finally, we highlight exciting future directions and interdisciplinary collaborations that are needed to unravel principles of robust and adaptive locomotion.

Divergent visual ecology of Drosophila species drives object-tracking strategies matched to landscape sparsity

Maintaining stable gaze while tracking moving objects is commonplace across animal taxa, yet how diverse ecological needs impact these processes is poorly understood. During flight, the fruit-eating fly Drosophila melanogaster maintains course by making smooth steering adjustments to fixate the image of the distant visual background on the retina, while executing body saccades to investigate nearby objects such as food sources. Cactophilic Drosophila mojavensis live where there is no canopy; rather, the flora forming visual "background" and "objects" are one and the same. We tested whether D. mojavensis have adapted their flight control strategies for a visually sparse landscape. We used a magnetic tether that allows free movement in the yaw axis. In response to a textured bar moving across a similarly textured stationary background, D. melanogaster fixates the background, thereby stabilizing gaze while integrating bar dynamics to trigger tracking saccades. By contrast, two mojavensis subspecies in the repleta subgroup and one species in the melanogaster subgroup steer to smoothly fixate the bar, seemingly ignoring the stationary surround. Desert flies execute frequent bar-tracking saccades, but theirs are triggered when rotational velocity lags the bar. Thus, D. melanogaster, which lives in visually cluttered cosmopolitan habitats, leverages the optical disparities between nearby objects and distant foliage for a hybrid control strategy: "ground-fixate, object-saccade." Flies in distant phylogenetic subgroups with similar visual ecology use a "fixate-and-saccade" strategy, which would be adaptive in a visually sparse environment where individual landscape features are both approached and used to maintain a straight course.

Human proprioceptive gaze stabilization during passive body rotations underneath a fixed head

The present study explored the presence of torsional gaze-stabilization to proprioceptive neck activation in humans. Thirteen healthy subjects (6 female, mean age 25) were exposed to passive body rotations while maintaining a head-fixed, gravitationally upright, position. Participants were seated in a mechanical sled, their heads placed in a chin rest embedded in a wooden beam while wearing an eye tracker attached to the beam using strong rubber bands to ensure head stability. The body was passively rotated underneath the head both in darkness and while viewing a projected visual scene. Static torsional gaze positions were compared between the baseline position prior to the stimulation, and immediately after the final body tilt had been reached. Results showed that passive neck flexion produced ocular torsion when combined with a visual background. The eyes exhibited rotations in the opposite direction of the neck's extension, matching a hypothetical head tilt in the same direction as the sled. This corresponded with a predicted head rotation aimed at straightening the head in relation to the body. No such response was seen during trials in darkness. Altogether, these findings suggest that proprioception may produce a predictive gaze-stabilizing response in humans.

Temporal dynamics of ocular torsion and vertical vergence during visual, vestibular, and visuovestibular rotations

Ocular torsion and vertical divergence reflect the brain's sensorimotor integration of motion through the vestibulo-ocular reflex (VOR) and the optokinetic reflex (OKR) to roll rotations. Torsion and vergence however express different response patterns depending on several motion variables, but research on their temporal dynamics remains limited. This study investigated the onset times of ocular torsion (OT) and vertical vergence (VV) during visual, vestibular, and visuovestibular motion, as well as their relative decay rates following prolonged optokinetic stimulations. Temporal characteristics were retrieved from three separate investigations where the level of visual clutter and acceleration were controlled. Video eye-tracking was used to retrieve the eye-movement parameters from a total of 41 healthy participants across all trials. Ocular torsion consistently initiated earlier than vertical vergence, particularly evident under intensified visual information density, and higher clutter levels were associated with more balanced decay rates. Additionally, stimulation modality and accelerations affected the onsets of both eye movements, with visuovestibular motion triggering earlier responses compared to vestibular motion, and increased accelerations leading to earlier onsets for both movements. The present study showed that joint visuovestibular responses produced more rapid onsets, indicating a synergetic sensorimotor process. It also showed that visual content acted as a fusional force during the decay period, and imposed greater influence over the torsional onset compared to vergence. Acceleration, by contrast, did not affect the temporal relationship between the two eye movements. Altogether, these findings provide insights into the sensorimotor integration of the vestibulo-ocular and optokinetic reflex arcs.

Asynchronous haltere input drives specific wing and head movements in Drosophila

Halteres are multifunctional mechanosensory organs unique to the true flies (Diptera). A set of reduced hindwings, the halteres beat at the same frequency as the lift-generating forewings and sense inertial forces via mechanosensory campaniform sensilla. Though haltere ablation makes stable flight impossible, the specific role of wing-synchronous input has not been established. Using small iron filings attached to the halteres of tethered flies and an alternating electromagnetic field, we experimentally decoupled the wings and halteres of flying Drosophila and observed the resulting changes in wingbeat amplitude and head orientation. We find that asynchronous haltere input results in fast amplitude changes in the wing (hitches), but does not appreciably move the head. In multi-modal experiments, we find that wing and gaze optomotor responses are disrupted differently by asynchronous input. These effects of wing-asynchronous haltere input suggest that specific sensory information is necessary for maintaining wing amplitude stability and adaptive gaze control.

Sensorimotor adaptation to destabilizing dynamics in weakly electric fish

Humans and other animals can readily learn to compensate for changes in the dynamics of movement. Such changes can result from an injury or changes in the weight of carried objects. These changes in dynamics can lead not only to reduced performance but also to dramatic instabilities. We evaluated the impacts of compensatory changes in control policies in relation to stability and robustness in Eigenmannia virescens, a species of weakly electric fish. We discovered that these fish retune their sensorimotor control system in response to experimentally generated destabilizing dynamics. Specifically, we used an augmented reality system to manipulate sensory feedback during an image stabilization task in which a fish maintained its position within a refuge. The augmented reality system measured the fish's movements in real time. These movements were passed through a high-pass filter and multiplied by a gain factor before being fed back to the refuge motion. We adjusted the gain factor to gradually destabilize the fish's sensorimotor loop. The fish retuned their sensorimotor control system to compensate for the experimentally induced destabilizing dynamics. This retuning was partially maintained when the augmented reality feedback was abruptly removed. The compensatory changes in sensorimotor control improved tracking performance as well as control-theoretic measures of robustness, including reduced sensitivity to disturbances and improved phase margins.

A Proposed Mechanism for Visual Vertigo: Post-Concussion Patients Have Higher Gain From Visual Input Into Subcortical Gaze Stabilization

Purpose

Post-concussion syndrome (PCS) is commonly associated with dizziness and visual motion sensitivity. This case-control study set out to explore altered motion processing in PCS by measuring gaze stabilization as a reflection of the capacity of the brain to integrate motion, and it aimed to uncover mechanisms of injury where invasive subcortical recordings are not feasible.

Methods

A total of 554 eye movements were analyzed in 10 PCS patients and nine healthy controls across 171 trials. Optokinetic and vestibulo-ocular reflexes were recorded using a head-mounted eye tracker while participants were exposed to visual, vestibular, and visuo-vestibular motion stimulations in the roll plane. Torsional and vergence eye movements were analyzed in terms of slow-phase velocities, gain, nystagmus frequency, and sensory-specific contributions toward gaze stabilization.

Results

Participants expressed eye-movement responses consistent with expected gaze stabilization; slow phases were fastest for visuo-vestibular trials and slowest for visual stimulations (P < 0.001) and increased with stimulus acceleration (P < 0.001). Concussed patients demonstrated increased gain from visual input to gaze stabilization (P = 0.005), faster slow phases (P = 0.013), earlier nystagmus beats (P = 0.003), and higher relative visual influence over the gaze-stabilizing response (P = 0.001), presenting robust effect sizes despite the limited population size.

Conclusions

The enhanced neural responsiveness to visual motion in PCS, combined with semi-intact visuo-vestibular integration, presented a subcortical hierarchy for altered gaze stabilization. Drawing on comparable animal trials, findings suggest that concussed patients may suffer from diffuse injuries to inhibiting pathways for optokinetic information, likely early in the visuo-vestibular hierarchy of sensorimotor integration. These findings offer context for common but elusive symptoms, presenting a neurological explanation for motion sensitivity and visual vertigo in PCS.

Drosophila flying in augmented reality reveals the vision-based control autonomy of the optomotor response

For walking, swimming, and flying animals, the optomotor response is essential to stabilize gaze. How flexible is the optomotor response? Classic work in Drosophila has argued that flies adapt flight control under augmented visual feedback conditions during goal-directed bar fixation. However, whether the lower-level, reflexive optomotor response can similarly adapt to augmented visual feedback (partially autonomous) or not (autonomous) over long timescales is poorly understood. To address this question, we developed an augmented reality paradigm to study the vision-based control autonomy of the yaw optomotor response of flying fruit flies (Drosophila). Flies were placed in a flight simulator, which permitted free body rotation about the yaw axis. By feeding back body movements in real time to a visual display, we augmented and inverted visual feedback. Thus, this experimental paradigm caused a constant visual error between expected and actual visual feedback to study potential adaptive visuomotor control. By combining experiments with control theory, we demonstrate that the optomotor response is autonomous during augmented reality flight bouts of up to 30 min, which exceeds the reported learning epoch during bar fixation. Agreement between predictions from linear systems theory and experimental data supports the notion that the optomotor response is approximately linear and time invariant within our experimental assay. Even under positive visual feedback, which revealed the stability limit of flies in augmented reality, the optomotor response was autonomous. Our results support a hierarchical motor control architecture in flies with fast and autonomous reflexes at the bottom and more flexible behavior at higher levels.

The motor apparatus of head movements in the Oleander hawkmoth (Daphnis nerii, Lepidoptera)

Head movements of insects play a vital role in diverse locomotory behaviors including flying and walking. Because insect eyes move minimally within their sockets, their head movements are essential to reduce visual blur and maintain a stable gaze. As in most vertebrates, gaze stabilization behavior in insects requires the integration of both visual and mechanosensory feedback by the neck motor neurons. Although visual feedback is derived from the optic flow over the retina of their compound eyes, mechanosensory feedback is derived from their organs of balance, similar to the vestibular system in vertebrates. In Diptera, vestibular feedback is derived from the halteres-modified hindwings that evolved into mechanosensory organs-and is integrated with visual feedback to actuate compensatory head movements. However, non-Dipteran insects, including Lepidoptera, lack halteres. In these insects, vestibular feedback is obtained from the antennal Johnston's organs but it is not well-understood how it integrates with visual feedback during head movements. Indeed, although head movements are well-studied in flies, the underlying motor apparatus in non-Dipteran taxa has received relatively less attention. As a first step toward understanding compensatory head movements in the Oleander hawkmoth Daphnis nerii, we image the anatomy and architecture of their neck joint sclerites and muscles using X-ray microtomography, and the associated motor neurons using fluorescent dye fills and confocal microscopy. Based on these morphological data, we propose testable hypotheses about the putative function of specific neck muscles during head movements, which can shed light on their role in neck movements and gaze stabilization.

Neural mechanisms to incorporate visual counterevidence in self-movement estimation

In selecting appropriate behaviors, animals should weigh sensory evidence both for and against specific beliefs about the world. For instance, animals measure optic flow to estimate and control their own rotation. However, existing models of flow detection can be spuriously triggered by visual motion created by objects moving in the world. Here, we show that stationary patterns on the retina, which constitute evidence against observer rotation, suppress inappropriate stabilizing rotational behavior in the fruit fly Drosophila. In silico experiments show that artificial neural networks (ANNs) that are optimized to distinguish observer movement from external object motion similarly detect stationarity and incorporate negative evidence. Employing neural measurements and genetic manipulations, we identified components of the circuitry for stationary pattern detection, which runs parallel to the fly's local motion and optic-flow detectors. Our results show how the fly brain incorporates negative evidence to improve heading stability, exemplifying how a compact brain exploits geometrical constraints of the visual world.

Drinking on the wing: water collection in polarotactic horseflies

Many insects detect water bodies by observing the linearly polarised light which is reflected from the water surface. Polarotactic horseflies exhibit acrobatic manoeuvres above the water and are able to plunge on its surface, collect a droplet and fly away. This behaviour is extremely fast and has not yet been analysed. We recorded the flight patterns and kinematics of drinking horseflies using a pair of high-speed cameras. The animals of both sexes are attracted to water puddles where they make short, millisecond pitstops to collect a droplet of water that is then presumably drank "on the wing". Before the collection, the flies perform several low-altitude flybys above the puddle. After a few passes, the fly suddenly reverses its body orientation, decelerates, briefly touches the water surface and immediately flies away, usually with a droplet carried between its front legs. During the approach flight, the horseflies fly low but do not show any angular preference. Thus, they view the reflections from the sky, sun, or vegetation with a wide band of ventral ommatidia. Polarotaxis in drinking horseflies is a very robust visually guided behaviour, which operates at a broad range of intensities and various spectral compositions of reflected light.

Descending neurons of the hoverfly respond to pursuits of artificial targets

Many animals use motion vision information to control dynamic behaviors. Predatory animals, for example, show an exquisite ability to detect rapidly moving prey, followed by pursuit and capture. Such target detection is not only used by predators but is also important in conspecific interactions, such as for male hoverflies defending their territories against conspecific intruders. Visual target detection is believed to be subserved by specialized target-tuned neurons found in a range of species, including vertebrates and arthropods. However, how these target-tuned neurons respond to actual pursuit trajectories is currently not well understood. To redress this, we recorded extracellularly from target-selective descending neurons (TSDNs) in male Eristalis tenax hoverflies. We show that they have dorso-frontal receptive fields with a preferred direction up and away from the visual midline. We reconstructed visual flow fields as experienced during pursuits of artificial targets (black beads). We recorded TSDN responses to six reconstructed pursuits and found that each neuron responded consistently at remarkably specific time points but that these time points differed between neurons. We found that the observed spike probability was correlated with the spike probability predicted from each neuron's receptive field and size tuning. Interestingly, however, the overall response rate was low, with individual neurons responding to only a small part of each reconstructed pursuit. In contrast, the TSDN population responded to substantially larger proportions of the pursuits but with lower probability. This large variation between neurons could be useful if different neurons control different parts of the behavioral output.

Neural mechanisms to incorporate visual counterevidence in self motion estimation

In selecting appropriate behaviors, animals should weigh sensory evidence both for and against specific beliefs about the world. For instance, animals measure optic flow to estimate and control their own rotation. However, existing models of flow detection can confuse the movement of external objects with genuine self motion. Here, we show that stationary patterns on the retina, which constitute negative evidence against self rotation, are used by the fruit fly Drosophila to suppress inappropriate stabilizing rotational behavior. In silico experiments show that artificial neural networks optimized to distinguish self and world motion similarly detect stationarity and incorporate negative evidence. Employing neural measurements and genetic manipulations, we identified components of the circuitry for stationary pattern detection, which runs parallel to the fly's motion- and optic flow-detectors. Our results exemplify how the compact brain of the fly incorporates negative evidence to improve heading stability, exploiting geometrical constraints of the visual world.

Visual versus visual-inertial guidance in hawks pursuing terrestrial targets

The aerial interception behaviour of falcons is well modelled by a guidance law called proportional navigation, which commands steering at a rate proportional to the angular rate of the line-of-sight from predator to prey. Because the line-of-sight rate is defined in an inertial frame of reference, proportional navigation must be implemented using visual-inertial sensor fusion. By contrast, the aerial pursuit behaviour of hawks chasing terrestrial targets is better modelled by a mixed guidance law combining information on the line-of-sight rate with information on the deviation angle between the attacker's velocity and the line-of-sight. Here we ask whether this behaviour may be controlled using visual information alone. We use high-speed motion capture to record n = 228 flights from N = 4 Harris' hawks Parabuteo unicinctus, and show that proportional navigation and mixed guidance both model their trajectories well. The mixed guidance law also models the data closely when visual-inertial information on the line-of-sight rate is replaced by visual information on the motion of the target relative to its background. Although the visual-inertial form of the mixed guidance law provides the closest fit, all three guidance laws provide an adequate phenomenological model of the behavioural data, whilst making different predictions on the physiological pathways involved.

Proprioception gates visual object fixation in flying flies

Visual object tracking in animals as diverse as felines, frogs, and fish supports behaviors including predation, predator avoidance, and landscape navigation. Decades of experimental results show that a rigidly body-fixed tethered fly in a "virtual reality" visual flight simulator steers to follow the motion of a vertical bar, thereby "fixating" it on visual midline. This behavior likely reflects a desire to seek natural features such as plant stalks and has inspired algorithms for visual object tracking predicated on robust responses to bar velocity, particularly near visual midline. Using a modified flight simulator equipped with a magnetic pivot to allow frictionless turns about the yaw axis, we discovered that bar fixation as well as smooth steering responses to bar velocity are attenuated or eliminated in yaw-free conditions. Body-fixed Drosophila melanogaster respond to bar oscillation on a stationary ground with frequency-matched wing kinematics and fixate the bar on midline. Yaw-free flies respond to the same stimulus by ignoring the bar and maintaining their original heading. These differences are driven by proprioceptive signals, rather than visual signals, as artificially "clamping" a bar in the periphery of a yaw-free fly has no effect. When presented with a bar and ground oscillating at different frequencies, a yaw-free fly follows the frequency of the ground only, whereas a body-fixed fly robustly steers at the frequencies of both the bar and ground. Our findings support a model in which proprioceptive feedback promote active damping of high-gain optomotor responses to object motion.

Optic flow based spatial vision in insects

The optic flow, i.e., the displacement of retinal images of objects in the environment induced by self-motion, is an important source of spatial information, especially for fast-flying insects. Spatial information over a wide range of distances, from the animal's immediate surroundings over several hundred metres to kilometres, is necessary for mediating behaviours, such as landing manoeuvres, collision avoidance in spatially complex environments, learning environmental object constellations and path integration in spatial navigation. To facilitate the processing of spatial information, the complexity of the optic flow is often reduced by active vision strategies. These result in translations and rotations being largely separated by a saccadic flight and gaze mode. Only the translational components of the optic flow contain spatial information. In the first step of optic flow processing, an array of local motion detectors provides a retinotopic spatial proximity map of the environment. This local motion information is then processed in parallel neural pathways in a task-specific manner and used to control the different components of spatial behaviour. A particular challenge here is that the distance information extracted from the optic flow does not represent the distances unambiguously, but these are scaled by the animal's speed of locomotion. Possible ways of coping with this ambiguity are discussed.

Contrast independent biologically inspired translational optic flow estimation

The visual systems of insects are relatively simple compared to humans. However, they enable navigation through complex environments where insects perform exceptional levels of obstacle avoidance. Biology uses two separable modes of optic flow to achieve this: rapid gaze fixation (rotational motion known as saccades); and the inter-saccadic translational motion. While the fundamental process of insect optic flow has been known since the 1950's, so too has its dependence on contrast. The surrounding visual pathways used to overcome environmental dependencies are less well known. Previous work has shown promise for low-speed rotational motion estimation, but a gap remained in the estimation of translational motion, in particular the estimation of the time to impact. To consistently estimate the time to impact during inter-saccadic translatory motion, the fundamental limitation of contrast dependence must be overcome. By adapting an elaborated rotational velocity estimator from literature to work for translational motion, this paper proposes a novel algorithm for overcoming the contrast dependence of time to impact estimation using nonlinear spatio-temporal feedforward filtering. By applying bioinspired processes, approximately 15 points per decade of statistical discrimination were achieved when estimating the time to impact to a target across 360 background, distance, and velocity combinations: a 17-fold increase over the fundamental process. These results show the contrast dependence of time to impact estimation can be overcome in a biologically plausible manner. This, combined with previous results for low-speed rotational motion estimation, allows for contrast invariant computational models designed on the principles found in the biological visual system, paving the way for future visually guided systems.

Small-amplitude head oscillations result from a multimodal head stabilization reflex in hawkmoths

In flying insects, head stabilization is an essential reflex that helps to reduce motion blur during fast aerial manoeuvres. This reflex is multimodal and requires the integration of visual and antennal mechanosensory feedback in hawkmoths, each operating as a negative-feedback-control loop. As in any negative-feedback system, the head stabilization system possesses inherent oscillatory dynamics that depend on the rate at which the sensorimotor components of the reflex operate. Consistent with this expectation, we observed small-amplitude oscillations in the head motion (or head wobble) of the oleander hawkmoth, Daphnis nerii, which are accentuated when sensory feedback is aberrant. Here, we show that these oscillations emerge from the inherent dynamics of the multimodal reflex underlying gaze stabilization, and that the amplitude of head wobble is a function of both the visual feedback and antennal mechanosensory feedback from the Johnston's organs. Our data support the hypothesis that head wobble results from a multimodal, dynamically stabilized reflex loop that mediates head positioning.

Visual and motor signatures of locomotion dynamically shape a population code for feature detection in Drosophila

Natural vision is dynamic: as an animal moves, its visual input changes dramatically. How can the visual system reliably extract local features from an input dominated by self-generated signals? In Drosophila, diverse local visual features are represented by a group of projection neurons with distinct tuning properties. Here, we describe a connectome-based volumetric imaging strategy to measure visually evoked neural activity across this population. We show that local visual features are jointly represented across the population, and a shared gain factor improves trial-to-trial coding fidelity. A subset of these neurons, tuned to small objects, is modulated by two independent signals associated with self-movement, a motor-related signal, and a visual motion signal associated with rotation of the animal. These two inputs adjust the sensitivity of these feature detectors across the locomotor cycle, selectively reducing their gain during saccades and restoring it during intersaccadic intervals. This work reveals a strategy for reliable feature detection during locomotion.

The role of colour patterns for the recognition of flowers by bees

Bees discriminate between many different colours of flower petals, but it is not well understood how they perceive and learn patterns frequently found in flowers with colourful structures. We used multi-spectral imaging to explore chromatic cues in concentric flower patterns as they are seen through the low-resolution eyes of the honeybee. We find a diversity of colour combinations, which suggests that plants might exploit the sensory capabilities of pollinators, like bees, that learn colours easily. A consistent feature is that the surround of the pattern has a stronger chromatic contrast to the foliage background than the centre. This can potentially facilitate the fast identification of floral objects within colourful scenes when a foraging bee moves through a flower patch. In behavioural experiments we trained and tested bees with three types of concentric patterns. They recognized and discriminated patterns accurately in most tests, relying flexibly on both chromatic and spatial cues. Only rarely, depending on the training stimulus, chromatic cues determined their choices whilst pattern cues were ignored. The variability of floral designs and the bees' flexibility in recalling colour and spatial information suggest a role for colour vision in pattern processing. Implications for the signalling strategies of flowers are discussed. This article is part of the theme issue 'Understanding colour vision: molecular, physiological, neuronal and behavioural studies in arthropods'.

Matched function of the neuropil processing optic flow in flies and crabs: the lobula plate mediates optomotor responses in Neohelice granulata

When an animal rotates (whether it is an arthropod, a fish, a bird or a human) a drift of the visual panorama occurs over its retina, termed optic flow. The image is stabilized by compensatory behaviours (driven by the movement of the eyes, head or the whole body depending on the animal) collectively termed optomotor responses. The dipteran lobula plate has been consistently linked with optic flow processing and the control of optomotor responses. Crabs have a neuropil similarly located and interconnected in the optic lobes, therefore referred to as a lobula plate too. Here we show that the crabs' lobula plate is required for normal optomotor responses since the response was lost or severely impaired in animals whose lobula plate had been lesioned. The effect was behaviour-specific, since avoidance responses to approaching visual stimuli were not affected. Crabs require simpler optic flow processing than flies (because they move slower and in two-dimensional instead of three-dimensional space), consequently their lobula plates are relatively smaller. Nonetheless, they perform the same essential role in the visual control of behaviour. Our findings add a fundamental piece to the current debate on the evolutionary relationship between the lobula plates of insects and crustaceans.

Conserved subcortical processing in visuo-vestibular gaze control

Gaze stabilization compensates for movements of the head or external environment to minimize image blurring. Multisensory information stabilizes the scene on the retina via the vestibulo-ocular (VOR) and optokinetic (OKR) reflexes. While the organization of neuronal circuits underlying VOR is well-described across vertebrates, less is known about the contribution and evolution of the OKR and the basic structures allowing visuo-vestibular integration. To analyze these neuronal pathways underlying visuo-vestibular integration, we developed a setup using a lamprey eye-brain-labyrinth preparation, which allowed coordinating electrophysiological recordings, vestibular stimulation with a moving platform, and visual stimulation via screens. Lampreys exhibit robust visuo-vestibular integration, with optokinetic information processed in the pretectum that can be downregulated from tectum. Visual and vestibular inputs are integrated at several subcortical levels. Additionally, saccades are present in the form of nystagmus. Thus, all basic components of the visuo-vestibular control of gaze were present already at the dawn of vertebrate evolution.

Impact of walking speed and motion adaptation on optokinetic nystagmus-like head movements in the blowfly Calliphora

The optokinetic nystagmus is a gaze-stabilizing mechanism reducing motion blur by rapid eye rotations against the direction of visual motion, followed by slower syndirectional eye movements minimizing retinal slip speed. Flies control their gaze through head turns controlled by neck motor neurons receiving input directly, or via descending neurons, from well-characterized directional-selective interneurons sensitive to visual wide-field motion. Locomotion increases the gain and speed sensitivity of these interneurons, while visual motion adaptation in walking animals has the opposite effects. To find out whether flies perform an optokinetic nystagmus, and how it may be affected by locomotion and visual motion adaptation, we recorded head movements of blowflies on a trackball stimulated by progressive and rotational visual motion. Flies flexibly responded to rotational stimuli with optokinetic nystagmus-like head movements, independent of their locomotor state. The temporal frequency tuning of these movements, though matching that of the upstream directional-selective interneurons, was only mildly modulated by walking speed or visual motion adaptation. Our results suggest flies flexibly control their gaze to compensate for rotational wide-field motion by a mechanism similar to an optokinetic nystagmus. Surprisingly, the mechanism is less state-dependent than the response properties of directional-selective interneurons providing input to the neck motor system.

Integration of visual and antennal mechanosensory feedback during head stabilization in hawkmoths

During flight maneuvers, insects exhibit compensatory head movements which are essential for stabilizing the visual field on their retina, reducing motion blur, and supporting visual self-motion estimation. In Diptera, such head movements are mediated via visual feedback from their compound eyes that detect retinal slip, as well as rapid mechanosensory feedback from their halteres - the modified hindwings that sense the angular rates of body rotations. Because non-Dipteran insects lack halteres, it is not known if mechanosensory feedback about body rotations plays any role in their head stabilization response. Diverse non-Dipteran insects are known to rely on visual and antennal mechanosensory feedback for flight control. In hawkmoths, for instance, reduction of antennal mechanosensory feedback severely compromises their ability to control flight. Similarly, when the head movements of freely flying moths are restricted, their flight ability is also severely impaired. The role of compensatory head movements as well as multimodal feedback in insect flight raises an interesting question: in insects that lack halteres, what sensory cues are required for head stabilization? Here, we show that in the nocturnal hawkmoth Daphnis nerii, compensatory head movements are mediated by combined visual and antennal mechanosensory feedback. We subjected tethered moths to open-loop body roll rotations under different lighting conditions, and measured their ability to maintain head angle in the presence or absence of antennal mechanosensory feedback. Our study suggests that head stabilization in moths is mediated primarily by visual feedback during roll movements at lower frequencies, whereas antennal mechanosensory feedback is required when roll occurs at higher frequency. These findings are consistent with the hypothesis that control of head angle results from a multimodal feedback loop that integrates both visual and antennal mechanosensory feedback, albeit at different latencies. At adequate light levels, visual feedback is sufficient for head stabilization primarily at low frequencies of body roll. However, under dark conditions, antennal mechanosensory feedback is essential for the control of head movements at high frequencies of body roll.

Motor-related signals support localization invariance for stable visual perception

Our ability to perceive a stable visual world in the presence of continuous movements of the body, head, and eyes has puzzled researchers in the neuroscience field for a long time. We reformulated this problem in the context of hierarchical convolutional neural networks (CNNs)-whose architectures have been inspired by the hierarchical signal processing of the mammalian visual system-and examined perceptual stability as an optimization process that identifies image-defining features for accurate image classification in the presence of movements. Movement signals, multiplexed with visual inputs along overlapping convolutional layers, aided classification invariance of shifted images by making the classification faster to learn and more robust relative to input noise. Classification invariance was reflected in activity manifolds associated with image categories emerging in late CNN layers and with network units acquiring movement-associated activity modulations as observed experimentally during saccadic eye movements. Our findings provide a computational framework that unifies a multitude of biological observations on perceptual stability under optimality principles for image classification in artificial neural networks.

Fast tuning of posture control by visual feedback underlies gaze stabilization in walking Drosophila

Locomotion requires a balance between mechanical stability and movement flexibility to achieve behavioral goals despite noisy neuromuscular systems, but rarely is it considered how this balance is orchestrated. We combined virtual reality tools with quantitative analysis of behavior to examine how Drosophila uses self-generated visual information (reafferent visual feedback) to control gaze during exploratory walking. We found that flies execute distinct motor programs coordinated across the body to maximize gaze stability. However, the presence of inherent variability in leg placement relative to the body jeopardizes fine control of gaze due to posture-stabilizing adjustments that lead to unintended changes in course direction. Surprisingly, whereas visual feedback is dispensable for head-body coordination, we found that self-generated visual signals tune postural reflexes to rapidly prevent turns rather than to promote compensatory rotations, a long-standing idea for visually guided course control. Together, these findings support a model in which visual feedback orchestrates the interplay between posture and gaze stability in a manner that is both goal dependent and motor-context specific.

Design, Analysis and Experiment of a Bridge-Type Piezoelectric Actuator for Infrared Image Stabilization

Piezoelectric actuators are widely used in the optical field due to their high precision, compact structure, flexible design, and fast response. This paper presents a novel piezoelectric actuator with a bridge-type mechanism, which can be used to stabilize the images of an infrared imaging system. The bridge amplification mechanism is used to amplify the actuation displacement, and its structural parameters are optimized by the response surface method. The control strategy of the image stabilization system is formulated, and the overall structure of the infrared image stabilization system is designed according to the principle of image stabilization and the control strategy. The prototype was fabricated and verified by a series of experiments. In the test, the laminated piezoelectric ceramics are used as the driving element, and its maximum output displacement was about 17 μm under a voltage of 100 V. Firstly, the performance of the piezoelectric amplification mechanism was tested, and the maximum displacement of the piezoelectric micro-motion mechanism was 115 μm. The displacement amplification ratio of the mechanism was 5.7. Then, the step distance and response time of the micro-displacement mechanism were measured by inputting the stepping signal. When the input voltage increased to 3 V, 5 V, and 7 V, the stepping displacements of the mechanism were 2.4 μm, 4.1 μm, and 5.8 μm. Finally, the image stabilization effect of the designed mechanism was verified by imaging timing control and feedback signal processing.

Mechanisms of punctuated vision in fly flight

To guide locomotion, animals control gaze via movements of their eyes, head, and/or body, but how the nervous system controls gaze during complex motor tasks remains elusive. In many animals, shifts in gaze consist of periods of smooth movement punctuated by rapid eye saccades. Notably, eye movements are constrained by anatomical limits, which requires resetting eye position. By studying tethered, flying fruit flies (Drosophila), we show that flies perform stereotyped head saccades to reset gaze, analogous to optokinetic nystagmus in primates. Head-reset saccades interrupted head smooth movement for as little as 50 ms-representing less than 5% of the total flight time-thereby enabling punctuated gaze stabilization. By revealing the passive mechanics of the neck joint, we show that head-reset saccades leverage the neck's natural elastic recoil, enabling mechanically assisted redirection of gaze. The consistent head orientation at saccade initiation, the influence of the head's angular position on saccade rate, the decrease in wing saccade frequency in head-fixed flies, and the decrease in head-reset saccade rate in flies with their head range of motion restricted together implicate proprioception as the primary trigger of head-reset saccades. Wing-reset saccades were influenced by head orientation, establishing a causal link between neck sensory signals and the execution of body saccades. Head-reset saccades were abolished when flies switched to a landing state, demonstrating that head movements are gated by behavioral state. We propose a control architecture for active vision systems with limits in sensor range of motion. VIDEO ABSTRACT.

Wireless steerable vision for live insects and insect-scale robots

Vision serves as an essential sensory input for insects but consumes substantial energy resources. The cost to support sensitive photoreceptors has led many insects to develop high visual acuity in only small retinal regions and evolve to move their visual systems independent of their bodies through head motion. By understanding the trade-offs made by insect vision systems in nature, we can design better vision systems for insect-scale robotics in a way that balances energy, computation, and mass. Here, we report a fully wireless, power-autonomous, mechanically steerable vision system that imitates head motion in a form factor small enough to mount on the back of a live beetle or a similarly sized terrestrial robot. Our electronics and actuator weigh 248 milligrams and can steer the camera over 60° based on commands from a smartphone. The camera streams "first person" 160 pixels-by-120 pixels monochrome video at 1 to 5 frames per second (fps) to a Bluetooth radio from up to 120 meters away. We mounted this vision system on two species of freely walking live beetles, demonstrating that triggering image capture using an onboard accelerometer achieves operational times of up to 6 hours with a 10-milliamp hour battery. We also built a small, terrestrial robot (1.6 centimeters by 2 centimeters) that can move at up to 3.5 centimeters per second, support vision, and operate for 63 to 260 minutes. Our results demonstrate that steerable vision can enable object tracking and wide-angle views for 26 to 84 times lower energy than moving the whole robot.

The Critical Role of Head Movements for Spatial Representation During Bumblebees Learning Flight

Bumblebees perform complex flight maneuvers around the barely visible entrance of their nest upon their first departures. During these flights bees learn visual information about the surroundings, possibly including its spatial layout. They rely on this information to return home. Depth information can be derived from the apparent motion of the scenery on the bees' retina. This motion is shaped by the animal's flight and orientation: Bees employ a saccadic flight and gaze strategy, where rapid turns of the head (saccades) alternate with flight segments of apparently constant gaze direction (intersaccades). When during intersaccades the gaze direction is kept relatively constant, the apparent motion contains information about the distance of the animal to environmental objects, and thus, in an egocentric reference frame. Alternatively, when the gaze direction rotates around a fixed point in space, the animal perceives the depth structure relative to this pivot point, i.e., in an allocentric reference frame. If the pivot point is at the nest-hole, the information is nest-centric. Here, we investigate in which reference frames bumblebees perceive depth information during their learning flights. By precisely tracking the head orientation, we found that half of the time, the head appears to pivot actively. However, only few of the corresponding pivot points are close to the nest entrance. Our results indicate that bumblebees perceive visual information in several reference frames when they learn about the surroundings of a behaviorally relevant location.

Timing precision in fly flight control: integrating mechanosensory input with muscle physiology

Animals rapidly collect and act on incoming information to navigate complex environments, making the precise timing of sensory feedback critical in the context of neural circuit function. Moreover, the timing of sensory input determines the biomechanical properties of muscles that undergo cyclic length changes, as during locomotion. Both of these issues come to a head in the case of flying insects, as these animals execute steering manoeuvres at timescales approaching the upper limits of performance for neuromechanical systems. Among insects, flies stand out as especially adept given their ability to execute manoeuvres that require sub-millisecond control of steering muscles. Although vision is critical, here I review the role of rapid, wingbeat-synchronous mechanosensory feedback from the wings and structures unique to flies, the halteres. The visual system and descending interneurons of the brain employ a spike rate coding scheme to relay commands to the wing steering system. By contrast, mechanosensory feedback operates at faster timescales and in the language of motor neurons, i.e. spike timing, allowing wing and haltere input to dynamically structure the output of the wing steering system. Although the halteres have been long known to provide essential input to the wing steering system as gyroscopic sensors, recent evidence suggests that the feedback from these vestigial hindwings is under active control. Thus, flies may accomplish manoeuvres through a conserved hindwing circuit, regulating the firing phase-and thus, the mechanical power output-of the wing steering muscles.

Hybrid visual control in fly flight: insights into gaze shift via saccades

Flies fly by alternating between periods of fixation and body saccades, analogous to how our own eyes move. Gaze fixation via smooth movement in fly flight has been studied extensively, but comparatively less is known about the mechanism by which flies trigger and control body saccades to shift their gaze. Why do flies implement a hybrid fixate-and-saccade locomotion strategy? Here we review recent developments that provide new insights into this question. We focus on the interplay between smooth movement and saccades, the trigger classes of saccades, and the timeline of saccade execution. We emphasize recent mechanistic advances in Drosophila, where genetic tools have enabled cellular circuit analysis at an unprecedented level in a flying insect. In addition, we review trade-offs in behavioral paradigms used to study saccades. Throughout we highlight exciting avenues for future research in the control of fly flight.

Two pursuit strategies for a single sensorimotor control task in blowfly

Effective visuomotor coordination is a necessary requirement for the survival of many terrestrial, aquatic, and aerial animal species. We studied the kinematics of aerial pursuit in the blowfly Lucilia sericata using an actuated dummy as target for freely flying males. We found that the flies perform target tracking in the horizontal plane and target interception in the vertical plane. Our behavioural data suggest that the flies’ trajectory changes are a controlled combination of target heading angle and of the rate of change of the bearing angle. We implemented control laws in kinematic models and found that the contributions of proportional navigation strategy are negligible. We concluded that the difference between horizontal and vertical control relates to the difference in target heading angle the fly keeps constant: 0° in azimuth and 23° in elevation. Our work suggests that male Lucilia control both horizontal and vertical steerings by employing proportional controllers to the error angles. In horizontal plane, this controller operates at time delays as small as 10 ms, the fastest steering response observed in any flying animal, so far.

Active vision shapes and coordinates flight motor responses in flies

Animals use active sensing to respond to sensory inputs and guide future motor decisions. In flight, flies generate a pattern of head and body movements to stabilize gaze. How the brain relays visual information to control head and body movements and how active head movements influence downstream motor control remains elusive. Using a control theoretic framework, we studied the optomotor gaze stabilization reflex in tethered flight and quantified how head movements stabilize visual motion and shape wing steering efforts in fruit flies (Drosophila). By shaping visual inputs, head movements increased the gain of wing steering responses and coordination between stimulus and wings, pointing to a tight coupling between head and wing movements. Head movements followed the visual stimulus in as little as 10 ms-a delay similar to the human vestibulo-ocular reflex-whereas wing steering responses lagged by more than 40 ms. This timing difference suggests a temporal order in the flow of visual information such that the head filters visual information eliciting downstream wing steering responses. Head fixation significantly decreased the mechanical power generated by the flight motor by reducing wingbeat frequency and overall thrust. By simulating an elementary motion detector array, we show that head movements shift the effective visual input dynamic range onto the sensitivity optimum of the motion vision pathway. Taken together, our results reveal a transformative influence of active vision on flight motor responses in flies. Our work provides a framework for understanding how to coordinate moving sensors on a moving body.

How do hoverflies use their righting reflex?

When taking off from a sloping surface, flies have to reorient themselves dorsoventrally and stabilize their body by actively controlling their flapping wings. We have observed that righting is achieved solely by performing a rolling manoeuvre. How flies manage to do this has not yet been elucidated. It was observed here for the first time that hoverfly reorientation is entirely achieved within 6 wingbeats (48.8 ms) at angular roll velocities of up to 10×103 deg s-1 and that the onset of their head rotation consistently follows that of their body rotation after a time lag of 16 ms. The insects' body roll was found to be triggered by the asymmetric wing stroke amplitude, as expected. The righting process starts immediately with the first wingbeat and seems unlikely to depend on visual feedback. A dynamic model for the fly's righting reflex is presented, which accounts for the head/body movements and the time lag recorded in these experiments. This model consists of a closed-loop control of the body roll, combined with a feedforward control of the head/body angle. During the righting manoeuvre, a strong coupling seems to exist between the activation of the halteres (which measure the body's angular speed) and the gaze stabilization reflex. These findings again confirm the fundamental role played by the halteres in both body and head stabilization processes.

Dog eye movements are slower than human eye movements

Eye movement of a species reflects the visual behavior strategy that it has adapted to during its evolution. What are eye movements of domestic dogs (Canis lupus familiaris) like? Investigations of dog eye movements per se have not been done, despite the increasing number of visuo-cognitive studies in dogs using eye-tracking systems. To fill this gap, we have recorded dog eye movements using a video-based eye-tracking system, and compared the dog data to that of humans. We found dog saccades follow the systematic relationships between saccade metrics previously shown in humans and other animal species. Yet, the details of the relationships, and the quantities of each metric of dog saccades and fixations differed from those of humans. Overall, dog saccades were slower and fixations were longer than those of humans. We hope our findings contribute to existing comparative analyses of eye movement across animal species, and also to improvement of algorithms used for classifying eye movement data of dogs.

Inertial Sensing and Encoding of Self-Motion: Structural and Functional Similarities across Metazoan Taxa

To properly orient and navigate, moving animals must obtain information about the position and motion of their bodies. Animals detect inertial signals resulting from body accelerations and rotations using a variety of sensory systems. In this review, we briefly summarize current knowledge on inertial sensing across widely disparate animal taxa with an emphasis on neuronal coding and sensory transduction. We outline systems built around mechanosensory hair cells, including the chordate vestibular complex and the statocysts seen in many marine invertebrates. We next compare these to schemes employed by flying insects for managing inherently unstable aspects of flapping flight, built around comparable mechanosensory cells but taking unique advantage of the physics of rotating systems to facilitate motion encoding. Finally, we highlight fundamental similarities across taxa with respect to the partnering of inertial senses with visual senses and conclude with a discussion of the functional utility of maintaining a multiplicity of encoding schemes for self-motion information.

A novel setup for 3D chasing behavior analysis in free flying flies

BACKGROUND

Insects catching prey or mates on the wing perform one of the fastest behaviours observed in nature. Some dipteran flies are aerial acrobats specialized to detect, chase and capture their targets within the blink of an eye. Studies of aerial pursuits and its underlying sensorimotor control have been a long-standing subject of interest in neuroethology research.

NEW METHOD

We designed an actuated dummy target to trigger chasing flights in male blowflies. Our setup generates arbitrary 2D target trajectories in the horizontal plane combining translation up to 1 m/s and angular rotation up to 720°/s.

RESULTS

Using stereovision methods we reconstructed target and pursuer positions every 5 ms with a maximum 3D error of 5 mm. The pursuer's body pitch and yaw angles were resolved within an error range of 6deg. An embedded observation point provides a close-up view of the pursuer's final approach and enables us to measure its body roll angle. We observed banked turns and sideslip which have not been reported for chasing blowflies in the past.

COMPARISON WITH EXISTING METHOD(S)

Previous studies focused on pursuit along circular paths or interception of translating targets while our method allows us to generate more complex target trajectories. Measurements of body orientation in earlier accounts were limited to the heading direction while we extended the analysis to include the full body orientation during pursuit.

CONCLUSIONS

Our setup offers an opportunity to investigate kinematics and governing visual parameters of chasing behaviour in species up to the size of blowflies under a large variety of experimental conditions.

Insect and insect-inspired aerodynamics: unsteadiness, structural mechanics and flight control

Flying insects impress by their versatility and have been a recurrent source of inspiration for engineering devices. A large body of literature has focused on various aspects of insect flight, with an essential part dedicated to the dynamics of flapping wings and their intrinsically unsteady aerodynamic mechanisms. Insect wings flex during flight and a better understanding of structural mechanics and aeroelasticity is emerging. Most recently, insights from solid and fluid mechanics have been integrated with physiological measurements from visual and mechanosensors in the context of flight control in steady airs and through turbulent conditions. We review the key recent advances concerning flight in unsteady environments and how the multi-body mechanics of the insect structure-wings and body-are at the core of the flight control question. The issues herein should be considered when applying bio-informed design principles to robotic flapping wings.

Head movements quadruple the range of speeds encoded by the insect motion vision system in hawkmoths

Flying insects use compensatory head movements to stabilize gaze. Like other optokinetic responses, these movements can reduce image displacement, motion and misalignment, and simplify the optic flow field. Because gaze is imperfectly stabilized in insects, we hypothesized that compensatory head movements serve to extend the range of velocities of self-motion that the visual system encodes. We tested this by measuring head movements in hawkmoths Hyles lineata responding to full-field visual stimuli of differing oscillation amplitudes, oscillation frequencies and spatial frequencies. We used frequency-domain system identification techniques to characterize the head's roll response, and simulated how this would have affected the output of the motion vision system, modelled as a computational array of Reichardt detectors. The moths' head movements were modulated to allow encoding of both fast and slow self-motion, effectively quadrupling the working range of the visual system for flight control. By using its own output to drive compensatory head movements, the motion vision system thereby works as an adaptive sensor, which will be especially beneficial in nocturnal species with inherently slow vision. Studies of the ecology of motion vision must therefore consider the tuning of motion-sensitive interneurons in the context of the closed-loop systems in which they function.