Feed-forward and visual feedback control of head roll orientation in wasps (Polistes humilis, Vespidae, Hymenoptera)

Integration of visual and antennal mechanosensory feedback during head stabilization in hawkmoths

During flight maneuvers, insects exhibit compensatory head movements which are essential for stabilizing the visual field on their retina, reducing motion blur, and supporting visual self-motion estimation. In Diptera, such head movements are mediated via visual feedback from their compound eyes that detect retinal slip, as well as rapid mechanosensory feedback from their halteres - the modified hindwings that sense the angular rates of body rotations. Because non-Dipteran insects lack halteres, it is not known if mechanosensory feedback about body rotations plays any role in their head stabilization response. Diverse non-Dipteran insects are known to rely on visual and antennal mechanosensory feedback for flight control. In hawkmoths, for instance, reduction of antennal mechanosensory feedback severely compromises their ability to control flight. Similarly, when the head movements of freely flying moths are restricted, their flight ability is also severely impaired. The role of compensatory head movements as well as multimodal feedback in insect flight raises an interesting question: in insects that lack halteres, what sensory cues are required for head stabilization? Here, we show that in the nocturnal hawkmoth Daphnis nerii, compensatory head movements are mediated by combined visual and antennal mechanosensory feedback. We subjected tethered moths to open-loop body roll rotations under different lighting conditions, and measured their ability to maintain head angle in the presence or absence of antennal mechanosensory feedback. Our study suggests that head stabilization in moths is mediated primarily by visual feedback during roll movements at lower frequencies, whereas antennal mechanosensory feedback is required when roll occurs at higher frequency. These findings are consistent with the hypothesis that control of head angle results from a multimodal feedback loop that integrates both visual and antennal mechanosensory feedback, albeit at different latencies. At adequate light levels, visual feedback is sufficient for head stabilization primarily at low frequencies of body roll. However, under dark conditions, antennal mechanosensory feedback is essential for the control of head movements at high frequencies of body roll.

Bumblebees display characteristics of active vision during robust obstacle avoidance flight

Insects are remarkable flyers and capable of navigating through highly cluttered environments. We tracked the head and thorax of bumblebees freely flying in a tunnel containing vertically oriented obstacles to uncover the sensorimotor strategies used for obstacle detection and collision avoidance. Bumblebees presented all the characteristics of active vision during flight by stabilizing their head relative to the external environment and maintained close alignment between their gaze and flightpath. Head stabilization increased motion contrast of nearby features against the background to enable obstacle detection. As bees approached obstacles, they appeared to modulate avoidance responses based on the relative retinal expansion velocity (RREV) of obstacles and their maximum evasion acceleration was linearly related to RREVmax. Finally, bees prevented collisions through rapid roll manoeuvres implemented by their thorax. Overall, the combination of visuo-motor strategies of bumblebees highlights elegant solutions developed by insects for visually guided flight through cluttered environments.

The Critical Role of Head Movements for Spatial Representation During Bumblebees Learning Flight

Bumblebees perform complex flight maneuvers around the barely visible entrance of their nest upon their first departures. During these flights bees learn visual information about the surroundings, possibly including its spatial layout. They rely on this information to return home. Depth information can be derived from the apparent motion of the scenery on the bees' retina. This motion is shaped by the animal's flight and orientation: Bees employ a saccadic flight and gaze strategy, where rapid turns of the head (saccades) alternate with flight segments of apparently constant gaze direction (intersaccades). When during intersaccades the gaze direction is kept relatively constant, the apparent motion contains information about the distance of the animal to environmental objects, and thus, in an egocentric reference frame. Alternatively, when the gaze direction rotates around a fixed point in space, the animal perceives the depth structure relative to this pivot point, i.e., in an allocentric reference frame. If the pivot point is at the nest-hole, the information is nest-centric. Here, we investigate in which reference frames bumblebees perceive depth information during their learning flights. By precisely tracking the head orientation, we found that half of the time, the head appears to pivot actively. However, only few of the corresponding pivot points are close to the nest entrance. Our results indicate that bumblebees perceive visual information in several reference frames when they learn about the surroundings of a behaviorally relevant location.

Analysing Head-Thorax Choreography During Free-Flights in Bumblebees

Animals coordinate their various body parts, sometimes in elaborate manners to swim, walk, climb, fly, and navigate their environment. The coordination of body parts is essential to behaviors such as, chasing, escaping, landing, and the extraction of relevant information. For example, by shaping the movement of the head and body in an active and controlled manner, flying insects structure their flights to facilitate the acquisition of distance information. They condense their turns into a short period of time (the saccade) interspaced by a relatively long translation (the intersaccade). However, due to technological limitations, the precise coordination of the head and thorax during insects' free-flight remains unclear. Here, we propose methods to analyse the orientation of the head and thorax of bumblebees Bombus terrestris, to segregate the trajectories of flying insects into saccades and intersaccades by using supervised machine learning (ML) techniques, and finally to analyse the coordination between head and thorax by using artificial neural networks (ANN). The segregation of flights into saccades and intersaccades by ML, based on the thorax angular velocities, decreased the misclassification by 12% compared to classically used methods. Our results demonstrate how machine learning techniques can be used to improve the analyses of insect flight structures and to learn about the complexity of head-body coordination. We anticipate our assay to be a starting point for more sophisticated experiments and analysis on freely flying insects. For example, the coordination of head and body movements during collision avoidance, chasing behavior, or negotiation of gaps could be investigated by monitoring the head and thorax orientation of freely flying insects within and across behavioral tasks, and in different species.

Active vision shapes and coordinates flight motor responses in flies

Animals use active sensing to respond to sensory inputs and guide future motor decisions. In flight, flies generate a pattern of head and body movements to stabilize gaze. How the brain relays visual information to control head and body movements and how active head movements influence downstream motor control remains elusive. Using a control theoretic framework, we studied the optomotor gaze stabilization reflex in tethered flight and quantified how head movements stabilize visual motion and shape wing steering efforts in fruit flies (Drosophila). By shaping visual inputs, head movements increased the gain of wing steering responses and coordination between stimulus and wings, pointing to a tight coupling between head and wing movements. Head movements followed the visual stimulus in as little as 10 ms-a delay similar to the human vestibulo-ocular reflex-whereas wing steering responses lagged by more than 40 ms. This timing difference suggests a temporal order in the flow of visual information such that the head filters visual information eliciting downstream wing steering responses. Head fixation significantly decreased the mechanical power generated by the flight motor by reducing wingbeat frequency and overall thrust. By simulating an elementary motion detector array, we show that head movements shift the effective visual input dynamic range onto the sensitivity optimum of the motion vision pathway. Taken together, our results reveal a transformative influence of active vision on flight motor responses in flies. Our work provides a framework for understanding how to coordinate moving sensors on a moving body.

How do hoverflies use their righting reflex?

When taking off from a sloping surface, flies have to reorient themselves dorsoventrally and stabilize their body by actively controlling their flapping wings. We have observed that righting is achieved solely by performing a rolling manoeuvre. How flies manage to do this has not yet been elucidated. It was observed here for the first time that hoverfly reorientation is entirely achieved within 6 wingbeats (48.8 ms) at angular roll velocities of up to 10×103 deg s-1 and that the onset of their head rotation consistently follows that of their body rotation after a time lag of 16 ms. The insects' body roll was found to be triggered by the asymmetric wing stroke amplitude, as expected. The righting process starts immediately with the first wingbeat and seems unlikely to depend on visual feedback. A dynamic model for the fly's righting reflex is presented, which accounts for the head/body movements and the time lag recorded in these experiments. This model consists of a closed-loop control of the body roll, combined with a feedforward control of the head/body angle. During the righting manoeuvre, a strong coupling seems to exist between the activation of the halteres (which measure the body's angular speed) and the gaze stabilization reflex. These findings again confirm the fundamental role played by the halteres in both body and head stabilization processes.

The roles of vision and antennal mechanoreception in hawkmoth flight control

Flying animals need continual sensory feedback about their body position and orientation for flight control. The visual system provides essential but slow feedback. In contrast, mechanosensory channels can provide feedback at much shorter timescales. How the contributions from these two senses are integrated remains an open question in most insect groups. In Diptera, fast mechanosensory feedback is provided by organs called halteres and is crucial for the control of rapid flight manoeuvres, while vision controls manoeuvres in lower temporal frequency bands. Here, we have investigated the visual-mechanosensory integration in the hawkmoth Macroglossum stellatarum. They represent a large group of insects that use Johnston’s organs in their antennae to provide mechanosensory feedback on perturbations in body position. Our experiments show that antennal mechanosensory feedback specifically mediates fast flight manoeuvres, but not slow ones. Moreover, we did not observe compensatory interactions between antennal and visual feedback.

Vision-only egomotion estimation in 6DOF using a sky compass

A novel pure-vision egomotion estimation algorithm is presented, with extensions to Unmanned Aerial Systems (UAS) navigation through visual odometry. Our proposed method computes egomotion in two stages using panoramic images segmented into sky and ground regions. Rotations (in 3DOF) are estimated by using a customised algorithm to measure the motion of the sky image, which is affected only by the rotation of the aircraft, and not by its translation. The rotation estimate is then used to derotate the optic flow field generated by the ground, from which the translation of the aircraft (in 3DOF) is estimated by another customised, iterative algorithm. Segmentation of the rotation and translation estimations allows for a partial relaxation of the planar ground assumption, inherently increasing the robustness of the approach. The translation vectors are scaled using stereo-based height to compute the current UAS position through path integration for closed-loop navigation. Outdoor field tests of our approach in a small quadrotor UAS suggest that the technique is comparable to the performance of existing state-of-the-art vision-based navigation algorithms, whilst also removing all dependence on additional sensors, such as an IMU or global positioning system (GPS).

Complex gaze stabilization in mantis shrimp

Almost all animals, regardless of the anatomy of the eyes, require some level of gaze stabilization in order to see the world clearly and without blur. For the mantis shrimp, achieving gaze stabilization is unusually challenging as their eyes have an unprecedented scope for movement in all three rotational degrees of freedom: yaw, pitch and torsion. We demonstrate that the species Odontodactylus scyllarus performs stereotypical gaze stabilization in the yaw degree of rotational freedom, which is accompanied by simultaneous changes in the pitch and torsion rotation of the eye. Surprisingly, yaw gaze stabilization performance is unaffected by both the torsional pose and the rate of torsional rotation of the eye. Further to this, we show, for the first time, a lack of a torsional gaze stabilization response in the stomatopod visual system. In the light of these findings, we suggest that the neural wide-field motion detection network in the stomatopod visual system may follow a radially symmetric organization to compensate for the potentially disorientating effects of torsional eye movements, a system likely to be unique to stomatopods.

3-D Maps and Compasses in the Brain

The world has a complex, three-dimensional (3-D) spatial structure, but until recently the neural representation of space was studied primarily in planar horizontal environments. Here we review the emerging literature on allocentric spatial representations in 3-D and discuss the relations between 3-D spatial perception and the underlying neural codes. We suggest that the statistics of movements through space determine the topology and the dimensionality of the neural representation, across species and different behavioral modes. We argue that hippocampal place-cell maps are metric in all three dimensions, and might be composed of 2-D and 3-D fragments that are stitched together into a global 3-D metric representation via the 3-D head-direction cells. Finally, we propose that the hippocampal formation might implement a neural analogue of a Kalman filter, a standard engineering algorithm used for 3-D navigation.

Evolution of Biological Image Stabilization

The use of vision to coordinate behavior requires an efficient control design that stabilizes the world on the retina or directs the gaze towards salient features in the surroundings. With a level gaze, visual processing tasks are simplified and behaviorally relevant features from the visual environment can be extracted. No matter how simple or sophisticated the eye design, mechanisms have evolved across phyla to stabilize gaze. In this review, we describe functional similarities in eyes and gaze stabilization reflexes, emphasizing their fundamental role in transforming sensory information into motor commands that support postural and locomotor control. We then focus on gaze stabilization design in flying insects and detail some of the underlying principles. Systems analysis reveals that gaze stabilization often involves several sensory modalities, including vision itself, and makes use of feedback as well as feedforward signals. Independent of phylogenetic distance, the physical interaction between an animal and its natural environment - its available senses and how it moves - appears to shape the adaptation of all aspects of gaze stabilization.

Head movements quadruple the range of speeds encoded by the insect motion vision system in hawkmoths

Flying insects use compensatory head movements to stabilize gaze. Like other optokinetic responses, these movements can reduce image displacement, motion and misalignment, and simplify the optic flow field. Because gaze is imperfectly stabilized in insects, we hypothesized that compensatory head movements serve to extend the range of velocities of self-motion that the visual system encodes. We tested this by measuring head movements in hawkmoths Hyles lineata responding to full-field visual stimuli of differing oscillation amplitudes, oscillation frequencies and spatial frequencies. We used frequency-domain system identification techniques to characterize the head's roll response, and simulated how this would have affected the output of the motion vision system, modelled as a computational array of Reichardt detectors. The moths' head movements were modulated to allow encoding of both fast and slow self-motion, effectively quadrupling the working range of the visual system for flight control. By using its own output to drive compensatory head movements, the motion vision system thereby works as an adaptive sensor, which will be especially beneficial in nocturnal species with inherently slow vision. Studies of the ecology of motion vision must therefore consider the tuning of motion-sensitive interneurons in the context of the closed-loop systems in which they function.

Head orientation of walking blowflies is controlled by visual and mechanical cues

During locomotion animals employ visual and mechanical cues in order to establish the orientation of their head, which reflects the orientation of the visual coordinate system. However, in certain situations, contradictory cues may suggest different orientations relative to the environment. We recorded blowflies walking on a horizontal or tilted surface surrounded by visual cues suggesting a variety of orientations. We found that the different orientations relative to gravity of visual cues and walking surface were integrated, with the orientation of the surface being the major contributor to head orientation, while visual cues and gravity also play an important role. In contrast, visual cues did not affect body orientation much. Cue integration was modeled as the weighted sum of orientations suggested by the different cues. Our model suggests that in case of lacking visual cues more weight is given to gravity.

Controlled banked turns in coleopteran flight measured by a miniature wireless inertial measurement unit

The mechanisms and principles of insect flight have long been investigated by researchers working on micro and nano air vehicles (MAVs/NAVs). However, studies of insect flight maneuvers require high speed filming and high spatial resolution in a small experimental space, or the tethering of the insect to a fixed place. Under such artificial conditions, the insects may deviate its flying behavior from that of regular flight. In this study, we mounted a tiny wireless system, or 'backpack', on live beetles (Mecynorrhina torquata; length 62 ± 8 mm; mass 7.4 ± 1.3 g) freely flying in a large laboratory space. The backpack contains a micro inertial measurement unit (IMU) that was especially designed and manufactured for this purpose. Owing to the small mass (∼1.30 g) and dimensions (∼2.3 cm²) of the backpack and the high accuracy of the IMU, we could remotely record the beetle in free flight. The free flight data revealed a strong linear correlation between the roll angle and yaw angular velocity. The strength of the correlation was quantified by the correlation coefficients and mean values. The change in roll angle preceded the change in yaw angular velocity. Moreover, there were frequent fluctuations in the roll angular velocity, which were uncorrelated with the yaw angular velocity. Apart from the strong correlation, these findings imply that Mecynorrhina torquata actively manipulates its roll rotation without coupling to the yaw rotation.

To crash or not to crash: how do hoverflies cope with free-fall situations and weightlessness?

Insects’ aptitude to perform hovering, automatic landing and tracking tasks involves accurately controlling their head and body roll and pitch movements, but how this attitude control depends on an internal estimation of gravity orientation is still an open question. Gravity perception in flying insects has mainly been studied in terms of grounded animals' tactile orientation responses, but it has not yet been established whether hoverflies use gravity perception cues to detect a nearly weightless state at an early stage. Ground-based microgravity simulators provide biologists with useful tools for studying the effects of changes in gravity. However, in view of the cost and the complexity of these set-ups, an alternative Earth-based free-fall procedure was developed with which flying insects can be briefly exposed to microgravity under various visual conditions. Hoverflies frequently initiated wingbeats in response to an imposed free fall in all the conditions tested, but managed to avoid crashing only in variably structured visual environments, and only episodically in darkness. Our results reveal that the crash-avoidance performance of these insects in various visual environments suggests the existence of a multisensory control system based mainly on vision rather than gravity perception.

Head roll stabilisation in the nocturnal bull ant Myrmecia pyriformis: implications for visual navigation

Ant foragers are known to memorise visual scenes that allow them to repeatedly travel along idiosyncratic routes and to return to specific places. Guidance is provided by a comparison between visual memories and current views, which critically depends on how well the attitude of the visual system is controlled. Here we show that nocturnal bull ants stabilise their head to varying degrees against locomotion-induced body roll movements, and this ability decreases as light levels fall. There are always un-compensated head roll oscillations that match the frequency of the stride cycle. Head roll stabilisation involves both visual and non-visual cues as ants compensate for body roll in complete darkness and also respond with head roll movements when confronted with visual pattern oscillations. We show that imperfect head roll control degrades navigation-relevant visual information and discuss ways in which navigating ants may deal with this problem.

Behavioural evidence for a visual and proprioceptive control of head roll in hoverflies (Episyrphus balteatus)

The ability of hoverflies to control their head orientation with respect to their body contributes importantly to their agility and their autonomous navigation abilities. Many tasks performed by this insect during flight, especially while hovering, involve a head stabilization reflex. This reflex, which is mediated by multisensory channels, prevents the visual processing from being disturbed by motion blur and maintains a consistent perception of the visual environment. The so-called dorsal light response (DLR) is another head control reflex, which makes insects sensitive to the brightest part of the visual field. In this study, we experimentally validate and quantify the control loop driving the head roll with respect to the horizon in hoverflies. The new approach developed here consisted of using an upside-down horizon in a body roll paradigm. In this unusual configuration, tethered flying hoverflies surprisingly no longer use purely vision-based control for head stabilization. These results shed new light on the role of neck proprioceptor organs in head and body stabilization with respect to the horizon. Based on the responses obtained with male and female hoverflies, an improved model was then developed in which the output signals delivered by the neck proprioceptor organs are combined with the visual error in the estimated position of the body roll. An internal estimation of the body roll angle with respect to the horizon might explain the extremely accurate flight performances achieved by some hovering insects.

Wing-pitch modulation in maneuvering fruit flies is explained by an interplay between aerodynamics and a torsional spring

While the wing kinematics of many flapping insects have been well characterized, understanding the underlying sensory, neural, and physiological mechanisms that determine these kinematics is still a challenge. Two main difficulties in understanding the physiological mechanisms arise from the complexity of the interaction between a flapping wing and its own unsteady flow, as well as the intricate mechanics of the insect wing hinge, which is among the most complicated joints in the animal kingdom. These difficulties call for the application of reduced-order approaches. Here this strategy is used to model the torques exerted by the wing hinge along the wing-pitch axis of maneuvering fruit flies as a damped torsional spring with elastic and damping coefficients as well as a rest angle. Furthermore, we model the air flows using simplified quasistatic aerodynamics. Our findings suggest that flies take advantage of the passive coupling between aerodynamics and the damped torsional spring to indirectly control their wing-pitch kinematics by modulating the spring parameters. The damped torsional-spring model explains the changes measured in wing-pitch kinematics during roll correction maneuvers through modulation of the spring damping and elastic coefficients. These results, in conjunction with the previous literature, indicate that flies can accurately control their wing-pitch kinematics on a sub-wing-beat time scale by modulating all three effective spring parameters on longer time scales.

Controlling roll perturbations in fruit flies

Owing to aerodynamic instabilities, stable flapping flight requires ever-present fast corrective actions. Here, we investigate how flies control perturbations along their body roll angle, which is unstable and their most sensitive degree of freedom. We glue a magnet to each fly and apply a short magnetic pulse that rolls it in mid-air. Fast video shows flies correct perturbations up to 100° within 30 ± 7 ms by applying a stroke-amplitude asymmetry that is well described by a linear proportional-integral controller. For more aggressive perturbations, we show evidence for nonlinear and hierarchical control mechanisms. Flies respond to roll perturbations within 5 ms, making this correction reflex one of the fastest in the animal kingdom.

Flying over uneven moving terrain based on optic-flow cues without any need for reference frames or accelerometers

Two bio-inspired guidance principles involving no reference frame are presented here and were implemented in a rotorcraft, which was equipped with panoramic optic flow (OF) sensors but (as in flying insects) no accelerometer. To test these two guidance principles, we built a tethered tandem rotorcraft called BeeRotor (80 grams), which was tested flying along a high-roofed tunnel. The aerial robot adjusts its pitch and hence its speed, hugs the ground and lands safely without any need for an inertial reference frame. The rotorcraft's altitude and forward speed are adjusted via two OF regulators piloting the lift and the pitch angle on the basis of the common-mode and differential rotor speeds, respectively. The robot equipped with two wide-field OF sensors was tested in order to assess the performances of the following two systems of guidance involving no inertial reference frame: (i) a system with a fixed eye orientation based on the curved artificial compound eye (CurvACE) sensor, and (ii) an active system of reorientation based on a quasi-panoramic eye which constantly realigns its gaze, keeping it parallel to the nearest surface followed. Safe automatic terrain following and landing were obtained with CurvACE under dim light to daylight conditions and the active eye-reorientation system over rugged, changing terrain, without any need for an inertial reference frame.

Bio-inspired modeling and implementation of the ocelli visual system of flying insects

Two visual sensing modalities in insects, the ocelli and compound eyes, provide signals used for flight stabilization and navigation. In this article, a generalized model of the ocellar visual system is developed for a 3-D visual simulation environment based on behavioral, anatomical, and electrophysiological data from several species. A linear measurement model is estimated from Monte Carlo simulation in a cluttered urban environment relating state changes of the vehicle to the outputs of the ocellar model. A fully analog-printed circuit board sensor based on this model is designed and fabricated. Open-loop characterization of the sensor to visual stimuli induced by self motion is performed. Closed-loop stabilizing feedback of the sensor in combination with optic flow sensors is implemented onboard a quadrotor micro-air vehicle and its impulse response is characterized.

Control of moth flight posture is mediated by wing mechanosensory feedback

Flying insects rapidly stabilize after perturbations using both visual and mechanosensory inputs for active control. Insect halteres are mechanosensory organs that encode inertial forces to aid rapid course correction during flight but serve no aerodynamic role and are specific to two orders of insects (Diptera and Strepsiptera). Aside from the literature on halteres and recent work on the antennae of the hawkmoth Manduca sexta, it is unclear how other flying insects use mechanosensory information to control body dynamics. The mechanosensory structures found on the halteres, campaniform sensilla, are also present on wings, suggesting that the wings can encode information about flight dynamics. We show that the neurons innervating these sensilla on the forewings of M. sexta exhibit spike-timing precision comparable to that seen in previous reports of campaniform sensilla, including haltere neurons. In addition, by attaching magnets to the wings of moths and subjecting these animals to a simulated pitch stimulus via a rotating magnetic field during tethered flight, we elicited the same vertical abdominal flexion reflex these animals exhibit in response to visual or inertial pitch stimuli. Our results indicate that, in addition to their role as actuators during locomotion, insect wings serve as sensors that initiate reflexes that control body dynamics.

Figure-ground discrimination behavior in Drosophila. II. Visual influences on head movement behavior

Visual identification of small moving targets is a challenge for all moving animals. Their own motion generates displacement of the visual surroundings, inducing wide-field optic flow across the retina. Wide-field optic flow is used to sense perturbations in the flight course. Both ego-motion and corrective optomotor responses confound any attempt to track a salient target moving independently of the visual surroundings. What are the strategies that flying animals use to discriminate small-field figure motion from superimposed wide-field background motion? We examined how fruit flies adjust their gaze in response to a compound visual stimulus comprising a small moving figure against an independently moving wide-field ground, which they do by re-orienting their head or their flight trajectory. We found that fixing the head in place impairs object fixation in the presence of ground motion, and that head movements are necessary for stabilizing wing steering responses to wide-field ground motion when a figure is present. When a figure is moving relative to a moving ground, wing steering responses follow components of both the figure and ground trajectories, but head movements follow only the ground motion. To our knowledge, this is the first demonstration that wing responses can be uncoupled from head responses and that the two follow distinct trajectories in the case of simultaneous figure and ground motion. These results suggest that whereas figure tracking by wing kinematics is independent of head movements, head movements are important for stabilizing ground motion during active figure tracking.