Shallow neural networks trained to detect collisions recover features of visual loom-selective neurons

Fly navigational responses to odor motion and gradient cues are tuned to plume statistics

Odor cues guide animals to food and mates. Different environmental conditions can create differently patterned odor plumes, making navigation more challenging. Prior work has shown that animals turn upwind when they detect odor and cast crosswind when they lose it. Animals with bilateral olfactory sensors can also detect directional odor cues, such as odor gradient and odor motion. It remains unknown how animals use these two directional odor cues to guide crosswind navigation in odor plumes with distinct statistics. Here, we investigate this problem theoretically and experimentally. We show that these directional odor cues provide complementary information for navigation in different plume environments. We numerically analyzed real plumes to show that odor gradient cues are more informative about crosswind directions in relatively smooth odor plumes, while odor motion cues are more informative in turbulent or complex plumes. Neural networks trained to optimize crosswind turning converge to distinctive network structures that are tuned to odor gradient cues in smooth plumes and to odor motion cues in complex plumes. These trained networks improve the performance of artificial agents navigating plume environments that match the training environment. By recording Drosophila fruit flies as they navigated different odor plume environments, we verified that flies show the same correspondence between informative cues and plume types. Fly turning in the crosswind direction is correlated with odor gradients in smooth plumes and with odor motion in complex plumes. Overall, these results demonstrate that these directional odor cues are complementary across environments, and that animals exploit this relationship.

Significance

Many animals use smell to find food and mates, often navigating complex odor plumes shaped by environmental conditions. While upwind movement upon odor detection is well established, less is known about how animals steer crosswind to stay in the plume. We show that directional odor cues-gradients and motion-guide crosswind navigation differently depending on plume structure. Gradients carry more information in smooth plumes, while motion dominates in turbulent ones. Neural network trained to optimize crosswind navigation reflect this distinction, developing gradient sensitivity in smooth environments and motion sensitivity in complex ones. Experimentally, fruit flies adjust their turning behavior to prioritize the most informative cue in each context. These findings likely generalize to other animals navigating similarly structured odor plumes.

A computationally efficient and robust looming perception model based on dynamic neural field

There are primarily two classes of bio-inspired looming perception visual systems. The first class employs hierarchical neural networks inspired by well-acknowledged anatomical pathways responsible for looming perception, and the second maps nonlinear relationships between physical stimulus attributes and neuronal activity. However, even with multi-layered structures, the former class is sometimes fragile in looming selectivity, i.e., the ability to well discriminate between approaching and other categories of movements. While the latter class leaves qualms regarding how to encode visual movements to indicate physical attributes like angular velocity/size. Beyond those, we propose a novel looming perception model based on dynamic neural field (DNF). The DNF is a brain-inspired framework that incorporates both lateral excitation and inhibition within the field through instant feedback, it could be an easily-built model to fulfill the looming sensitivity observed in biological visual systems. To achieve our target of looming perception with computational efficiency, we introduce a single-field DNF with adaptive lateral interactions and dynamic activation threshold. The former mechanism creates antagonism to translating motion, and the latter suppresses excitation during receding. Accordingly, the proposed model exhibits the strongest response to moving objects signaling approaching over other types of external stimuli. The effectiveness of the proposed model is supported by relevant mathematical analysis and ablation study. The computational efficiency and robustness of the model are verified through systematic experiments including on-line collision-detection tasks in micro-mobile robots, at success rate of 93% compared with state-of-the-art methods. The results demonstrate its superiority over the model-based methods concerning looming perception.

A Computationally Efficient Neuronal Model for Collision Detection with Contrast Polarity-Specific Feed-Forward Inhibition

Animals utilize their well-evolved dynamic vision systems to perceive and evade collision threats. Driven by biological research, bio-inspired models based on lobula giant movement detectors (LGMDs) address certain gaps in constructing artificial collision-detecting vision systems with robust selectivity, offering reliable, low-cost, and miniaturized collision sensors across various scenes. Recent progress in neuroscience has revealed the energetic advantages of dendritic arrangements presynaptic to the LGMDs, which receive contrast polarity-specific signals on separate dendritic fields. Specifically, feed-forward inhibitory inputs arise from parallel ON/OFF pathways interacting with excitation. However, none of the previous research has investigated the evolution of a computational LGMD model with feed-forward inhibition (FFI) separated by opposite polarity. This study fills this vacancy by presenting an optimized neuronal model where FFI is divided into ON/OFF channels, each with distinct synaptic connections. To align with the energy efficiency of biological systems, we introduce an activation function associated with neural computation of FFI and interactions between local excitation and lateral inhibition within ON/OFF channels, ignoring non-active signal processing. This approach significantly improves the time efficiency of the LGMD model, focusing only on substantial luminance changes in image streams. The proposed neuronal model not only accelerates visual processing in relatively stationary scenes but also maintains robust selectivity to ON/OFF-contrast looming stimuli. Additionally, it can suppress translational motion to a moderate extent. Comparative testing with state-of-the-art based on ON/OFF channels was conducted systematically using a range of visual stimuli, including indoor structured and complex outdoor scenes. The results demonstrated significant time savings in silico while retaining original collision selectivity. Furthermore, the optimized model was implemented in the embedded vision system of a micro-mobile robot, achieving the highest success ratio of collision avoidance at 97.51% while nearly halving the processing time compared with previous models. This highlights a robust and parsimonious collision-sensing mode that effectively addresses real-world challenges.

Visual looming is a primitive for human emotion

The neural computations for looming detection are strikingly similar across species. In mammals, information about approaching threats is conveyed from the retina to the midbrain superior colliculus, where approach variables are computed to enable defensive behavior. Although neuroscientific theories posit that midbrain representations contribute to emotion through connectivity with distributed brain systems, it remains unknown whether a computational system for looming detection can predict both defensive behavior and phenomenal experience in humans. Here, we show that a shallow convolutional neural network based on the Drosophila visual system predicts defensive blinking to looming objects in infants and superior colliculus responses to optical expansion in adults. Further, the neural network's responses to naturalistic video clips predict self-reported emotion largely by way of subjective arousal. These findings illustrate how a simple neural network architecture optimized for a species-general task relevant for survival explains motor and experiential components of human emotion.

Mapping model units to visual neurons reveals population code for social behaviour

The rich variety of behaviours observed in animals arises through the interplay between sensory processing and motor control. To understand these sensorimotor transformations, it is useful to build models that predict not only neural responses to sensory input1-5 but also how each neuron causally contributes to behaviour6,7. Here we demonstrate a novel modelling approach to identify a one-to-one mapping between internal units in a deep neural network and real neurons by predicting the behavioural changes that arise from systematic perturbations of more than a dozen neuronal cell types. A key ingredient that we introduce is 'knockout training', which involves perturbing the network during training to match the perturbations of the real neurons during behavioural experiments. We apply this approach to model the sensorimotor transformations of Drosophila melanogaster males during a complex, visually guided social behaviour8-11. The visual projection neurons at the interface between the optic lobe and central brain form a set of discrete channels12, and prior work indicates that each channel encodes a specific visual feature to drive a particular behaviour13,14. Our model reaches a different conclusion: combinations of visual projection neurons, including those involved in non-social behaviours, drive male interactions with the female, forming a rich population code for behaviour. Overall, our framework consolidates behavioural effects elicited from various neural perturbations into a single, unified model, providing a map from stimulus to neuronal cell type to behaviour, and enabling future incorporation of wiring diagrams of the brain15 into the model.

Visual looming is a primitive for human emotion

Looming objects afford threat of collision across the animal kingdom. Defensive responses to looming and neural computations for looming detection are strikingly conserved across species. In mammals, information about rapidly approaching threats is conveyed from the retina to the midbrain superior colliculus, where variables that indicate the position and velocity of approach are computed to enable defensive behavior. Although neuroscientific theories posit that midbrain representations contribute to emotion through connectivity with distributed brain systems, it remains unknown whether a computational system for looming detection can predict both defensive behavior and phenomenal experience in humans. Here, we show that a shallow convolutional neural network based on the Drosophila visual system predicts defensive blinking to looming objects in infants and superior colliculus responses to optical expansion in adults. Further, the responses of the convolutional network to a broad array of naturalistic video clips predict self-reported emotion largely on the basis of subjective arousal. Our findings illustrate how motor and experiential components of human emotion relate to species-general systems for survival in unpredictable environments.

An Angular Acceleration Based Looming Detector for Moving UAVs

Visual perception equips unmanned aerial vehicles (UAVs) with increasingly comprehensive and instant environmental perception, rendering it a crucial technology in intelligent UAV obstacle avoidance. However, the rapid movements of UAVs cause significant changes in the field of view, affecting the algorithms' ability to extract the visual features of collisions accurately. As a result, algorithms suffer from a high rate of false alarms and a delay in warning time. During the study of visual field angle curves of different orders, it was found that the peak times of the curves of higher-order information on the angular size of looming objects are linearly related to the time to collision (TTC) and occur before collisions. This discovery implies that encoding higher-order information on the angular size could resolve the issue of response lag. Furthermore, the fact that the image of a looming object adjusts to meet several looming visual cues compared to the background interference implies that integrating various field-of-view characteristics will likely enhance the model's resistance to motion interference. Therefore, this paper presents a concise A-LGMD model for detecting looming objects. The model is based on image angular acceleration and addresses problems related to imprecise feature extraction and insufficient time series modeling to enhance the model's ability to rapidly and precisely detect looming objects during the rapid self-motion of UAVs. The model draws inspiration from the lobula giant movement detector (LGMD), which shows high sensitivity to acceleration information. In the proposed model, higher-order information on the angular size is abstracted by the network and fused with multiple visual field angle characteristics to promote the selective response to looming objects. Experiments carried out on synthetic and real-world datasets reveal that the model can efficiently detect the angular acceleration of an image, filter out insignificant background motion, and provide early warnings. These findings indicate that the model could have significant potential in embedded collision detection systems of micro or small UAVs.

Heterogeneous presynaptic receptive fields contribute to directional tuning in starburst amacrine cells

The processing of visual information by retinal starburst amacrine cells (SACs) involves transforming excitatory input from bipolar cells (BCs) into directional calcium output. While previous studies have suggested that an asymmetry in the kinetic properties of BCs along the soma-dendritic axes of the postsynaptic cell could enhance directional tuning at the level of individual branches, it remains unclear whether biologically relevant presynaptic kinetics contribute to direction selectivity (DS) when visual stimulation engages the entire dendritic tree. To address this question, we built multicompartmental models of the bipolar-SAC circuit and trained them to boost directional tuning. We report that despite significant dendritic crosstalk and dissimilar directional preferences along the dendrites that occur during whole-cell stimulation, the rules that guide BC kinetics leading to optimal DS are similar to the single-dendrite condition. To correlate model predictions to empirical findings, we utilized two-photon glutamate imaging to study the dynamics of bipolar release onto ON- and OFF-starburst dendrites in the murine retina. We reveal diverse presynaptic dynamics in response to motion in both BC populations; algorithms trained on the experimental data suggested that the differences in the temporal release kinetics are likely to correspond to heterogeneous receptive field properties among the different BC types, including the spatial extent of the center and surround components. In addition, we demonstrate that circuit architecture composed of presynaptic units with experimentally recorded dynamics could enhance directional drive but not to levels that replicate empirical findings, suggesting other DS mechanisms are required to explain SAC function. Our study provides new insights into the complex mechanisms underlying DS in retinal processing and highlights the potential contribution of presynaptic kinetics to the computation of visual information by SACs.

Heterogeneous presynaptic receptive fields contribute to directional tuning in starburst amacrine cells

The processing of visual information by retinal starburst amacrine cells (SACs) involves transforming excitatory input from bipolar cells (BCs) into directional calcium output. While previous studies have suggested that an asymmetry in the kinetic properties of bipolar cells along the soma-dendritic axes of the postsynaptic cell could enhance directional tuning at the level of individual branches, it remains unclear whether biologically relevant presynaptic kinetics contribute to direction selectivity when visual stimulation engages the entire dendritic tree. To address this question, we built multicompartmental models of the bipolar-SAC circuit and trained them to boost directional tuning. We report that despite significant dendritic crosstalk and dissimilar directional preferences along the dendrites that occur during whole-cell stimulation, the rules that guide BC kinetics leading to optimal directional selectivity are similar to the single-dendrite condition. To correlate model predictions to empirical findings, we utilized two-photon glutamate imaging to study the dynamics of bipolar release onto ON- and OFF-starburst dendrites in the murine retina. We reveal diverse presynaptic dynamics in response to motion in both BC populations; algorithms trained on the experimental data suggested that the differences in the temporal release kinetics are likely to correspond to heterogeneous receptive field (RF) properties among the different BC types, including the spatial extent of the center and surround components. In addition, we demonstrate that circuit architecture composed of presynaptic units with experimentally recorded dynamics could enhance directional drive but not to levels that replicate empirical findings, suggesting other DS mechanisms are required to explain SAC function. Our study provides new insights into the complex mechanisms underlying direction selectivity in retinal processing and highlights the potential contribution of presynaptic kinetics to the computation of visual information by starburst amacrine cells.

Motion perception based on ON/OFF channels: A survey

Motion perception is an essential ability for animals and artificially intelligent systems interacting effectively, safely with surrounding objects and environments. Biological visual systems, that have naturally evolved over hundreds-million years, are quite efficient and robust for motion perception, whereas artificial vision systems are far from such capability. This paper argues that the gap can be significantly reduced by formulation of ON/OFF channels in motion perception models encoding luminance increment (ON) and decrement (OFF) responses within receptive field, separately. Such signal-bifurcating structure has been found in neural systems of many animal species articulating early motion is split and processed in segregated pathways. However, the corresponding biological substrates, and the necessity for artificial vision systems have never been elucidated together, leaving concerns on uniqueness and advantages of ON/OFF channels upon building dynamic vision systems to address real world challenges. This paper highlights the importance of ON/OFF channels in motion perception through surveying current progress covering both neuroscience and computationally modelling works with applications. Compared to related literature, this paper for the first time provides insights into implementation of different selectivity to directional motion of looming, translating, and small-sized target movement based on ON/OFF channels in keeping with soundness and robustness of biological principles. Existing challenges and future trends of such bio-plausible computational structure for visual perception in connection with hotspots of machine learning, advanced vision sensors like event-driven camera finally are discussed.

Visual processing in the fly, from photoreceptors to behavior

Originally a genetic model organism, the experimental use of Drosophila melanogaster has grown to include quantitative behavioral analyses, sophisticated perturbations of neuronal function, and detailed sensory physiology. A highlight of these developments can be seen in the context of vision, where pioneering studies have uncovered fundamental and generalizable principles of sensory processing. Here we begin with an overview of vision-guided behaviors and common methods for probing visual circuits. We then outline the anatomy and physiology of brain regions involved in visual processing, beginning at the sensory periphery and ending with descending motor control. Areas of focus include contrast and motion detection in the optic lobe, circuits for visual feature selectivity, computations in support of spatial navigation, and contextual associative learning. Finally, we look to the future of fly visual neuroscience and discuss promising topics for further study.

Bioinspired figure-ground discrimination via visual motion smoothing

Flies detect and track moving targets among visual clutter, and this process mainly relies on visual motion. Visual motion is analyzed or computed with the pathway from the retina to T4/T5 cells. The computation of local directional motion was formulated as an elementary movement detector (EMD) model more than half a century ago. Solving target detection or figure-ground discrimination problems can be equivalent to extracting boundaries between a target and the background based on the motion discontinuities in the output of a retinotopic array of EMDs. Individual EMDs cannot measure true velocities, however, due to their sensitivity to pattern properties such as luminance contrast and spatial frequency content. It remains unclear how local directional motion signals are further integrated to enable figure-ground discrimination. Here, we present a computational model inspired by fly motion vision. Simulations suggest that the heavily fluctuating output of an EMD array is naturally surmounted by a lobula network, which is hypothesized to be downstream of the local motion detectors and have parallel pathways with distinct directional selectivity. The lobula network carries out a spatiotemporal smoothing operation for visual motion, especially across time, enabling the segmentation of moving figures from the background. The model qualitatively reproduces experimental observations in the visually evoked response characteristics of one type of lobula columnar (LC) cell. The model is further shown to be robust to natural scene variability. Our results suggest that the lobula is involved in local motion-based target detection.

Contrast polarity-specific mapping improves efficiency of neuronal computation for collision detection

Neurons receive information through their synaptic inputs, but the functional significance of how those inputs are mapped on to a cell's dendrites remains unclear. We studied this question in a grasshopper visual neuron that tracks approaching objects and triggers escape behavior before an impending collision. In response to black approaching objects, the neuron receives OFF excitatory inputs that form a retinotopic map of the visual field onto compartmentalized, distal dendrites. Subsequent processing of these OFF inputs by active membrane conductances allows the neuron to discriminate the spatial coherence of such stimuli. In contrast, we show that ON excitatory synaptic inputs activated by white approaching objects map in a random manner onto a more proximal dendritic field of the same neuron. The lack of retinotopic synaptic arrangement results in the neuron's inability to discriminate the coherence of white approaching stimuli. Yet, the neuron retains the ability to discriminate stimulus coherence for checkered stimuli of mixed ON/OFF polarity. The coarser mapping and processing of ON stimuli thus has a minimal impact, while reducing the total energetic cost of the circuit. Further, we show that these differences in ON/OFF neuronal processing are behaviorally relevant, being tightly correlated with the animal's escape behavior to light and dark stimuli of variable coherence. Our results show that the synaptic mapping of excitatory inputs affects the fine stimulus discrimination ability of single neurons and document the resulting functional impact on behavior.