High Accuracy Decoding of Dynamical Motion from a Large Retinal Population

Information transfer and recovery for the sense of touch

Transformation of postsynaptic potentials into action potentials is the rate-limiting step of communication in neural networks. The efficiency of this intracellular information transfer also powerfully shapes stimulus representations in sensory cortices. Using whole-cell recordings and information-theoretic measures, we show herein that somatic postsynaptic potentials accurately represent stimulus location on a trial-by-trial basis in single neurons, even 4 synapses away from the sensory periphery in the whisker system. This information is largely lost during action potential generation but can be rapidly (<20 ms) recovered using complementary information in local populations in a cell-type-specific manner. These results show that as sensory information is transferred from one neural locus to another, the circuits reconstruct the stimulus with high fidelity so that sensory representations of single neurons faithfully represent the stimulus in the periphery, but only in their postsynaptic potentials, resulting in lossless information processing for the sense of touch in the primary somatosensory cortex.

Robust Decoding of Rich Dynamical Visual Scenes With Retinal Spikes

Sensory information transmitted to the brain activates neurons to create a series of coping behaviors. Understanding the mechanisms of neural computation and reverse engineering the brain to build intelligent machines requires establishing a robust relationship between stimuli and neural responses. Neural decoding aims to reconstruct the original stimuli that trigger neural responses. With the recent upsurge of artificial intelligence, neural decoding provides an insightful perspective for designing novel algorithms of brain-machine interface. For humans, vision is the dominant contributor to the interaction between the external environment and the brain. In this study, utilizing the retinal neural spike data collected over multi trials with visual stimuli of two movies with different levels of scene complexity, we used a neural network decoder to quantify the decoded visual stimuli with six different metrics for image quality assessment establishing comprehensive inspection of decoding. With the detailed and systematical study of the effect and single and multiple trials of data, different noise in spikes, and blurred images, our results provide an in-depth investigation of decoding dynamical visual scenes using retinal spikes. These results provide insights into the neural coding of visual scenes and services as a guideline for designing next-generation decoding algorithms of neuroprosthesis and other devices of brain-machine interface.

Retinal ganglion cells encode the direction of motion outside their classical receptive field

Retinal ganglion cells (RGCs) typically respond to light stimulation over their spatially restricted receptive field. Using large-scale recordings in the mouse retina, we show that a subset of non- direction-selective (DS) RGCs exhibit asymmetric activity, selective to motion direction, in response to a stimulus crossing an area far beyond the classic receptive field. The extraclassical response arises via inputs from an asymmetric distal zone and is enhanced by desensitization mechanisms and an inherent DS component, creating a network of neurons responding to motion toward the optic disc. Pharmacological manipulations revealed the necessity of glycinergic amacrine cells for this response. Using in vivo recordings, we identified similar extraclassical responses in lateral geniculate nucleus neurons, suggesting such non conventional DS information is transferred to downstream structures. Our results suggest a complex integration of motion direction processing across the visual field, which arises beyond the classical receptive field boundaries.

Biological computation through recurrence

One of the defining features of living systems is their adaptability to changing environmental conditions. This requires organisms to extract temporal and spatial features of their environment, and use that information to compute the appropriate response. In the last two decades, a growing body of work, mainly coming from the machine learning and computational neuroscience fields, has shown that such complex information processing can be performed by recurrent networks. Temporal computations arise in these networks through the interplay between the external stimuli and the network's internal state. In this article we review our current understanding of how recurrent networks can be used by biological systems, from cells to brains, for complex information processing. Rather than focusing on sophisticated, artificial recurrent architectures such as long short-term memory (LSTM) networks, here we concentrate on simpler network structures and learning algorithms that can be expected to have been found by evolution. We also review studies showing evidence of naturally occurring recurrent networks in living organisms. Lastly, we discuss some relevant evolutionary aspects concerning the emergence of this natural computation paradigm.

Decoding dynamic visual scenes across the brain hierarchy

Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Computational capabilities of a multicellular reservoir computing system

The capacity of cells to process information is currently used to design cell-based tools for ecological, industrial, and biomedical applications such as detecting dangerous chemicals or for bioremediation. In most applications, individual cells are used as the information processing unit. However, single cell engineering is limited by the necessary molecular complexity and the accompanying metabolic burden of synthetic circuits. To overcome these limitations, synthetic biologists have begun engineering multicellular systems that combine cells with designed subfunctions. To further advance information processing in synthetic multicellular systems, we introduce the application of reservoir computing. Reservoir computers (RCs) approximate a temporal signal processing task via a fixed-rule dynamic network (the reservoir) with a regression-based readout. Importantly, RCs eliminate the need of network rewiring, as different tasks can be approximated with the same reservoir. Previous work has already demonstrated the capacity of single cells, as well as populations of neurons, to act as reservoirs. In this work, we extend reservoir computing in multicellular populations with the widespread mechanism of diffusion-based cell-to-cell signaling. As a proof-of-concept, we simulated a reservoir made of a 3D community of cells communicating via diffusible molecules and used it to approximate a range of binary signal processing tasks, focusing on two benchmark functions-computing median and parity functions from binary input signals. We demonstrate that a diffusion-based multicellular reservoir is a feasible synthetic framework for performing complex temporal computing tasks that provides a computational advantage over single cell reservoirs. We also identified a number of biological properties that can affect the computational performance of these processing systems.

Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction

Objective. Retinal prostheses are promising devices to restore vision for patients with severe age-related macular degeneration or retinitis pigmentosa disease. The visual processing mechanism embodied in retinal prostheses play an important role in the restoration effect. Its performance depends on our understanding of the retina's working mechanism and the evolvement of computer vision models. Recently, remarkable progress has been made in the field of processing algorithm for retinal prostheses where the new discovery of the retina's working principle and state-of-the-arts computer vision models are combined together.Approach. We investigated the related research on artificial intelligence techniques for retinal prostheses. The processing algorithm in these studies could be attributed to three types: computer vision-related methods, biophysical models, and deep learning models.Main results. In this review, we first illustrate the structure and function of the normal and degenerated retina, then demonstrate the vision rehabilitation mechanism of three representative retinal prostheses. It is necessary to summarize the computational frameworks abstracted from the normal retina. In addition, the development and feature of three types of different processing algorithms are summarized. Finally, we analyze the bottleneck in existing algorithms and propose our prospect about the future directions to improve the restoration effect.Significance. This review systematically summarizes existing processing models for predicting the response of the retina to external stimuli. What's more, the suggestions for future direction may inspire researchers in this field to design better algorithms for retinal prostheses.

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

Decoding Pixel-Level Image Features from Two-Photon Calcium Signals of Macaque Visual Cortex

Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Linear Response of General Observables in Spiking Neuronal Network Models

We establish a general linear response relation for spiking neuronal networks, based on chains with unbounded memory. This relation allow us to predict the influence of a weak amplitude time dependent external stimuli on spatio-temporal spike correlations, from the spontaneous statistics (without stimulus) in a general context where the memory in spike dynamics can extend arbitrarily far in the past. Using this approach, we show how the linear response is explicitly related to the collective effect of the stimuli, intrinsic neuronal dynamics, and network connectivity on spike train statistics. We illustrate our results with numerical simulations performed over a discrete time integrate and fire model.

Modeling a population of retinal ganglion cells with restricted Boltzmann machines

The retina is a complex circuit of the central nervous system whose aim is to encode visual stimuli prior the higher order processing performed in the visual cortex. Due to the importance of its role, modeling the retina to advance in interpreting its spiking activity output is a well studied problem. In particular, it has been shown that latent variable models can be used to model the joint distribution of Retinal Ganglion Cells (RGCs). In this work, we validate the applicability of Restricted Boltzmann Machines to model the spiking activity responses of a large a population of RGCs recorded with high-resolution electrode arrays. In particular, we show that latent variables can encode modes in the RGC activity distribution that are closely related to the visual stimuli. In contrast to previous work, we further validate our findings by comparing results associated with recordings from retinas under normal and altered encoding conditions obtained by pharmacological manipulation. In these conditions, we observe that the model reflects well-known physiological behaviors of the retina. Finally, we show that we can also discover temporal patterns, associated with distinct dynamics of the stimuli.

What the salamander eye has been telling the vision scientist's brain

Salamanders have been habitual residents of research laboratories for more than a century, and their history in science is tightly interwoven with vision research. Nevertheless, many vision scientists - even those working with salamanders - may be unaware of how much our knowledge about vision, and particularly the retina, has been shaped by studying salamanders. In this review, we take a tour through the salamander history in vision science, highlighting the main contributions of salamanders to our understanding of the vertebrate retina. We further point out specificities of the salamander visual system and discuss the perspectives of this animal system for future vision research.

Neural variability determines coding strategies for natural self-motion in macaque monkeys

We have previously reported that central neurons mediating vestibulo-spinal reflexes and self-motion perception optimally encode natural self-motion (Mitchell et al., 2018). Importantly however, the vestibular nuclei also comprise other neuronal classes that mediate essential functions such as the vestibulo-ocular reflex (VOR) and its adaptation. Here we show that heterogeneities in resting discharge variability mediate a trade-off between faithful encoding and optimal coding via temporal whitening. Specifically, neurons displaying lower variability did not whiten naturalistic self-motion but instead faithfully represented the stimulus' detailed time course, while neurons displaying higher variability displayed temporal whitening. Using a well-established model of VOR pathways, we demonstrate that faithful stimulus encoding is necessary to generate the compensatory eye movements found experimentally during naturalistic self-motion. Our findings suggest a novel functional role for variability toward establishing different coding strategies: (1) faithful stimulus encoding for generating the VOR; (2) optimized coding via temporal whitening for other vestibular functions.

Global Motion Processing by Populations of Direction-Selective Retinal Ganglion Cells

Simple stimuli have been critical to understanding neural population codes in sensory systems. Yet it remains necessary to determine the extent to which this understanding generalizes to more complex conditions. To examine this problem, we measured how populations of direction-selective ganglion cells (DSGCs) from the retinas of male and female mice respond to a global motion stimulus with its direction and speed changing dynamically. We then examined the encoding and decoding of motion direction in both individual and populations of DSGCs. Individual cells integrated global motion over ∼200 ms, and responses were tuned to direction. However, responses were sparse and broadly tuned, which severely limited decoding performance from small DSGC populations. In contrast, larger populations compensated for response sparsity, enabling decoding with high temporal precision (<100 ms). At these timescales, correlated spiking was minimal and had little impact on decoding performance, unlike results obtained using simpler local motion stimuli decoded over longer timescales. We use these data to define different DSGC population decoding regimes that use or mitigate correlated spiking to achieve high-spatial versus high-temporal resolution.SIGNIFICANCE STATEMENT ON-OFF direction-selective ganglion cells (ooDSGCs) in the mammalian retina are typically thought to signal local motion to the brain. However, several recent studies suggest they may signal global motion. Here we analyze the fidelity of encoding and decoding global motion in a natural scene across large populations of ooDSGCs. We show that large populations of DSGCs are capable of signaling rapid changes in global motion.

Towards optogenetic vision restoration with high resolution

In many cases of inherited retinal degenerations, ganglion cells are spared despite photoreceptor cell death, making it possible to stimulate them to restore visual function. Several studies have shown that it is possible to express an optogenetic protein in ganglion cells and make them light sensitive, a promising strategy to restore vision. However the spatial resolution of optogenetically-reactivated retinas has rarely been measured, especially in the primate. Since the optogenetic protein is also expressed in axons, it is unclear if these neurons will only be sensitive to the stimulation of a small region covering their somas and dendrites, or if they will also respond to any stimulation overlapping with their axon, dramatically impairing spatial resolution. Here we recorded responses of mouse and macaque retinas to random checkerboard patterns following an in vivo optogenetic therapy. We show that optogenetically activated ganglion cells are each sensitive to a small region of visual space. A simple model based on this small receptive field predicted accurately their responses to complex stimuli. From this model, we simulated how the entire population of light sensitive ganglion cells would respond to letters of different sizes. We then estimated the maximal acuity expected by a patient, assuming it could make an optimal use of the information delivered by this reactivated retina. The obtained acuity is above the limit of legal blindness. Our model also makes interesting predictions on how acuity might vary upon changing the therapeutic strategy, assuming an optimal use of the information present in the retinal activity. Optogenetic therapy could thus potentially lead to high resolution vision, under conditions that our model helps to determinine.

In many cases of blindness, ganglion cells, the retinal output, remain functional. A promising strategy to restore vision is to express optogenetic proteins in ganglion cells. However, it is not clear what is the resolution of this new light sensor. A major concern is that axons might become light sensitive, and a focal stimulation would activate a very broad area of the retina, dramatically impairing spatial resolution. Here we show that this is not the case. Ganglion cells are activated only by stimulations close to their soma. Using a combination of data analysis and modeling based on mouse and non-human primate retina recordings, we show that the acuity expected with this therapy could be above the level of legal blindness.

Estimating information in time-varying signals

Across diverse biological systems-ranging from neural networks to intracellular signaling and genetic regulatory networks-the information about changes in the environment is frequently encoded in the full temporal dynamics of the network nodes. A pressing data-analysis challenge has thus been to efficiently estimate the amount of information that these dynamics convey from experimental data. Here we develop and evaluate decoding-based estimation methods to lower bound the mutual information about a finite set of inputs, encoded in single-cell high-dimensional time series data. For biological reaction networks governed by the chemical Master equation, we derive model-based information approximations and analytical upper bounds, against which we benchmark our proposed model-free decoding estimators. In contrast to the frequently-used k-nearest-neighbor estimator, decoding-based estimators robustly extract a large fraction of the available information from high-dimensional trajectories with a realistic number of data samples. We apply these estimators to previously published data on Erk and Ca2+ signaling in mammalian cells and to yeast stress-response, and find that substantial amount of information about environmental state can be encoded by non-trivial response statistics even in stationary signals. We argue that these single-cell, decoding-based information estimates, rather than the commonly-used tests for significant differences between selected population response statistics, provide a proper and unbiased measure for the performance of biological signaling networks.

Evolutionary aspects of reservoir computing

Reservoir computing (RC) is a powerful computational paradigm that allows high versatility with cheap learning. While other artificial intelligence approaches need exhaustive resources to specify their inner workings, RC is based on a reservoir with highly nonlinear dynamics that does not require a fine tuning of its parts. These dynamics project input signals into high-dimensional spaces, where training linear readouts to extract input features is vastly simplified. Thus, inexpensive learning provides very powerful tools for decision-making, controlling dynamical systems, classification, etc. RC also facilitates solving multiple tasks in parallel, resulting in a high throughput. Existing literature focuses on applications in artificial intelligence and neuroscience. We review this literature from an evolutionary perspective. RC's versatility makes it a great candidate to solve outstanding problems in biology, which raises relevant questions. Is RC as abundant in nature as its advantages should imply? Has it evolved? Once evolved, can it be easily sustained? Under what circumstances? (In other words, is RC an evolutionarily stable computing paradigm?) To tackle these issues, we introduce a conceptual morphospace that would map computational selective pressures that could select for or against RC and other computing paradigms. This guides a speculative discussion about the questions above and allows us to propose a solid research line that brings together computation and evolution with RC as test model of the proposed hypotheses. This article is part of the theme issue 'Liquid brains, solid brains: How distributed cognitive architectures process information'.

Optimal Decoding of Cellular Identities in a Genetic Network

In developing organisms, spatially prescribed cell identities are thought to be determined by the expression levels of multiple genes. Quantitative tests of this idea, however, require a theoretical framework capable of exposing the rules and precision of cell specification over developmental time. We use the gap gene network in the early fly embryo as an example to show how expression levels of the four gap genes can be jointly decoded into an optimal specification of position with 1% accuracy. The decoder correctly predicts, with no free parameters, the dynamics of pair-rule expression patterns at different developmental time points and in various mutant backgrounds. Precise cellular identities are thus available at the earliest stages of development, contrasting the prevailing view of positional information being slowly refined across successive layers of the patterning network. Our results suggest that developmental enhancers closely approximate a mathematically optimal decoding strategy.

Activity Correlations between Direction-Selective Retinal Ganglion Cells Synergistically Enhance Motion Decoding from Complex Visual Scenes

Neurons in sensory systems are often tuned to particular stimulus features. During complex naturalistic stimulation, however, multiple features may simultaneously affect neuronal responses, which complicates the readout of individual features. To investigate feature representation under complex stimulation, we studied how direction-selective ganglion cells in salamander retina respond to texture motion where direction, velocity, and spatial pattern inside the receptive field continuously change. We found that the cells preserve their direction preference under this stimulation, yet their direction encoding becomes ambiguous due to simultaneous activation by luminance changes. The ambiguities can be resolved by considering populations of direction-selective cells with different preferred directions. This gives rise to synergistic motion decoding, yielding more information from the population than the summed information from single-cell responses. Strong positive response correlations between cells with different preferred directions amplify this synergy. Our results show how correlated population activity can enhance feature extraction in complex visual scenes.

•

Direction-selective ganglion cells respond to motion as well as luminance changes

•

This obscures the readout of direction from single cells under complex texture motion

•

Population decoding improves direction readout supralinearly over individual cells

•

Strong spike correlations further enhance readout through increased synergy

Derivatives and inverse of cascaded linear+nonlinear neural models

In vision science, cascades of Linear+Nonlinear transforms are very successful in modeling a number of perceptual experiences. However, the conventional literature is usually too focused on only describing the forward input-output transform. Instead, in this work we present the mathematics of such cascades beyond the forward transform, namely the Jacobian matrices and the inverse. The fundamental reason for this analytical treatment is that it offers useful analytical insight into the psychophysics, the physiology, and the function of the visual system. For instance, we show how the trends of the sensitivity (volume of the discrimination regions) and the adaptation of the receptive fields can be identified in the expression of the Jacobian w.r.t. the stimulus. This matrix also tells us which regions of the stimulus space are encoded more efficiently in multi-information terms. The Jacobian w.r.t. the parameters shows which aspects of the model have bigger impact in the response, and hence their relative relevance. The analytic inverse implies conditions for the response and model parameters to ensure appropriate decoding. From the experimental and applied perspective, (a) the Jacobian w.r.t. the stimulus is necessary in new experimental methods based on the synthesis of visual stimuli with interesting geometrical properties, (b) the Jacobian matrices w.r.t. the parameters are convenient to learn the model from classical experiments or alternative goal optimization, and (c) the inverse is a promising model-based alternative to blind machine-learning methods for neural decoding that do not include meaningful biological information. The theory is checked by building and testing a vision model that actually follows a modular Linear+Nonlinear program. Our illustrative derivable and invertible model consists of a cascade of modules that account for brightness, contrast, energy masking, and wavelet masking. To stress the generality of this modular setting we show examples where some of the canonical Divisive Normalization modules are substituted by equivalent modules such as the Wilson-Cowan interaction model (at the V1 cortex) or a tone-mapping model (at the retina).

Clinical Impact of Spontaneous Hyperactivity in Degenerating Retinas: Significance for Diagnosis, Symptoms, and Treatment

Hereditary retinal degenerations result from varied pathophysiologic mechanisms, all ultimately characterized by photoreceptor dysfunction and death. Hence, much research on these diseases has concentrated on the outer retina. Over the past decade or so increasing attention has focused on concomitant changes in complex inner retinal neural circuits that process visual signals for transmission to the brain. One striking abnormality develops before the ultimately profound anatomic disruption of the inner retina. Highly elevated spontaneous activity was first demonstrated in central nervous system visual centers in vivo by Dräger and Hubel (1978), and subsequently has been confirmed in vitro, now in multiple animal models and by multiple investigators (see other contributions to this Research Topic). What evidence exists that this phenomenon occurs in human patients with retinal degeneration, and what is the ultimate effect of spontaneous hyperactivity in the output neurons, the retinal ganglion cells? Here I summarize abnormalities of visual perception among patients with retinal degeneration that may arise from hyperactivity. Next, I consider the disruption of neural encoding and anatomic connectivity that may result within the retina and in downstream visual centers of the brain. I then consider how specific characteristics of hyperactivity may distinguish various forms or stages of retinal degeneration, potentially helping in the near future to refine diagnosis and/or treatment choices for different patients. Finally, I review how consideration of these features may help optimize pharmacologic, gene, stem cell, prosthetic or other therapies to forestall visual loss or restore sight.

Nonlinear decoding of a complex movie from the mammalian retina

Retina is a paradigmatic system for studying sensory encoding: the transformation of light into spiking activity of ganglion cells. The inverse problem, where stimulus is reconstructed from spikes, has received less attention, especially for complex stimuli that should be reconstructed "pixel-by-pixel". We recorded around a hundred neurons from a dense patch in a rat retina and decoded movies of multiple small randomly-moving discs. We constructed nonlinear (kernelized and neural network) decoders that improved significantly over linear results. An important contribution to this was the ability of nonlinear decoders to reliably separate between neural responses driven by locally fluctuating light signals, and responses at locally constant light driven by spontaneous-like activity. This improvement crucially depended on the precise, non-Poisson temporal structure of individual spike trains, which originated in the spike-history dependence of neural responses. We propose a general principle by which downstream circuitry could discriminate between spontaneous and stimulus-driven activity based solely on higher-order statistical structure in the incoming spike trains.

Blindfold learning of an accurate neural metric

The brain has no direct access to physical stimuli but only to the spiking activity evoked in sensory organs. It is unclear how the brain can learn representations of the stimuli based on those noisy, correlated responses alone. Here we show how to build an accurate distance map of responses solely from the structure of the population activity of retinal ganglion cells. We introduce the Temporal Restricted Boltzmann Machine to learn the spatiotemporal structure of the population activity and use this model to define a distance between spike trains. We show that this metric outperforms existing neural distances at discriminating pairs of stimuli that are barely distinguishable. The proposed method provides a generic and biologically plausible way to learn to associate similar stimuli based on their spiking responses, without any other knowledge of these stimuli.

Learning to make external sensory stimulus predictions using internal correlations in populations of neurons

To compensate for sensory processing delays, the visual system must make predictions to ensure timely and appropriate behaviors. Recent work has found predictive information about the stimulus in neural populations early in vision processing, starting in the retina. However, to utilize this information, cells downstream must be able to read out the predictive information from the spiking activity of retinal ganglion cells. Here we investigate whether a downstream cell could learn efficient encoding of predictive information in its inputs from the correlations in the inputs themselves, in the absence of other instructive signals. We simulate learning driven by spiking activity recorded in salamander retina. We model a downstream cell as a binary neuron receiving a small group of weighted inputs and quantify the predictive information between activity in the binary neuron and future input. Input weights change according to spike timing-dependent learning rules during a training period. We characterize the readouts learned under spike timing-dependent synaptic update rules, finding that although the fixed points of learning dynamics are not associated with absolute optimal readouts they convey nearly all of the information conveyed by the optimal readout. Moreover, we find that learned perceptrons transmit position and velocity information of a moving-bar stimulus nearly as efficiently as optimal perceptrons. We conclude that predictive information is, in principle, readable from the perspective of downstream neurons in the absence of other inputs. This suggests an important role for feedforward prediction in sensory encoding.

Multiplexed computations in retinal ganglion cells of a single type

In the early visual system, cells of the same type perform the same computation in different places of the visual field. How these cells code together a complex visual scene is unclear. A common assumption is that cells of a single-type extract a single-stimulus feature to form a feature map, but this has rarely been observed directly. Using large-scale recordings in the rat retina, we show that a homogeneous population of fast OFF ganglion cells simultaneously encodes two radically different features of a visual scene. Cells close to a moving object code quasilinearly for its position, while distant cells remain largely invariant to the object's position and, instead, respond nonlinearly to changes in the object's speed. We develop a quantitative model that accounts for this effect and identify a disinhibitory circuit that mediates it. Ganglion cells of a single type thus do not code for one, but two features simultaneously. This richer, flexible neural map might also be present in other sensory systems.

Closed-Loop Estimation of Retinal Network Sensitivity by Local Empirical Linearization

Understanding how sensory systems process information depends crucially on identifying which features of the stimulus drive the response of sensory neurons, and which ones leave their response invariant. This task is made difficult by the many nonlinearities that shape sensory processing. Here, we present a novel perturbative approach to understand information processing by sensory neurons, where we linearize their collective response locally in stimulus space. We added small perturbations to reference stimuli and tested if they triggered visible changes in the responses, adapting their amplitude according to the previous responses with closed-loop experiments. We developed a local linear model that accurately predicts the sensitivity of the neural responses to these perturbations. Applying this approach to the rat retina, we estimated the optimal performance of a neural decoder and showed that the nonlinear sensitivity of the retina is consistent with an efficient encoding of stimulus information. Our approach can be used to characterize experimentally the sensitivity of neural systems to external stimuli locally, quantify experimentally the capacity of neural networks to encode sensory information, and relate their activity to behavior.

Probabilistic models for neural populations that naturally capture global coupling and criticality

Advances in multi-unit recordings pave the way for statistical modeling of activity patterns in large neural populations. Recent studies have shown that the summed activity of all neurons strongly shapes the population response. A separate recent finding has been that neural populations also exhibit criticality, an anomalously large dynamic range for the probabilities of different population activity patterns. Motivated by these two observations, we introduce a class of probabilistic models which takes into account the prior knowledge that the neural population could be globally coupled and close to critical. These models consist of an energy function which parametrizes interactions between small groups of neurons, and an arbitrary positive, strictly increasing, and twice differentiable function which maps the energy of a population pattern to its probability. We show that: 1) augmenting a pairwise Ising model with a nonlinearity yields an accurate description of the activity of retinal ganglion cells which outperforms previous models based on the summed activity of neurons; 2) prior knowledge that the population is critical translates to prior expectations about the shape of the nonlinearity; 3) the nonlinearity admits an interpretation in terms of a continuous latent variable globally coupling the system whose distribution we can infer from data. Our method is independent of the underlying system’s state space; hence, it can be applied to other systems such as natural scenes or amino acid sequences of proteins which are also known to exhibit criticality.

Populations of sensory neurons represent information about the outside environment in a collective fashion. A salient property of this distributed neural code is criticality. Yet most models used to date to analyze recordings from large neural populations do not take this observation explicitly into account. Here we aim to bridge this gap by designing probabilistic models whose structure reflects the expectation that the population is close to critical. We show that such principled approach improves previously considered models, and we demonstrate a connection between our models and the presence of continuous latent variables which is a recently proposed mechanism underlying criticality in many natural systems.

Joint Encoding of Object Motion and Motion Direction in the Salamander Retina

The processing of motion in visual scenes is important for detecting and tracking moving objects as well as for monitoring self-motion through the induced optic flow. Specialized neural circuits have been identified in the vertebrate retina for detecting motion direction or for distinguishing between object motion and self-motion, although little is known about how information about these distinct features of visual motion is combined. The salamander retina, which is a widely used model system for analyzing retinal function, contains object-motion-sensitive (OMS) ganglion cells, which strongly respond to local motion signals but are suppressed by global image motion. Yet, direction-selective (DS) ganglion cells have been conspicuously absent from characterizations of the salamander retina, despite their ubiquity in other model systems. We here show that the retina of axolotl salamanders contains at least two distinct classes of DS ganglion cells. For one of these classes, the cells display a strong preference for local over global motion in addition to their direction selectivity (OMS-DS cells) and thereby combine sensitivity to two distinct motion features. The OMS-DS cells are further distinct from standard (non-OMS) DS cells by their smaller receptive fields and different organization of preferred motion directions. Our results suggest that the two classes of DS cells specialize to encode motion direction of local and global motion stimuli, respectively, even for complex composite motion scenes. Furthermore, although the salamander DS cells are OFF-type, there is a strong analogy to the systems of ON and ON-OFF DS cells in the mammalian retina.

SIGNIFICANCE STATEMENT

The retina contains specialized cells for motion processing. Among the retinal ganglion cells, which form the output neurons of the retina, some are known to report the direction of a moving stimulus (direction-selective cells), and others distinguish the motion of an object from a moving background. But little is known about how information about local object motion and information about motion direction interact. Here, we report that direction-selective ganglion cells can be identified in the salamander retina, where their existence had been unclear. Furthermore, there are two independent systems of direction-selective cells, and one of these combines direction selectivity with sensitivity to local motion. The output of these cells could assist in tracking moving objects and estimating their future position.

Sensory noise predicts divisive reshaping of receptive fields

In order to respond reliably to specific features of their environment, sensory neurons need to integrate multiple incoming noisy signals. Crucially, they also need to compete for the interpretation of those signals with other neurons representing similar features. The form that this competition should take depends critically on the noise corrupting these signals. In this study we show that for the type of noise commonly observed in sensory systems, whose variance scales with the mean signal, sensory neurons should selectively divide their input signals by their predictions, suppressing ambiguous cues while amplifying others. Any change in the stimulus context alters which inputs are suppressed, leading to a deep dynamic reshaping of neural receptive fields going far beyond simple surround suppression. Paradoxically, these highly variable receptive fields go alongside and are in fact required for an invariant representation of external sensory features. In addition to offering a normative account of context-dependent changes in sensory responses, perceptual inference in the presence of signal-dependent noise accounts for ubiquitous features of sensory neurons such as divisive normalization, gain control and contrast dependent temporal dynamics.

Probing the functional impact of sub-retinal prosthesis

Retinal prostheses are promising tools for recovering visual functions in blind patients but, unfortunately, with still poor gains in visual acuity. Improving their resolution is thus a key challenge that warrants understanding its origin through appropriate animal models. Here, we provide a systematic comparison between visual and prosthetic activations of the rat primary visual cortex (V1). We established a precise V1 mapping as a functional benchmark to demonstrate that sub-retinal implants activate V1 at the appropriate position, scalable to a wide range of visual luminance, but with an aspect-ratio and an extent much larger than expected. Such distorted activation profile can be accounted for by the existence of two sources of diffusion, passive diffusion and activation of ganglion cells’ axons en passant. Reverse-engineered electrical pulses based on impedance spectroscopy is the only solution we tested that decreases the extent and aspect-ratio, providing a promising solution for clinical applications.

DOI:http://dx.doi.org/10.7554/eLife.12687.001

One of the most common causes of blindness is a disorder called retinitis pigmentosa. In a healthy eye, the surface at the back of the eye – called the retina – contains cells called photoreceptors that detect light and convert it into electrical signals for the brain to process. In people with retinitis pigmentosa, these photoreceptor cells die off gradually, which leads to loss of vision.

The only treatment available for retinitis pigmentosa is to have an artificial retina implanted into the eye. The artificial retina consists of an array of tiny electrodes, which take over from the damaged photoreceptors and generate electrical signals. The person with the implant perceives these electrical signals as bright flashes called “phosphenes”. However, the phosphenes are too large and imprecise to provide the person with vision that is good enough for tasks such as walking unaided or reading.

To find out why artificial retinas produce such poor resolution, Roux et al. compared how a rat’s brain responds to either natural visual stimuli or activation of implanted an array of micro-electrodes. Both the micro-electrodes and the natural stimuli activated the same areas of the brain. However, the micro-electrodes produced larger and more elongated patterns of activation. This is because the electrical currents generated by the micro-electrodes diffused throughout the retinal tissue and activated other neurons besides those intended. To overcome this problem, Roux et al. tested different ways of stimulating the micro-electrodes in order to identify those that induce the desired patterns of brain activity. This approach – known as reverse engineering – did indeed improve the performance of the micro-electrode array.

The next step is to extend these findings, which were obtained in healthy rats, to non-human primates or animal models of retinitis pigmentosa to better understand the condition in humans. In addition, combining the current approach with other existing techniques should further improve the vision that can be achieved with artificial retinas.

DOI:http://dx.doi.org/10.7554/eLife.12687.002

Functional MRI Representational Similarity Analysis Reveals a Dissociation between Discriminative and Relative Location Information in the Human Visual System

Neural responses in visual cortex are governed by a topographic mapping from retinal locations to cortical responses. Moreover, at the voxel population level early visual cortex (EVC) activity enables accurate decoding of stimuli locations. However, in many cases information enabling one to discriminate between locations (i.e., discriminative information) may be less relevant than information regarding the relative location of two objects (i.e., relative information). For example, when planning to grab a cup, determining whether the cup is located at the same retinal location as the hand is hardly relevant, whereas the location of the cup relative to the hand is crucial for performing the action. We have previously used multivariate pattern analysis techniques to measure discriminative location information, and found the highest levels in EVC, in line with other studies. Here we show, using representational similarity analysis, that availability of discriminative information in fMRI activation patterns does not entail availability of relative information. Specifically, we find that relative location information can be reliably extracted from activity patterns in posterior intraparietal sulcus (pIPS), but not from EVC, where we find the spatial representation to be warped. We further show that this variability in relative information levels between regions can be explained by a computational model based on an array of receptive fields. Moreover, when the model's receptive fields are extended to include inhibitory surround regions, the model can account for the spatial warping in EVC. These results demonstrate how size and shape properties of receptive fields in human visual cortex contribute to the transformation of discriminative spatial representations into relative spatial representations along the visual stream.

Efficiency turns the table on neural encoding, decoding and noise

Sensory neurons are usually described with an encoding model, for example, a function that predicts their response from the sensory stimulus using a receptive field (RF) or a tuning curve. However, central to theories of sensory processing is the notion of 'efficient coding'. We argue here that efficient coding implies a completely different neural coding strategy. Instead of a fixed encoding model, neural populations would be described by a fixed decoding model (i.e. a model reconstructing the stimulus from the neural responses). Because the population solves a global optimization problem, individual neurons are variable, but not noisy, and have no truly invariant tuning curve or receptive field. We review recent experimental evidence and implications for neural noise correlations, robustness and adaptation.

Physiological and morphological characterization of ganglion cells in the salamander retina

Retinal ganglion cells (RGCs) integrate visual information from the retina and transmit collective signals to the brain. A systematic investigation of functional and morphological characteristics of various types of RGCs is important to comprehensively understand how the visual system encodes and transmits information via various RGC pathways. This study evaluated both physiological and morphological properties of 67 RGCs in dark-adapted flat-mounted salamander retina by examining light-evoked cation and chloride current responses via voltage-clamp recordings and visualizing morphology by Lucifer yellow fluorescence with a confocal microscope. Six groups of RGCs were described: asymmetrical ON-OFF RGCs, symmetrical ON RGCs, OFF RGCs, and narrow-, medium- and wide-field ON-OFF RGCs. Dendritic field diameters of RGCs ranged 102-490 μm: narrow field (<200 μm, 31% of RGCs), medium field (200-300 μm, 45%) and wide field (>300 μm, 24%). Dendritic ramification patterns of RGCs agree with the sublamina A/B rule. 34% of RGCs were monostratified, 24% bistratified and 42% diffusely stratified. 70% of ON RGCs and OFF RGCs were monostratified. Wide-field RGCs were diffusely stratified. 82% of RGCs generated light-evoked ON-OFF responses, while 11% generated ON responses and 7% OFF responses. Response sensitivity analysis suggested that some RGCs obtained separated rod/cone bipolar cell inputs whereas others obtained mixed bipolar cell inputs. 25% of neurons in the RGC layer were displaced amacrine cells. Although more types may be defined by more refined classification criteria, this report is to incorporate more physiological properties into RGC classification.

Philosophy of the Spike: Rate-Based vs. Spike-Based Theories of the Brain

Does the brain use a firing rate code or a spike timing code? Considering this controversial question from an epistemological perspective, I argue that progress has been hampered by its problematic phrasing. It takes the perspective of an external observer looking at whether those two observables vary with stimuli, and thereby misses the relevant question: which one has a causal role in neural activity? When rephrased in a more meaningful way, the rate-based view appears as an ad hoc methodological postulate, one that is practical but with virtually no empirical or theoretical support.