Inference of neuronal functional circuitry with spike-triggered non-negative matrix factorization

Filter-based models of suppression in retinal ganglion cells: Comparison and generalization across species and stimuli

The dichotomy of excitation and suppression is one of the canonical mechanisms explaining the complexity of neural activity. Computational models of the interplay of excitation and suppression in single neurons aim at investigating how this interaction affects a neuron's spiking responses and shapes the encoding of sensory stimuli. Here, we compare the performance of three filter-based stimulus-encoding models for predicting retinal ganglion cell responses recorded from axolotl, mouse, and marmoset retina to different types of temporally varying visual stimuli. Suppression in these models is implemented via subtractive or divisive interactions of stimulus filters or by a response-driven feedback module. For the majority of ganglion cells, the subtractive and divisive models perform similarly and outperform the feedback model as well as a linear-nonlinear (LN) model with no suppression. Comparison between the subtractive and the divisive model depends on cell type, species, and stimulus components, with the divisive model generalizing best across temporal stimulus frequencies and visual contrast and the subtractive model capturing in particular responses for slow temporal stimulus dynamics and for slow axolotl cells. Overall, we conclude that the divisive and subtractive models are well suited for capturing interactions of excitation and suppression in ganglion cells and perform best for different temporal regimes of these interactions.

Fluorescence microscopy through scattering media with robust matrix factorization

Biological tissues, as natural scattering media, inherently disrupt structural information, presenting significant challenges for optical imaging. Complex light propagation through tissue severely degrades image quality, limiting conventional fluorescence imaging techniques to superficial depths. Extracting meaningful information from random speckle patterns is, therefore, critical for deeper tissue imaging. In this study, we present RNP (robust non-negative principal matrix factorization), an approach that enables fluorescence microscopy under diverse scattering conditions. By integrating robust feature extraction with non-negativity constraints, RNP effectively addresses challenges posed by non-sparse signals and background interference in scattering tissue environments. The framework operates on a standard epi-fluorescence platform, eliminating the need for complex instrumentation or precise alignment. The results from imaging scattered cells and tissues demonstrate substantial improvements in robustness, field of view, depth of field, and image clarity. We anticipate that RNP will become a valuable tool for overcoming scattering challenges in fluorescence microscopy and driving advancements in biomedical research.

Subspace analysis identifies low-dimensional interactions between cortical and subcortical brain regions in social reward computation

Animals living in social environments evaluate rewards given to themselves and others. We previously showed that single-unit activities in cortical (the medial prefrontal cortex, MPFC) and subcortical (the dopaminergic midbrain nuclei, DA, and the lateral hypothalamus, LH) regions reflect social reward computation. Extending the single-neuron analyses, this study employs matrix and tensor decomposition methods to characterize population activity within single regions and interactions between pairs of regions. First, we determined the dimensionality of population activity and corresponding components in a single brain region. The dimensions of MPFC and LH were comparable, indicating similarities in the population activities of the two regions. In contrast, the dimensions of DA were considerably smaller, indicating that the activities were idiosyncratic. Further, "subspaces" shared between MPFC and DA, between MPFC and LH, and between LH and DA were identified. We found that a few components in MPFC and LH explained a large portion of population activities in DA, indicating that the neural computation of social rewards resides in a small subspace. Our findings demonstrate that a limited number of neural components within cortico-subcortical circuits regulate the social monitoring of rewards to oneself and others.

Robust Decoding of Rich Dynamical Visual Scenes With Retinal Spikes

Sensory information transmitted to the brain activates neurons to create a series of coping behaviors. Understanding the mechanisms of neural computation and reverse engineering the brain to build intelligent machines requires establishing a robust relationship between stimuli and neural responses. Neural decoding aims to reconstruct the original stimuli that trigger neural responses. With the recent upsurge of artificial intelligence, neural decoding provides an insightful perspective for designing novel algorithms of brain-machine interface. For humans, vision is the dominant contributor to the interaction between the external environment and the brain. In this study, utilizing the retinal neural spike data collected over multi trials with visual stimuli of two movies with different levels of scene complexity, we used a neural network decoder to quantify the decoded visual stimuli with six different metrics for image quality assessment establishing comprehensive inspection of decoding. With the detailed and systematical study of the effect and single and multiple trials of data, different noise in spikes, and blurred images, our results provide an in-depth investigation of decoding dynamical visual scenes using retinal spikes. These results provide insights into the neural coding of visual scenes and services as a guideline for designing next-generation decoding algorithms of neuroprosthesis and other devices of brain-machine interface.

Nonlinear receptive fields evoke redundant retinal coding of natural scenes

The role of the vertebrate retina in early vision is generally described by the efficient coding hypothesis1,2, which predicts that the retina reduces the redundancy inherent in natural scenes3 by discarding spatiotemporal correlations while preserving stimulus information4. It is unclear, however, whether the predicted decorrelation and redundancy reduction in the activity of ganglion cells, the retina's output neurons, hold under gaze shifts, which dominate the dynamics of the natural visual input5. We show here that species-specific gaze patterns in natural stimuli can drive correlated spiking responses both in and across distinct types of ganglion cells in marmoset as well as mouse retina. These concerted responses disrupt redundancy reduction to signal fixation periods with locally high spatial contrast. Model-based analyses of ganglion cell responses to natural stimuli show that the observed response correlations follow from nonlinear pooling of ganglion cell inputs. Our results indicate cell-type-specific deviations from efficient coding in retinal processing of natural gaze shifts.

A chromatic feature detector in the retina signals visual context changes

The retina transforms patterns of light into visual feature representations supporting behaviour. These representations are distributed across various types of retinal ganglion cells (RGCs), whose spatial and temporal tuning properties have been studied extensively in many model organisms, including the mouse. However, it has been difficult to link the potentially nonlinear retinal transformations of natural visual inputs to specific ethological purposes. Here, we discover a nonlinear selectivity to chromatic contrast in an RGC type that allows the detection of changes in visual context. We trained a convolutional neural network (CNN) model on large-scale functional recordings of RGC responses to natural mouse movies, and then used this model to search in silico for stimuli that maximally excite distinct types of RGCs. This procedure predicted centre colour opponency in transient suppressed-by-contrast (tSbC) RGCs, a cell type whose function is being debated. We confirmed experimentally that these cells indeed responded very selectively to Green-OFF, UV-ON contrasts. This type of chromatic contrast was characteristic of transitions from ground to sky in the visual scene, as might be elicited by head or eye movements across the horizon. Because tSbC cells performed best among all RGC types at reliably detecting these transitions, we suggest a role for this RGC type in providing contextual information (i.e. sky or ground) necessary for the selection of appropriate behavioural responses to other stimuli, such as looming objects. Our work showcases how a combination of experiments with natural stimuli and computational modelling allows discovering novel types of stimulus selectivity and identifying their potential ethological relevance.

Fixational eye movements enhance the precision of visual information transmitted by the primate retina

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Applying Super-Resolution and Tomography Concepts to Identify Receptive Field Subunits in the Retina

Spatially nonlinear stimulus integration by retinal ganglion cells lies at the heart of various computations performed by the retina. It arises from the nonlinear transmission of signals that ganglion cells receive from bipolar cells, which thereby constitute functional subunits within a ganglion cell's receptive field. Inferring these subunits from recorded ganglion cell activity promises a new avenue for studying the functional architecture of the retina. This calls for efficient methods, which leave sufficient experimental time to leverage the acquired knowledge for further investigating identified subunits. Here, we combine concepts from super-resolution microscopy and computed tomography and introduce super-resolved tomographic reconstruction (STR) as a technique to efficiently stimulate and locate receptive field subunits. Simulations demonstrate that this approach can reliably identify subunits across a wide range of model variations, and application in recordings of primate parasol ganglion cells validates the experimental feasibility. STR can potentially reveal comprehensive subunit layouts within only a few tens of minutes of recording time, making it ideal for online analysis and closed-loop investigations of receptive field substructure in retina recordings.

Fixational Eye Movements Enhance the Precision of Visual Information Transmitted by the Primate Retina

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Inference of monosynaptic connections from parallel spike trains: A review

This article presents a mini-review about the progress in inferring monosynaptic connections from spike trains of multiple neurons over the past twenty years. First, we explain a variety of meanings of "neuronal connectivity" in different research areas of neuroscience, such as structural connectivity, monosynaptic connectivity, and functional connectivity. Among these, we focus on the methods used to infer the monosynaptic connectivity from spike data. We then summarize the inference methods based on two main approaches, i.e., correlation-based and model-based approaches. Finally, we describe available source codes for connectivity inference and future challenges. Although inference will never be perfect, the accuracy of identifying the monosynaptic connections has improved dramatically in recent years due to continuous efforts.

Retrospective for the Dynamic Sensorium Competition for predicting large-scale mouse primary visual cortex activity from videos

Understanding how biological visual systems process information is challenging because of the nonlinear relationship between visual input and neuronal responses. Artificial neural networks allow computational neuroscientists to create predictive models that connect biological and machine vision. Machine learning has benefited tremendously from benchmarks that compare different model on the same task under standardized conditions. However, there was no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we established the SENSORIUM 2023 Benchmark Competition with dynamic input, featuring a new large-scale dataset from the primary visual cortex of ten mice. This dataset includes responses from 78,853 neurons to 2 hours of dynamic stimuli per neuron, together with the behavioral measurements such as running speed, pupil dilation, and eye movements. The competition ranked models in two tracks based on predictive performance for neuronal responses on a held-out test set: one focusing on predicting in-domain natural stimuli and another on out-of-distribution (OOD) stimuli to assess model generalization. As part of the NeurIPS 2023 competition track, we received more than 160 model submissions from 22 teams. Several new architectures for predictive models were proposed, and the winning teams improved the previous state-of-the-art model by 50%. Access to the dataset as well as the benchmarking infrastructure will remain online at www.sensorium-competition.net.

The Dynamic Sensorium competition for predicting large-scale mouse visual cortex activity from videos

Understanding how biological visual systems process information is challenging due to the complex nonlinear relationship between neuronal responses and high-dimensional visual input. Artificial neural networks have already improved our understanding of this system by allowing computational neuroscientists to create predictive models and bridge biological and machine vision. During the Sensorium 2022 competition, we introduced benchmarks for vision models with static input (i.e. images). However, animals operate and excel in dynamic environments, making it crucial to study and understand how the brain functions under these conditions. Moreover, many biological theories, such as predictive coding, suggest that previous input is crucial for current input processing. Currently, there is no standardized benchmark to identify state-of-the-art dynamic models of the mouse visual system. To address this gap, we propose the Sensorium 2023 Benchmark Competition with dynamic input (https://www.sensorium-competition.net/). This competition includes the collection of a new large-scale dataset from the primary visual cortex of ten mice, containing responses from over 78,000 neurons to over 2 hours of dynamic stimuli per neuron. Participants in the main benchmark track will compete to identify the best predictive models of neuronal responses for dynamic input (i.e. video). We will also host a bonus track in which submission performance will be evaluated on out-of-domain input, using withheld neuronal responses to dynamic input stimuli whose statistics differ from the training set. Both tracks will offer behavioral data along with video stimuli. As before, we will provide code, tutorials, and strong pre-trained baseline models to encourage participation. We hope this competition will continue to strengthen the accompanying Sensorium benchmarks collection as a standard tool to measure progress in large-scale neural system identification models of the entire mouse visual hierarchy and beyond.

Data-driven modelling of visual receptive fields: comparison between the generalized quadratic model and the nonlinear input model

Objective: Neurons in primary visual cortex (V1) display a range of sensitivity in their response to translations of their preferred visual features within their receptive field: from high specificity to a precise position through to complete invariance. This visual feature selectivity and invariance is frequently modeled by applying a selection of linear spatial filters to the input image, that define the feature selectivity, followed by a nonlinear function that combines the filter outputs, that defines the invariance, to predict the neural response. We compare two such classes of model, that are both popular and parsimonious, the generalized quadratic model (GQM) and the nonlinear input model (NIM). These two classes of model differ primarily in that the NIM can accommodate a greater diversity in the form of nonlinearity that is applied to the outputs of the filters.Approach: We compare the two model types by applying them to data from multielectrode recordings from cat primary visual cortex in response to spatially white Gaussian noise After fitting both classes of model to a database of 342 single units (SUs), we analyze the qualitative and quantitative differences in the visual feature processing performed by the two models and their ability to predict neural response.Main results: We find that the NIM predicts response rates on a held-out data at least as well as the GQM for 95% of SUs. Superior performance occurs predominantly for those units with above average spike rates and is largely due to the NIMs ability to capture aspects of the model's nonlinear function cannot be captured with the GQM rather than differences in the visual features being processed by the two different models.Significance: These results can help guide model choice for data-driven receptive field modelling.

Protocol for dissecting cascade computational components in neural networks of a visual system

Finding the complete functional circuits of neurons is a challenging problem in brain research. Here, we present a protocol, based on visual stimuli and spikes, for obtaining the complete circuit of recorded neurons using spike-triggered nonnegative matrix factorization. We describe steps for data preprocessing, inferring the spatial receptive field of the subunits, and analyzing the module matrix. This approach identifies computational components of the feedforward network of retinal ganglion cells and dissects the network structure based on natural image stimuli. For complete details on the use and execution of this protocol, please refer to Jia et al. (2021).1.

Artificial intelligence techniques for retinal prostheses: a comprehensive review and future direction

Objective. Retinal prostheses are promising devices to restore vision for patients with severe age-related macular degeneration or retinitis pigmentosa disease. The visual processing mechanism embodied in retinal prostheses play an important role in the restoration effect. Its performance depends on our understanding of the retina's working mechanism and the evolvement of computer vision models. Recently, remarkable progress has been made in the field of processing algorithm for retinal prostheses where the new discovery of the retina's working principle and state-of-the-arts computer vision models are combined together.Approach. We investigated the related research on artificial intelligence techniques for retinal prostheses. The processing algorithm in these studies could be attributed to three types: computer vision-related methods, biophysical models, and deep learning models.Main results. In this review, we first illustrate the structure and function of the normal and degenerated retina, then demonstrate the vision rehabilitation mechanism of three representative retinal prostheses. It is necessary to summarize the computational frameworks abstracted from the normal retina. In addition, the development and feature of three types of different processing algorithms are summarized. Finally, we analyze the bottleneck in existing algorithms and propose our prospect about the future directions to improve the restoration effect.Significance. This review systematically summarizes existing processing models for predicting the response of the retina to external stimuli. What's more, the suggestions for future direction may inspire researchers in this field to design better algorithms for retinal prostheses.

Systematic reduction of the dimensionality of natural scenes allows accurate predictions of retinal ganglion cell spike outputs

The mammalian retina engages a broad array of linear and nonlinear circuit mechanisms to convert natural scenes into retinal ganglion cell (RGC) spike outputs. Although many individual integration mechanisms are well understood, we know less about how multiple mechanisms interact to encode the complex spatial features present in natural inputs. Here, we identified key spatial features in natural scenes that shape encoding by primate parasol RGCs. Our approach identified simplifications in the spatial structure of natural scenes that minimally altered RGC spike responses. We observed that reducing natural movies into 16 linearly integrated regions described ∼80% of the structure of parasol RGC spike responses; this performance depended on the number of regions but not their precise spatial locations. We used simplified stimuli to design high-dimensional metamers that recapitulated responses to naturalistic movies. Finally, we modeled the retinal computations that convert flashed natural images into one-dimensional spike counts.

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

Retinal Encoding of Natural Scenes

An ultimate goal in retina science is to understand how the neural circuit of the retina processes natural visual scenes. Yet most studies in laboratories have long been performed with simple, artificial visual stimuli such as full-field illumination, spots of light, or gratings. The underlying assumption is that the features of the retina thus identified carry over to the more complex scenario of natural scenes. As the application of corresponding natural settings is becoming more commonplace in experimental investigations, this assumption is being put to the test and opportunities arise to discover processing features that are triggered by specific aspects of natural scenes. Here, we review how natural stimuli have been used to probe, refine, and complement knowledge accumulated under simplified stimuli, and we discuss challenges and opportunities along the way toward a comprehensive understanding of the encoding of natural scenes. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Neural System Identification With Spike-Triggered Non-Negative Matrix Factorization

Neuronal circuits formed in the brain are complex with intricate connection patterns. Such complexity is also observed in the retina with a relatively simple neuronal circuit. A retinal ganglion cell (GC) receives excitatory inputs from neurons in previous layers as driving forces to fire spikes. Analytical methods are required to decipher these components in a systematic manner. Recently a method called spike-triggered non-negative matrix factorization (STNMF) has been proposed for this purpose. In this study, we extend the scope of the STNMF method. By using retinal GCs as a model system, we show that STNMF can detect various computational properties of upstream bipolar cells (BCs), including spatial receptive field, temporal filter, and transfer nonlinearity. In addition, we recover synaptic connection strengths from the weight matrix of STNMF. Furthermore, we show that STNMF can separate spikes of a GC into a few subsets of spikes, where each subset is contributed by one presynaptic BC. Taken together, these results corroborate that STNMF is a useful method for deciphering the structure of neuronal circuits.

Robust Transcoding Sensory Information With Neural Spikes

Neural coding, including encoding and decoding, is one of the key problems in neuroscience for understanding how the brain uses neural signals to relate sensory perception and motor behaviors with neural systems. However, most of the existed studies only aim at dealing with the continuous signal of neural systems, while lacking a unique feature of biological neurons, termed spike, which is the fundamental information unit for neural computation as well as a building block for brain-machine interface. Aiming at these limitations, we propose a transcoding framework to encode multi-modal sensory information into neural spikes and then reconstruct stimuli from spikes. Sensory information can be compressed into 10% in terms of neural spikes, yet re-extract 100% of information by reconstruction. Our framework can not only feasibly and accurately reconstruct dynamical visual and auditory scenes, but also rebuild the stimulus patterns from functional magnetic resonance imaging (fMRI) brain activities. More importantly, it has a superb ability of noise immunity for various types of artificial noises and background signals. The proposed framework provides efficient ways to perform multimodal feature representation and reconstruction in a high-throughput fashion, with potential usage for efficient neuromorphic computing in a noisy environment.

Electrophysiological dataset from macaque visual cortical area MST in response to a novel motion stimulus

Establishing the cortical neural representation of visual stimuli is a central challenge of systems neuroscience. Publicly available data would allow a broad range of scientific analyses and hypothesis testing, but are rare and largely focused on the early visual system. To address the shortage of open data from higher visual areas, we provide a comprehensive dataset from a neurophysiology study in macaque monkey visual cortex that includes a complete record of extracellular action potential recordings from the extrastriate medial superior temporal (MST) area, behavioral data, and detailed stimulus records. It includes spiking activity of 172 single neurons recorded in 139 sessions from 4 hemispheres of 3 rhesus macaque monkeys. The data was collected across 3 experiments, designed to characterize the response properties of MST neurons to complex motion stimuli. This data can be used to elucidate visual information processing at the level of single neurons in a high-level area of primate visual cortex. Providing open access to this dataset also promotes the 3R-principle of responsible animal research.

Measurement(s)	spike train • eye movement measurement
Technology Type(s)	single-unit recording • eye tracking device
Factor Type(s)	direction, location, and speed of moving random dot patterns
Sample Characteristic - Organism	Macaca mulatta
Sample Characteristic - Environment	laboratory environment
Sample Characteristic - Location	Germany

Simple model for encoding natural images by retinal ganglion cells with nonlinear spatial integration

A central goal in sensory neuroscience is to understand the neuronal signal processing involved in the encoding of natural stimuli. A critical step towards this goal is the development of successful computational encoding models. For ganglion cells in the vertebrate retina, the development of satisfactory models for responses to natural visual scenes is an ongoing challenge. Standard models typically apply linear integration of visual stimuli over space, yet many ganglion cells are known to show nonlinear spatial integration, in particular when stimulated with contrast-reversing gratings. We here study the influence of spatial nonlinearities in the encoding of natural images by ganglion cells, using multielectrode-array recordings from isolated salamander and mouse retinas. We assess how responses to natural images depend on first- and second-order statistics of spatial patterns inside the receptive field. This leads us to a simple extension of current standard ganglion cell models. We show that taking not only the weighted average of light intensity inside the receptive field into account but also its variance over space can partly account for nonlinear integration and substantially improve response predictions of responses to novel images. For salamander ganglion cells, we find that response predictions for cell classes with large receptive fields profit most from including spatial contrast information. Finally, we demonstrate how this model framework can be used to assess the spatial scale of nonlinear integration. Our results underscore that nonlinear spatial stimulus integration translates to stimulation with natural images. Furthermore, the introduced model framework provides a simple, yet powerful extension of standard models and may serve as a benchmark for the development of more detailed models of the nonlinear structure of receptive fields.

For understanding how sensory systems operate in the natural environment, an important goal is to develop models that capture neuronal responses to natural stimuli. For retinal ganglion cells, which connect the eye to the brain, current standard models often fail to capture responses to natural visual scenes. This shortcoming is at least partly rooted in the fact that ganglion cells may combine visual signals over space in a nonlinear fashion. We here show that a simple model, which not only considers the average light intensity inside a cell’s receptive field but also the variance of light intensity over space, can partly account for these nonlinearities and thereby improve current standard models. This provides an easy-to-obtain benchmark for modeling ganglion cell responses to natural images.

Representing the dynamics of high-dimensional data with non-redundant wavelets

A crucial question in data science is to extract meaningful information embedded in high-dimensional data into a low-dimensional set of features that can represent the original data at different levels. Wavelet analysis is a pervasive method for decomposing time-series signals into a few levels with detailed temporal resolution. However, obtained wavelets are intertwined and over-represented across levels for each sample and across different samples within one population. Here, using neuroscience data of simulated spikes, experimental spikes, calcium imaging signals, and human electrocorticography signals, we leveraged conditional mutual information between wavelets for feature selection. The meaningfulness of selected features was verified to decode stimulus or condition with high accuracy yet using only a small set of features. These results provide a new way of wavelet analysis for extracting essential features of the dynamics of spatiotemporal neural data, which then enables to support novel model design of machine learning with representative features.

•

WCMI can extract meaningful information from high-dimensional data

•

Extracted features from neural signals are non-redundant

•

Simple decoders can read out these features with superb accuracy

One of the essential questions in data science is to extract meaningful information from high-dimensional data. A useful approach is to represent data using a few features that maintain the crucial information. The leading property of spatiotemporal data is foremost ever-changing dynamics in time. Wavelet analysis, as a classical method for disentangling time series, can capture temporal dynamics with detail. Here, we leveraged conditional mutual information between wavelets to select a small subset of non-redundant features. We demonstrated the efficiency and effectiveness of features using various types of neuroscience data with different sampling frequencies at the level of the single cell, cell population, and coarse-scale brain activity. Our results shed new insights into representing the dynamics of spatiotemporal data using a few fundamental features extracted by wavelet analysis, which may have wide implications to other types of data with rich temporal dynamics.

Revealing Fine Structures of the Retinal Receptive Field by Deep-Learning Networks

Deep convolutional neural networks (CNNs) have demonstrated impressive performance on many visual tasks. Recently, they became useful models for the visual system in neuroscience. However, it is still not clear what is learned by CNNs in terms of neuronal circuits. When a deep CNN with many layers is used for the visual system, it is not easy to compare the structure components of CNNs with possible neuroscience underpinnings due to highly complex circuits from the retina to the higher visual cortex. Here, we address this issue by focusing on single retinal ganglion cells with biophysical models and recording data from animals. By training CNNs with white noise images to predict neuronal responses, we found that fine structures of the retinal receptive field can be revealed. Specifically, convolutional filters learned are resembling biological components of the retinal circuit. This suggests that a CNN learning from one single retinal cell reveals a minimal neural network carried out in this cell. Furthermore, when CNNs learned from different cells are transferred between cells, there is a diversity of transfer learning performance, which indicates that CNNs are cell specific. Moreover, when CNNs are transferred between different types of input images, here white noise versus natural images, transfer learning shows a good performance, which implies that CNNs indeed capture the full computational ability of a single retinal cell for different inputs. Taken together, these results suggest that CNNs could be used to reveal structure components of neuronal circuits, and provide a powerful model for neural system identification.

Dissecting cascade computational components in spiking neural networks

Finding out the physical structure of neuronal circuits that governs neuronal responses is an important goal for brain research. With fast advances for large-scale recording techniques, identification of a neuronal circuit with multiple neurons and stages or layers becomes possible and highly demanding. Although methods for mapping the connection structure of circuits have been greatly developed in recent years, they are mostly limited to simple scenarios of a few neurons in a pairwise fashion; and dissecting dynamical circuits, particularly mapping out a complete functional circuit that converges to a single neuron, is still a challenging question. Here, we show that a recent method, termed spike-triggered non-negative matrix factorization (STNMF), can address these issues. By simulating different scenarios of spiking neural networks with various connections between neurons and stages, we demonstrate that STNMF is a persuasive method to dissect functional connections within a circuit. Using spiking activities recorded at neurons of the output layer, STNMF can obtain a complete circuit consisting of all cascade computational components of presynaptic neurons, as well as their spiking activities. For simulated simple and complex cells of the primary visual cortex, STNMF allows us to dissect the pathway of visual computation. Taken together, these results suggest that STNMF could provide a useful approach for investigating neuronal systems leveraging recorded functional neuronal activity.

It is well known that the computation of neuronal circuits is carried out through the staged and cascade structure of different types of neurons. Nevertheless, the information, particularly sensory information, is processed in a network primarily with feedforward connections through different pathways. A peculiar example is the early visual system, where light is transcoded by the retinal cells, routed by the lateral geniculate nucleus, and reached the primary visual cortex. One meticulous interest in recent years is to map out these physical structures of neuronal pathways. However, most methods so far are limited to taking snapshots of a static view of connections between neurons. It remains unclear how to obtain a functional and dynamical neuronal circuit beyond the simple scenarios of a few randomly sampled neurons. Using simulated spiking neural networks of visual pathways with different scenarios of multiple stages, mixed cell types, and natural image stimuli, we demonstrate that a recent computational tool, named spike-triggered non-negative matrix factorization, can resolve these issues. It enables us to recover the entire structural components of neural networks underlying the computation, together with the functional components of each individual neuron. Utilizing it for complex cells of the primary visual cortex allows us to reveal every underpinning of the nonlinear computation. Our results, together with other recent experimental and computational efforts, show that it is possible to systematically dissect neural circuitry with detailed structural and functional components.

Rapid Analysis of Visual Receptive Fields by Iterative Tomography

Many receptive fields in the early visual system show standard (center-surround) structure and can be analyzed using simple drifting patterns and a difference-of-Gaussians (DoG) model, which treats the receptive field as a linear filter of the visual image. But many other receptive fields show nonlinear properties such as selectivity for direction of movement. Such receptive fields are typically studied using discrete stimuli (moving or flashed bars and edges) and are modelled according to the features of the visual image to which they are most sensitive. Here, we harness recent advances in tomographic image analysis to characterize rapidly and simultaneously both the linear and nonlinear components of visual receptive fields. Spiking and intracellular voltage potential responses to briefly flashed bars are analyzed using non-negative matrix factorization (NNMF) and iterative reconstruction tomography (IRT). The method yields high-resolution receptive field maps of individual neurons and neuron ensembles in primate (marmoset, both sexes) lateral geniculate and rodent (mouse, male) retina. We show that the first two IRT components correspond to DoG-equivalent center and surround of standard [magnocellular (M) and parvocellular (P)] receptive fields in primate geniculate. The first two IRT components also reveal the spatiotemporal receptive field structure of nonstandard (on/off-rectifying) receptive fields. In rodent retina we combine NNMF-IRT with patch-clamp recording and dye injection to directly map spatial receptive fields to the underlying anatomy of retinal output neurons. We conclude that NNMF-IRT provides a rapid and flexible framework for study of receptive fields in the early visual system.

Unraveling neural coding of dynamic natural visual scenes via convolutional recurrent neural networks

Traditional models of retinal system identification analyze the neural response to artificial stimuli using models consisting of predefined components. The model design is limited to prior knowledge, and the artificial stimuli are too simple to be compared with stimuli processed by the retina. To fill in this gap with an explainable model that reveals how a population of neurons work together to encode the larger field of natural scenes, here we used a deep-learning model for identifying the computational elements of the retinal circuit that contribute to learning the dynamics of natural scenes. Experimental results verify that the recurrent connection plays a key role in encoding complex dynamic visual scenes while learning biological computational underpinnings of the retinal circuit. In addition, the proposed models reveal both the shapes and the locations of the spatiotemporal receptive fields of ganglion cells.

Statistical neuroscience in the single trial limit

Individual neurons often produce highly variable responses over nominally identical trials, reflecting a mixture of intrinsic 'noise' and systematic changes in the animal's cognitive and behavioral state. Disentangling these sources of variability is of great scientific interest in its own right, but it is also increasingly inescapable as neuroscientists aspire to study more complex and naturalistic animal behaviors. In these settings, behavioral actions never repeat themselves exactly and may rarely do so even approximately. Thus, new statistical methods that extract reliable features of neural activity using few, if any, repeated trials are needed. Accurate statistical modeling in this severely trial-limited regime is challenging, but still possible if simplifying structure in neural data can be exploited. We review recent works that have identified different forms of simplifying structure - including shared gain modulations across neural subpopulations, temporal smoothness in neural firing rates, and correlations in responses across behavioral conditions - and exploited them to reveal novel insights into the trial-by-trial operation of neural circuits.

The Relationship Between Visual Sensitivity and Eccentricity, Cone Density and Outer Segment Length in the Human Foveola

Purpose

The cellular topography of the human foveola, the central 1° diameter of the fovea, is strikingly non-uniform, with a steep increase of cone photoreceptor density and outer segment (OS) length toward its center. Here, we assessed to what extent the specific cellular organization of the foveola of an individual is reflected in visual sensitivity and if sensitivity peaks at the preferred retinal locus of fixation (PRL).

Methods

Increment sensitivity to small-spot, cone-targeted visual stimuli (1 × 1 arcmin, 543-nm light) was recorded psychophysically in four human participants at 17 locations concentric within a 0.2° diameter on and around the PRL with adaptive optics scanning laser ophthalmoscopy-based microstimulation. Sensitivity test spots were aligned with cell-resolved maps of cone density and cone OS length.

Results

Peak sensitivity was at neither the PRL nor the topographical center of the cone mosaic. Within the central 0.1° diameter, a plateau-like sensitivity profile was observed. Cone density and maximal OS length differed significantly across participants, correlating with their peak sensitivity. Based on these results, biophysical simulation allowed to develop a model of visual sensitivity in the foveola, with distance from the PRL (eccentricity), cone density, and OS length as parameters.

Conclusions

Small-spot sensitivity thresholds in healthy retinas will help to establish the range of normal foveolar function in cell-targeted vision testing. Because of the high reproducibility in replicate testing, threshold variability not explained by our model is assumed to be caused by individual cone and bipolar cell weighting at the specific target locations.

Nonlinear spatial integration in retinal bipolar cells shapes the encoding of artificial and natural stimuli

The retina dissects the visual scene into parallel information channels, which extract specific visual features through nonlinear processing. The first nonlinear stage is typically considered to occur at the output of bipolar cells, resulting from nonlinear transmitter release from synaptic terminals. In contrast, we show here that bipolar cells themselves can act as nonlinear processing elements at the level of their somatic membrane potential. Intracellular recordings from bipolar cells in the salamander retina revealed frequent nonlinear integration of visual signals within bipolar cell receptive field centers, affecting the encoding of artificial and natural stimuli. These nonlinearities provide sensitivity to spatial structure below the scale of bipolar cell receptive fields in both bipolar and downstream ganglion cells and appear to arise at the excitatory input into bipolar cells. Thus, our data suggest that nonlinear signal pooling starts earlier than previously thought: that is, at the input stage of bipolar cells.

•

Some retinal bipolar cells represent visual contrast in a nonlinear fashion

•

These bipolar cells also nonlinearly integrate visual signals over space

•

The spatial nonlinearity affects the encoding of natural stimuli by bipolar cells

•

The nonlinearity results from feedforward input, not from feedback inhibition

Linear and nonlinear chromatic integration in the mouse retina

The computations performed by a neural circuit depend on how it integrates its input signals into an output of its own. In the retina, ganglion cells integrate visual information over time, space, and chromatic channels. Unlike the former two, chromatic integration is largely unexplored. Analogous to classical studies of spatial integration, we here study chromatic integration in mouse retina by identifying chromatic stimuli for which activation from the green or UV color channel is maximally balanced by deactivation through the other color channel. This reveals nonlinear chromatic integration in subsets of On, Off, and On-Off ganglion cells. Unlike the latter two, nonlinear On cells display response suppression rather than activation under balanced chromatic stimulation. Furthermore, nonlinear chromatic integration occurs independently of nonlinear spatial integration, depends on contributions from the rod pathway and on surround inhibition, and may provide information about chromatic boundaries, such as the skyline in natural scenes.

Nonlinear Spatial Integration Underlies the Diversity of Retinal Ganglion Cell Responses to Natural Images

How neurons encode natural stimuli is a fundamental question for sensory neuroscience. In the early visual system, standard encoding models assume that neurons linearly filter incoming stimuli through their receptive fields, but artificial stimuli, such as contrast-reversing gratings, often reveal nonlinear spatial processing. We investigated to what extent such nonlinear processing is relevant for the encoding of natural images in retinal ganglion cells in mice of either sex.

How neurons encode natural stimuli is a fundamental question for sensory neuroscience. In the early visual system, standard encoding models assume that neurons linearly filter incoming stimuli through their receptive fields, but artificial stimuli, such as contrast-reversing gratings, often reveal nonlinear spatial processing. We investigated to what extent such nonlinear processing is relevant for the encoding of natural images in retinal ganglion cells in mice of either sex. We found that standard linear receptive field models yielded good predictions of responses to flashed natural images for a subset of cells but failed to capture the spiking activity for many others. Cells with poor model performance displayed pronounced sensitivity to fine spatial contrast and local signal rectification as the dominant nonlinearity. By contrast, sensitivity to high-frequency contrast-reversing gratings, a classical test for nonlinear spatial integration, was not a good predictor of model performance and thus did not capture the variability of nonlinear spatial integration under natural images. In addition, we also observed a class of nonlinear ganglion cells with inverse tuning for spatial contrast, responding more strongly to spatially homogeneous than to spatially structured stimuli. These findings highlight the diversity of receptive field nonlinearities as a crucial component for understanding early sensory encoding in the context of natural stimuli.

SIGNIFICANCE STATEMENT Experiments with artificial visual stimuli have revealed that many types of retinal ganglion cells pool spatial input signals nonlinearly. However, it is still unclear how relevant this nonlinear spatial integration is when the input signals are natural images. Here we analyze retinal responses to natural scenes in large populations of mouse ganglion cells. We show that nonlinear spatial integration strongly influences responses to natural images for some ganglion cells, but not for others. Cells with nonlinear spatial integration were sensitive to spatial structure inside their receptive fields, and a small group of cells displayed a surprising sensitivity to spatially homogeneous stimuli. Traditional analyses with contrast-reversing gratings did not predict this variability of nonlinear spatial integration under natural images.

Spike-triggered Clustering for Retinal Ganglion Cell Classification

Retinal ganglion cells (RGCs), the retina's output neurons, encode visual information through spiking. The RGC receptive field (RF) represents the basic unit of visual information processing in the retina. RFs are commonly estimated using the spike-triggered average (STA), which is the average of the stimulus patterns to which a given RGC is sensitive. Whereas STA, based on the concept of the average, is simple and intuitive, it leaves more complex structures in the RFs undetected. Alternatively, spike-triggered covariance (STC) analysis provides information on second-order RF statistics. However, STC is computationally cumbersome and difficult to interpret. Thus, the objective of this study was to propose and validate a new computational method, called spike-triggered clustering (STCL), specific for multimodal RFs. Specifically, RFs were fit with a Gaussian mixture model, which provides the means and covariances of multiple RF clusters. The proposed method recovered bipolar stimulus patterns in the RFs of ON-OFF cells, while the STA identified only ON and OFF RGCs, and the remaining RGCs were labeled as unknown types. In contrast, our new STCL analysis distinguished ON-OFF RGCs from the ON, OFF, and unknown RGC types classified by STA. Thus, the proposed method enables us to include ON-OFF RGCs prior to retinal information analysis.

Computational challenges and opportunities for a bi-directional artificial retina

A future artificial retina that can restore high acuity vision in blind people will rely on the capability to both read (observe) and write (control) the spiking activity of neurons using an adaptive, bi-directional and high-resolution device. Although current research is focused on overcoming the technical challenges of building and implanting such a device, exploiting its capabilities to achieve more acute visual perception will also require substantial computational advances. Using high-density large-scale recording and stimulation in the primate retina with an ex vivo multi-electrode array lab prototype, we frame several of the major computational problems, and describe current progress and future opportunities in solving them. First, we identify cell types and locations from spontaneous activity in the blind retina, and then efficiently estimate their visual response properties by using a low-dimensional manifold of inter-retina variability learned from a large experimental dataset. Second, we estimate retinal responses to a large collection of relevant electrical stimuli by passing current patterns through an electrode array, spike sorting the resulting recordings and using the results to develop a model of evoked responses. Third, we reproduce the desired responses for a given visual target by temporally dithering a diverse collection of electrical stimuli within the integration time of the visual system. Together, these novel approaches may substantially enhance artificial vision in a next-generation device.

What the salamander eye has been telling the vision scientist's brain

Salamanders have been habitual residents of research laboratories for more than a century, and their history in science is tightly interwoven with vision research. Nevertheless, many vision scientists - even those working with salamanders - may be unaware of how much our knowledge about vision, and particularly the retina, has been shaped by studying salamanders. In this review, we take a tour through the salamander history in vision science, highlighting the main contributions of salamanders to our understanding of the vertebrate retina. We further point out specificities of the salamander visual system and discuss the perspectives of this animal system for future vision research.

Inference of nonlinear receptive field subunits with spike-triggered clustering

Responses of sensory neurons are often modeled using a weighted combination of rectified linear subunits. Since these subunits often cannot be measured directly, a flexible method is needed to infer their properties from the responses of downstream neurons. We present a method for maximum likelihood estimation of subunits by soft-clustering spike-triggered stimuli, and demonstrate its effectiveness in visual neurons. For parasol retinal ganglion cells in macaque retina, estimated subunits partitioned the receptive field into compact regions, likely representing aggregated bipolar cell inputs. Joint clustering revealed shared subunits between neighboring cells, producing a parsimonious population model. Closed-loop validation, using stimuli lying in the null space of the linear receptive field, revealed stronger nonlinearities in OFF cells than ON cells. Responses to natural images, jittered to emulate fixational eye movements, were accurately predicted by the subunit model. Finally, the generality of the approach was demonstrated in macaque V1 neurons.

New Features of Receptive Fields in Mouse Retina through Spike-triggered Covariance

Retinal ganglion cells (RGCs) encode various spatiotemporal features of visual information into spiking patterns. The receptive field (RF) of each RGC is usually calculated by spike-triggered average (STA), which is fast and easy to understand, but limited to simple and unimodal RFs. As an alternative, spike-triggered covariance (STC) has been proposed to characterize more complex patterns in RFs. This study compares STA and STC for the characterization of RFs and demonstrates that STC has an advantage over STA for identifying novel spatiotemporal features of RFs in mouse RGCs. We first classified mouse RGCs into ON, OFF, and ON/OFF cells according to their response to full-field light stimulus, and then investigated the spatiotemporal patterns of RFs with random checkerboard stimulation, using both STA and STC analysis. We propose five sub-types (T1–T5) in the STC of mouse RGCs together with their physiological implications. In particular, the relatively slow biphasic pattern (T1) could be related to excitatory inputs from bipolar cells. The transient biphasic pattern (T2) allows one to characterize complex patterns in RFs of ON/OFF cells. The other patterns (T3–T5), which are contrasting, alternating, and monophasic patterns, could be related to inhibitory inputs from amacrine cells. Thus, combining STA and STC and considering the proposed sub-types unveil novel characteristics of RFs in the mouse retina and offer a more holistic understanding of the neural coding mechanisms of mouse RGCs.

Inferring synaptic inputs from spikes with a conductance-based neural encoding model

Descriptive statistical models of neural responses generally aim to characterize the mapping from stimuli to spike responses while ignoring biophysical details of the encoding process. Here, we introduce an alternative approach, the conductance-based encoding model (CBEM), which describes a mapping from stimuli to excitatory and inhibitory synaptic conductances governing the dynamics of sub-threshold membrane potential. Remarkably, we show that the CBEM can be fit to extracellular spike train data and then used to predict excitatory and inhibitory synaptic currents. We validate these predictions with intracellular recordings from macaque retinal ganglion cells. Moreover, we offer a novel quasi-biophysical interpretation of the Poisson generalized linear model (GLM) as a special case of the CBEM in which excitation and inhibition are perfectly balanced. This work forges a new link between statistical and biophysical models of neural encoding and sheds new light on the biophysical variables that underlie spiking in the early visual pathway.

Data-Driven Approaches to Understanding Visual Neuron Activity

With modern neurophysiological methods able to record neural activity throughout the visual pathway in the context of arbitrarily complex visual stimulation, our understanding of visual system function is becoming limited by the available models of visual neurons that can be directly related to such data. Different forms of statistical models are now being used to probe the cellular and circuit mechanisms shaping neural activity, understand how neural selectivity to complex visual features is computed, and derive the ways in which neurons contribute to systems-level visual processing. However, models that are able to more accurately reproduce observed neural activity often defy simple interpretations. As a result, rather than being used solely to connect with existing theories of visual processing, statistical modeling will increasingly drive the evolution of more sophisticated theories.

Unusual Physiological Properties of Smooth Monostratified Ganglion Cell Types in Primate Retina

The functions of the diverse retinal ganglion cell types in primates and the parallel visual pathways they initiate remain poorly understood. Here, unusual physiological and computational properties of the ON and OFF smooth monostratified ganglion cells are explored. Large-scale multi-electrode recordings from 48 macaque retinas revealed that these cells exhibit irregular receptive field structure composed of spatially segregated hotspots, quite different from the classic center-surround model of retinal receptive fields. Surprisingly, visual stimulation of different hotspots in the same cell produced spikes with subtly different spatiotemporal voltage signatures, consistent with a dendritic contribution to hotspot structure. Targeted visual stimulation and computational inference demonstrated strong nonlinear subunit properties associated with each hotspot, supporting a model in which the hotspots apply nonlinearities at a larger spatial scale than bipolar cells. These findings reveal a previously unreported nonlinear mechanism in the output of the primate retina that contributes to signaling spatial information.

Functional characterization of retinal ganglion cells using tailored nonlinear modeling

The mammalian retina encodes the visual world in action potentials generated by 20-50 functionally and anatomically-distinct types of retinal ganglion cell (RGC). Individual RGC types receive synaptic input from distinct presynaptic circuits; therefore, their responsiveness to specific features in the visual scene arises from the information encoded in synaptic input and shaped by postsynaptic signal integration and spike generation. Unfortunately, there is a dearth of tools for characterizing the computations reflected in RGC spike output. Therefore, we developed a statistical model, the separable Nonlinear Input Model, to characterize the excitatory and suppressive components of RGC receptive fields. We recorded RGC responses to a correlated noise ("cloud") stimulus in an in vitro preparation of mouse retina and found that our model accurately predicted RGC responses at high spatiotemporal resolution. It identified multiple receptive fields reflecting the main excitatory and suppressive components of the response of each neuron. Significantly, our model accurately identified ON-OFF cells and distinguished their distinct ON and OFF receptive fields, and it demonstrated a diversity of suppressive receptive fields in the RGC population. In total, our method offers a rich description of RGC computation and sets a foundation for relating it to retinal circuitry.

Neural correlates of sparse coding and dimensionality reduction

Supported by recent computational studies, there is increasing evidence that a wide range of neuronal responses can be understood as an emergent property of nonnegative sparse coding (NSC), an efficient population coding scheme based on dimensionality reduction and sparsity constraints. We review evidence that NSC might be employed by sensory areas to efficiently encode external stimulus spaces, by some associative areas to conjunctively represent multiple behaviorally relevant variables, and possibly by the basal ganglia to coordinate movement. In addition, NSC might provide a useful theoretical framework under which to understand the often complex and nonintuitive response properties of neurons in other brain areas. Although NSC might not apply to all brain areas (for example, motor or executive function areas) the success of NSC-based models, especially in sensory areas, warrants further investigation for neural correlates in other regions.

Probing Computation in the Primate Visual System at Single-Cone Resolution

Daylight vision begins when light activates cone photoreceptors in the retina, creating spatial patterns of neural activity. These cone signals are then combined and processed in downstream neural circuits, ultimately producing visual perception. Recent technical advances have made it possible to deliver visual stimuli to the retina that probe this processing by the visual system at its elementary resolution of individual cones. Physiological recordings from nonhuman primate retinas reveal the spatial organization of cone signals in retinal ganglion cells, including how signals from cones of different types are combined to support both spatial and color vision. Psychophysical experiments with human subjects characterize the visual sensations evoked by stimulating a single cone, including the perception of color. Future combined physiological and psychophysical experiments focusing on probing the elementary visual inputs are likely to clarify how neural processing generates our perception of the visual world.

Activity Correlations between Direction-Selective Retinal Ganglion Cells Synergistically Enhance Motion Decoding from Complex Visual Scenes

Neurons in sensory systems are often tuned to particular stimulus features. During complex naturalistic stimulation, however, multiple features may simultaneously affect neuronal responses, which complicates the readout of individual features. To investigate feature representation under complex stimulation, we studied how direction-selective ganglion cells in salamander retina respond to texture motion where direction, velocity, and spatial pattern inside the receptive field continuously change. We found that the cells preserve their direction preference under this stimulation, yet their direction encoding becomes ambiguous due to simultaneous activation by luminance changes. The ambiguities can be resolved by considering populations of direction-selective cells with different preferred directions. This gives rise to synergistic motion decoding, yielding more information from the population than the summed information from single-cell responses. Strong positive response correlations between cells with different preferred directions amplify this synergy. Our results show how correlated population activity can enhance feature extraction in complex visual scenes.

•

Direction-selective ganglion cells respond to motion as well as luminance changes

•

This obscures the readout of direction from single cells under complex texture motion

•

Population decoding improves direction readout supralinearly over individual cells

•

Strong spike correlations further enhance readout through increased synergy

The dynamic receptive fields of retinal ganglion cells

Retinal ganglion cells (RGCs) were one of the first classes of sensory neurons to be described in terms of a receptive field (RF). Over the last six decades, our understanding of the diversity of RGC types and the nuances of their response properties has grown exponentially. We will review the current understanding of RGC RFs mostly from studies in mammals, but including work from other vertebrates as well. We will argue for a new paradigm that embraces the fluidity of RGC RFs with an eye toward the neuroethology of vision. Specifically, we will focus on (1) different methods for measuring RGC RFs, (2) RF models, (3) feature selectivity and the distinction between fluid and stable RF properties, and (4) ideas about the future of understanding RGC RFs.

Spatial summation in the human fovea: Do normal optical aberrations and fixational eye movements have an effect?

Psychophysical inferences about the neural mechanisms supporting spatial vision can be undermined by uncertainties introduced by optical aberrations and fixational eye movements, particularly in fovea where the neuronal grain of the visual system is fine. We examined the effect of these preneural factors on photopic spatial summation in the human fovea using a custom adaptive optics scanning light ophthalmoscope that provided control over optical aberrations and retinal stimulus motion. Consistent with previous results, Ricco's area of complete summation encompassed multiple photoreceptors when measured with ordinary amounts of ocular aberrations and retinal stimulus motion. When both factors were minimized experimentally, summation areas were essentially unchanged, suggesting that foveal spatial summation is limited by postreceptoral neural pooling. We compared our behavioral data to predictions generated with a physiologically-inspired front-end model of the visual system, and were able to capture the shape of the summation curves obtained with and without pre-retinal factors using a single postreceptoral summing filter of fixed spatial extent. Given our data and modeling, neurons in the magnocellular visual pathway, such as parasol ganglion cells, provide a candidate neural correlate of Ricco's area in the central fovea.

Inferring hidden structure in multilayered neural circuits

A central challenge in sensory neuroscience involves understanding how neural circuits shape computations across cascaded cell layers. Here we attempt to reconstruct the response properties of experimentally unobserved neurons in the interior of a multilayered neural circuit, using cascaded linear-nonlinear (LN-LN) models. We combine non-smooth regularization with proximal consensus algorithms to overcome difficulties in fitting such models that arise from the high dimensionality of their parameter space. We apply this framework to retinal ganglion cell processing, learning LN-LN models of retinal circuitry consisting of thousands of parameters, using 40 minutes of responses to white noise. Our models demonstrate a 53% improvement in predicting ganglion cell spikes over classical linear-nonlinear (LN) models. Internal nonlinear subunits of the model match properties of retinal bipolar cells in both receptive field structure and number. Subunits have consistently high thresholds, supressing all but a small fraction of inputs, leading to sparse activity patterns in which only one subunit drives ganglion cell spiking at any time. From the model’s parameters, we predict that the removal of visual redundancies through stimulus decorrelation across space, a central tenet of efficient coding theory, originates primarily from bipolar cell synapses. Furthermore, the composite nonlinear computation performed by retinal circuitry corresponds to a boolean OR function applied to bipolar cell feature detectors. Our methods are statistically and computationally efficient, enabling us to rapidly learn hierarchical non-linear models as well as efficiently compute widely used descriptive statistics such as the spike triggered average (STA) and covariance (STC) for high dimensional stimuli. This general computational framework may aid in extracting principles of nonlinear hierarchical sensory processing across diverse modalities from limited data.

Computation in neural circuits arises from the cascaded processing of inputs through multiple cell layers. Each of these cell layers performs operations such as filtering and thresholding in order to shape a circuit’s output. It remains a challenge to describe both the computations and the mechanisms that mediate them given limited data recorded from a neural circuit. A standard approach to describing circuit computation involves building quantitative encoding models that predict the circuit response given its input, but these often fail to map in an interpretable way onto mechanisms within the circuit. In this work, we build two layer linear-nonlinear cascade models (LN-LN) in order to describe how the retinal output is shaped by nonlinear mechanisms in the inner retina. We find that these LN-LN models, fit to ganglion cell recordings alone, identify filters and nonlinearities that are readily mapped onto individual circuit components inside the retina, namely bipolar cells and the bipolar-to-ganglion cell synaptic threshold. This work demonstrates how combining simple prior knowledge of circuit properties with partial experimental recordings of a neural circuit’s output can yield interpretable models of the entire circuit computation, including parts of the circuit that are hidden or not directly observed in neural recordings.