Synaptic Rectification Controls Nonlinear Spatial Integration of Natural Visual Inputs

Evolution: Non-linear spatial integration for high-acuity vision

Visual systems evolve to align with species-specific behavior and ecology. By comparing closely related murid rodents, a new study demonstrates that changes in cell-type proportions enable distinct neural computations for high-acuity vision in diurnal species.

Asymmetries in the Architecture of ON and OFF Arbors in ON-OFF Direction-Selective Ganglion Cells

Direction selectivity is a fundamental feature in the visual system. In the retina, direction selectivity is independently computed by ON and OFF circuits. However, the advantages of extracting directional information from these two independent circuits are unclear. To gain insights, we examined the ON-OFF direction-selective ganglion cells (DSGCs), which recombine signals from both circuits. Specifically, we investigated the dendritic architecture of these neurons with the premise that asymmetries in architecture will provide insights into function. Scrutinizing the dendrites of dye-filled ON-OFF DSGCs reveals that the OFF arbors of these neurons are substantially denser. The increase in density can be primarily attributed to the higher branching seen in OFF arbors. Further, analysis of ON-OFF DSGCs in a previously published serial block-face electron microscopy dataset revealed that the denser OFF arbors packed more bipolar synapses per unit dendritic length. These asymmetries in the dendritic architecture suggest that the ON-OFF DSGC preferentially magnifies the synaptic drive of the OFF pathway, potentially allowing it to encode information distinct from the ON pathway.

Nonlinear receptive fields evoke redundant retinal coding of natural scenes

The role of the vertebrate retina in early vision is generally described by the efficient coding hypothesis1,2, which predicts that the retina reduces the redundancy inherent in natural scenes3 by discarding spatiotemporal correlations while preserving stimulus information4. It is unclear, however, whether the predicted decorrelation and redundancy reduction in the activity of ganglion cells, the retina's output neurons, hold under gaze shifts, which dominate the dynamics of the natural visual input5. We show here that species-specific gaze patterns in natural stimuli can drive correlated spiking responses both in and across distinct types of ganglion cells in marmoset as well as mouse retina. These concerted responses disrupt redundancy reduction to signal fixation periods with locally high spatial contrast. Model-based analyses of ganglion cell responses to natural stimuli show that the observed response correlations follow from nonlinear pooling of ganglion cell inputs. Our results indicate cell-type-specific deviations from efficient coding in retinal processing of natural gaze shifts.

Asymmetric Activation of Retinal ON and OFF Pathways by AOSLO Raster-Scanned Visual Stimuli

Adaptive optics scanning light ophthalmoscopy (AOSLO) enables high-resolution retinal imaging, eye tracking, and stimulus delivery in the living eye. AOSLO-mediated visual stimuli are created by temporally modulating the excitation light as it scans across the retina. As a result, each location within the field of view receives a brief flash of light during each scanner cycle (every 33-40 ms). Here we used in vivo calcium imaging with AOSLO to investigate the impact of this intermittent stimulation on the retinal ON and OFF pathways. Raster-scanned backgrounds exaggerated existing ON-OFF pathway asymmetries leading to high baseline activity in ON cells and increased response rectification in OFF cells.

Functional diversity in the output of the primate retina

The visual image transmitted by the retina to the brain has long been understood in terms of spatial filtering by the center-surround receptive fields of retinal ganglion cells (RGCs). Recently, this textbook view has been challenged by the stunning functional diversity and specificity observed in ∼40 distinct RGC types in the mouse retina. However, it is unclear whether the ∼20 morphologically and molecularly identified RGC types in primates exhibit similar functional diversity, or instead exhibit center-surround organization at different spatial scales. Here, we reveal striking and surprising functional diversity in macaque and human RGC types using large-scale multi-electrode recordings from isolated macaque and human retinas. In addition to the five well-known primate RGC types, 18-27 types were distinguished by their functional properties, likely revealing several previously unknown types. Surprisingly, many of these cell types exhibited striking non-classical receptive field structure, including irregular spatial and chromatic properties not previously reported in any species. Qualitatively similar results were observed in recordings from the human retina. The receptive fields of less-understood RGC types formed uniform mosaics covering visual space, confirming their classification, and the morphological counterparts of two types were established using single-cell recording. The striking receptive field diversity was paralleled by distinctive responses to natural movies and complexity of visual computation. These findings suggest that diverse RGC types, rather than merely filtering the scene at different spatial scales, instead play specialized roles in human vision.

Regional tuning of photoreceptor adaptation in the primate retina

Adaptation in cone photoreceptors allows our visual system to effectively operate over an enormous range of light intensities. However, little is known about the properties of cone adaptation in the specialized region of the primate central retina called the fovea, which is densely packed with cones and mediates high-acuity central vision. Here we show that macaque foveal cones exhibit weaker and slower luminance adaptation compared to cones in the peripheral retina. We find that this difference in adaptive properties between foveal and peripheral cones is due to differences in the magnitude of a hyperpolarization-activated current, Ih. This Ih current regulates the strength and time course of luminance adaptation in peripheral cones where it is more prominent than in foveal cones. A weaker and slower adaptation in foveal cones helps maintain a higher sensitivity for a longer duration which may be well-suited for maximizing the collection of high-acuity information at the fovea during gaze fixation between rapid eye movements.

Applying Super-Resolution and Tomography Concepts to Identify Receptive Field Subunits in the Retina

Spatially nonlinear stimulus integration by retinal ganglion cells lies at the heart of various computations performed by the retina. It arises from the nonlinear transmission of signals that ganglion cells receive from bipolar cells, which thereby constitute functional subunits within a ganglion cell's receptive field. Inferring these subunits from recorded ganglion cell activity promises a new avenue for studying the functional architecture of the retina. This calls for efficient methods, which leave sufficient experimental time to leverage the acquired knowledge for further investigating identified subunits. Here, we combine concepts from super-resolution microscopy and computed tomography and introduce super-resolved tomographic reconstruction (STR) as a technique to efficiently stimulate and locate receptive field subunits. Simulations demonstrate that this approach can reliably identify subunits across a wide range of model variations, and application in recordings of primate parasol ganglion cells validates the experimental feasibility. STR can potentially reveal comprehensive subunit layouts within only a few tens of minutes of recording time, making it ideal for online analysis and closed-loop investigations of receptive field substructure in retina recordings.

Spike desensitisation as a mechanism for high-contrast selectivity in retinal ganglion cells

In the vertebrate retina, several dozens of parallel channels relay information about the visual world to the brain. These channels are represented by the different types of retinal ganglion cells (RGCs), whose responses are rendered selective for distinct sets of visual features by various mechanisms. These mechanisms can be roughly grouped into synaptic interactions and cell-intrinsic mechanisms, with the latter including dendritic morphology as well as ion channel complement and distribution. Here, we investigate how strongly ion channel complement can shape RGC output by comparing two mouse RGC types, the well-described ON alpha cell and a little-studied ON cell that is EGFP-labelled in the Igfbp5 mouse line and displays an unusual selectivity for stimuli with high contrast. Using patch-clamp recordings and computational modelling, we show that a higher activation threshold and a pronounced slow inactivation of the voltage-gated Na+ channels contribute to the distinct contrast tuning and transient responses in ON Igfbp5 RGCs, respectively. In contrast, such a mechanism could not be observed in ON alpha cells. This study provides an example for the powerful role that the last stage of retinal processing can play in shaping RGC responses.

Synaptic Origins of the Complex Receptive Field Structure in Primate Smooth Monostratified Retinal Ganglion Cells

Considerable progress has been made in studying the receptive fields of the most common primate retinal ganglion cell (RGC) types, such as parasol RGCs. Much less is known about the rarer primate RGC types and the circuitry that gives rise to noncanonical receptive field structures. The goal of this study was to analyze synaptic inputs to smooth monostratified RGCs to determine the origins of their complex spatial receptive fields, which contain isolated regions of high sensitivity called "hotspots." Interestingly, smooth monostratified RGCs co-stratify with the well-studied parasol RGCs and are thus constrained to receiving input from bipolar and amacrine cells with processes sharing the same layer, raising the question of how their functional differences originate. Through 3D reconstructions of circuitry and synapses onto ON smooth monostratified and ON parasol RGCs from central macaque retina, we identified four distinct sampling strategies employed by smooth and parasol RGCs to extract diverse response properties from co-stratifying bipolar and amacrine cells. The two RGC types differed in the proportion of amacrine cell input, relative contributions of co-stratifying bipolar cell types, amount of synaptic input per bipolar cell, and spatial distribution of bipolar cell synapses. Our results indicate that the smooth RGC's complex receptive field structure arises through spatial asymmetries in excitatory bipolar cell input which formed several discrete clusters comparable with physiologically measured hotspots. Taken together, our results demonstrate how the striking differences between ON parasol and ON smooth monostratified RGCs arise from distinct strategies for sampling a common set of synaptic inputs.

Two Sides of the Same Coin: Efficient and Predictive Neural Coding

Some visual properties are consistent across a wide range of environments, while other properties are more labile. The efficient coding hypothesis states that many of these regularities in the environment can be discarded from neural representations, thus allocating more of the brain's dynamic range to properties that are likely to vary. This paradigm is less clear about how the visual system prioritizes different pieces of information that vary across visual environments. One solution is to prioritize information that can be used to predict future events, particularly those that guide behavior. The relationship between the efficient coding and future prediction paradigms is an area of active investigation. In this review, we argue that these paradigms are complementary and often act on distinct components of the visual input. We also discuss how normative approaches to efficient coding and future prediction can be integrated.

Diversity of Ganglion Cell Responses to Saccade-Like Image Shifts in the Primate Retina

Saccades are a fundamental part of natural vision. They interrupt fixations of the visual gaze and rapidly shift the image that falls onto the retina. These stimulus dynamics can cause activation or suppression of different retinal ganglion cells, but how they affect the encoding of visual information in different types of ganglion cells is largely unknown. Here, we recorded spiking responses to saccade-like shifts of luminance gratings from ganglion cells in isolated marmoset retinas and investigated how the activity depended on the combination of presaccadic and postsaccadic images. All identified cell types, On and Off parasol and midget cells, as well as a type of Large Off cells, displayed distinct response patterns, including particular sensitivity to either the presaccadic or the postsaccadic image or combinations thereof. In addition, Off parasol and Large Off cells, but not On cells, showed pronounced sensitivity to whether the image changed across the transition. Stimulus sensitivity of On cells could be explained based on their responses to step changes in light intensity, whereas Off cells, in particular, parasol and the Large Off cells, seem to be affected by additional interactions that are not triggered during simple light-intensity flashes. Together, our data show that ganglion cells in the primate retina are sensitive to different combinations of presaccadic and postsaccadic visual stimuli. This contributes to the functional diversity of the output signals of the retina and to asymmetries between On and Off pathways and provides evidence of signal processing beyond what is triggered by isolated steps in light intensity.SIGNIFICANCE STATEMENT Sudden eye movements (saccades) shift our direction of gaze, bringing new images in focus on our retinas. To study how retinal neurons deal with these rapid image transitions, we recorded spiking activity from ganglion cells, the output neurons of the retina, in isolated retinas of marmoset monkeys while shifting a projected image in a saccade-like fashion across the retina. We found that the cells do not just respond to the newly fixated image, but that different types of ganglion cells display different sensitivities to the presaccadic and postsaccadic stimulus patterns. Certain Off cells, for example, are sensitive to changes in the image across transitions, which contributes to differences between On and Off information channels and extends the range of encoded stimulus features.

Systematic reduction of the dimensionality of natural scenes allows accurate predictions of retinal ganglion cell spike outputs

The mammalian retina engages a broad array of linear and nonlinear circuit mechanisms to convert natural scenes into retinal ganglion cell (RGC) spike outputs. Although many individual integration mechanisms are well understood, we know less about how multiple mechanisms interact to encode the complex spatial features present in natural inputs. Here, we identified key spatial features in natural scenes that shape encoding by primate parasol RGCs. Our approach identified simplifications in the spatial structure of natural scenes that minimally altered RGC spike responses. We observed that reducing natural movies into 16 linearly integrated regions described ∼80% of the structure of parasol RGC spike responses; this performance depended on the number of regions but not their precise spatial locations. We used simplified stimuli to design high-dimensional metamers that recapitulated responses to naturalistic movies. Finally, we modeled the retinal computations that convert flashed natural images into one-dimensional spike counts.

Calcium-permeable AMPA receptors on AII amacrine cells mediate sustained signaling in the On-pathway of the primate retina

Midget and parasol ganglion cells (GCs) represent the major output channels from the primate eye to the brain. On-type midget and parasol GCs exhibit a higher background spike rate and thus can respond more linearly to contrast changes than their Off-type counterparts. Here, we show that a calcium-permeable AMPA receptor (CP-AMPAR) antagonist blocks background spiking and sustained light-evoked firing in On-type GCs while preserving transient light responses. These effects are selective for On-GCs and are occluded by a gap-junction blocker suggesting involvement of AII amacrine cells (AII-ACs). Direct recordings from AII-ACs, cobalt uptake experiments, and analyses of transcriptomic data confirm that CP-AMPARs are expressed by primate AII-ACs. Overall, our data demonstrate that under some background light levels, CP-AMPARs at the rod bipolar to AII-AC synapse drive sustained signaling in On-type GCs and thus contribute to the more linear contrast signaling of the primate On- versus Off-pathway.

Retinal Encoding of Natural Scenes

An ultimate goal in retina science is to understand how the neural circuit of the retina processes natural visual scenes. Yet most studies in laboratories have long been performed with simple, artificial visual stimuli such as full-field illumination, spots of light, or gratings. The underlying assumption is that the features of the retina thus identified carry over to the more complex scenario of natural scenes. As the application of corresponding natural settings is becoming more commonplace in experimental investigations, this assumption is being put to the test and opportunities arise to discover processing features that are triggered by specific aspects of natural scenes. Here, we review how natural stimuli have been used to probe, refine, and complement knowledge accumulated under simplified stimuli, and we discuss challenges and opportunities along the way toward a comprehensive understanding of the encoding of natural scenes. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Conserved circuits for direction selectivity in the primate retina

The detection of motion direction is a fundamental visual function and a classic model for neural computation. In the non-primate retina, direction selectivity arises in starburst amacrine cell (SAC) dendrites, which provide selective inhibition to direction-selective retinal ganglion cells (dsRGCs). Although SACs are present in primates, their connectivity and the existence of dsRGCs remain open questions. Here, we present a connectomic reconstruction of the primate ON SAC circuit from a serial electron microscopy volume of the macaque central retina. We show that the structural basis for the SACs' ability to confer directional selectivity on postsynaptic neurons is conserved. SACs selectively target a candidate homolog to the mammalian ON-sustained dsRGCs that project to the accessory optic system (AOS) and contribute to gaze-stabilizing reflexes. These results indicate that the capacity to compute motion direction is present in the retina, which is earlier in the primate visual system than classically thought.

Adaptation in cone photoreceptors contributes to an unexpected insensitivity of primate On parasol retinal ganglion cells to spatial structure in natural images

Neural circuits are constructed from nonlinear building blocks, and not surprisingly overall circuit behavior is often strongly nonlinear. But neural circuits can also behave near linearly, and some circuits shift from linear to nonlinear behavior depending on stimulus conditions. Such control of nonlinear circuit behavior is fundamental to neural computation. Here, we study a surprising stimulus dependence of the responses of macaque On (but not Off) parasol retinal ganglion cells: these cells respond nonlinearly to spatial structure in some stimuli but near linearly to spatial structure in others, including natural inputs. We show that these differences in the linearity of the integration of spatial inputs can be explained by a shift in the balance of excitatory and inhibitory synaptic inputs that originates at least partially from adaptation in the cone photoreceptors. More generally, this highlights how subtle asymmetries in signaling - here in the cone signals - can qualitatively alter circuit computation.

Simple model for encoding natural images by retinal ganglion cells with nonlinear spatial integration

A central goal in sensory neuroscience is to understand the neuronal signal processing involved in the encoding of natural stimuli. A critical step towards this goal is the development of successful computational encoding models. For ganglion cells in the vertebrate retina, the development of satisfactory models for responses to natural visual scenes is an ongoing challenge. Standard models typically apply linear integration of visual stimuli over space, yet many ganglion cells are known to show nonlinear spatial integration, in particular when stimulated with contrast-reversing gratings. We here study the influence of spatial nonlinearities in the encoding of natural images by ganglion cells, using multielectrode-array recordings from isolated salamander and mouse retinas. We assess how responses to natural images depend on first- and second-order statistics of spatial patterns inside the receptive field. This leads us to a simple extension of current standard ganglion cell models. We show that taking not only the weighted average of light intensity inside the receptive field into account but also its variance over space can partly account for nonlinear integration and substantially improve response predictions of responses to novel images. For salamander ganglion cells, we find that response predictions for cell classes with large receptive fields profit most from including spatial contrast information. Finally, we demonstrate how this model framework can be used to assess the spatial scale of nonlinear integration. Our results underscore that nonlinear spatial stimulus integration translates to stimulation with natural images. Furthermore, the introduced model framework provides a simple, yet powerful extension of standard models and may serve as a benchmark for the development of more detailed models of the nonlinear structure of receptive fields.

For understanding how sensory systems operate in the natural environment, an important goal is to develop models that capture neuronal responses to natural stimuli. For retinal ganglion cells, which connect the eye to the brain, current standard models often fail to capture responses to natural visual scenes. This shortcoming is at least partly rooted in the fact that ganglion cells may combine visual signals over space in a nonlinear fashion. We here show that a simple model, which not only considers the average light intensity inside a cell’s receptive field but also the variance of light intensity over space, can partly account for these nonlinearities and thereby improve current standard models. This provides an easy-to-obtain benchmark for modeling ganglion cell responses to natural images.

ON and OFF Signaling Pathways in the Retina and the Visual System

Visual processing starts at the retina of the eye, and signals are then transferred primarily to the visual cortex and the tectum. In the retina, multiple neural networks encode different aspects of visual input, such as color and motion. Subsequently, multiple neural streams in parallel convey unique aspects of visual information to cortical and subcortical regions. Bipolar cells, which are the second order neurons of the retina, separate visual signals evoked by light and dark contrasts and encode them to ON and OFF pathways, respectively. The interplay between ON and OFF neural signals is the foundation for visual processing for object contrast which underlies higher order stimulus processing. ON and OFF pathways have been classically thought to signal in a mirror-symmetric manner. However, while these two pathways contribute synergistically to visual perception in some instances, they have pronounced asymmetries suggesting independent operation in other cases. In this review, we summarize the role of the ON-OFF dichotomy in visual signaling, aiming to contribute to the understanding of visual recognition.

Morphology, Molecular Characterization, and Connections of Ganglion Cells in Primate Retina

The eye sends information about the visual world to the brain on over 20 parallel signal pathways, each specialized to signal features such as spectral reflection (color), edges, and motion of objects in the environment. Each pathway is formed by the axons of a separate type of retinal output neuron (retinal ganglion cell). In this review, we summarize what is known about the excitatory retinal inputs, brain targets, and gene expression patterns of ganglion cells in humans and nonhuman primates. We describe how most ganglion cell types receive their input from only one or two of the 11 types of cone bipolar cell and project selectively to only one or two target regions in the brain. We also highlight how genetic methods are providing tools to characterize ganglion cells and establish cross-species homologies.

Predictive encoding of motion begins in the primate retina

Predictive motion encoding is an important aspect of visually guided behavior that allows animals to estimate the trajectory of moving objects. Motion prediction is understood primarily in the context of translational motion, but the environment contains other types of behaviorally salient motion correlation such as those produced by approaching or receding objects. However, the neural mechanisms that detect and predictively encode these correlations remain unclear. We report here that four of the parallel output pathways in the primate retina encode predictive motion information, and this encoding occurs for several classes of spatiotemporal correlation that are found in natural vision. Such predictive coding can be explained by known nonlinear circuit mechanisms that produce a nearly optimal encoding, with transmitted information approaching the theoretical limit imposed by the stimulus itself. Thus, these neural circuit mechanisms efficiently separate predictive information from nonpredictive information during the encoding process.

Stimulus Contrast Affects Spatial Integration in the Lateral Geniculate Nucleus of Macaque Monkeys

Gain-control mechanisms adjust neuronal responses to accommodate the wide range of stimulus conditions in the natural environment. Contrast gain control and extraclassical surround suppression are two manifestations of gain control that govern the responses of neurons in the early visual system. Understanding how these two forms of gain control interact has important implications for the detection and discrimination of stimuli across a range of contrast conditions. Here, we report that stimulus contrast affects spatial integration in the lateral geniculate nucleus of alert macaque monkeys (male and female), whereby neurons exhibit a reduction in the strength of extraclassical surround suppression and an expansion in the preferred stimulus size with low-contrast stimuli compared with high-contrast stimuli. Effects were greater for magnocellular neurons than for parvocellular neurons, indicating stream-specific interactions between stimulus contrast and stimulus size. Within the magnocellular pathway, contrast-dependent effects were comparable for ON-center and OFF-center neurons, despite ON neurons having larger receptive fields, less pronounced surround suppression, and more pronounced contrast gain control than OFF neurons. Together, these findings suggest that the parallel streams delivering visual information from retina to primary visual cortex, serve not only to broaden the range of signals delivered to cortex, but also to provide a substrate for differential interactions between stimulus contrast and stimulus size that may serve to improve stimulus detection and stimulus discrimination under pathway-specific lower and higher contrast conditions, respectively.SIGNIFICANCE STATEMENT Stimulus contrast is a salient feature of visual scenes. Here we examine the influence of stimulus contrast on spatial integration in the lateral geniculate nucleus (LGN). Our results demonstrate that increases in contrast generally increase extraclassical suppression and decrease the size of optimal stimuli, indicating a reduction in the extent of visual space from which LGN neurons integrate signals. Differences between magnocellular and parvocellular neurons are noteworthy and further demonstrate that the feedforward parallel pathways to cortex increase the range of information conveyed for downstream cortical processing, a range broadened by diversity in the ON and OFF pathways. These results have important implications for more complex visual processing that underly the detection and discrimination of stimuli under varying natural conditions.

Identification of Multiple Noise Sources Improves Estimation of Neural Responses across Stimulus Conditions

Most models of neural responses are constructed to reproduce the average response to inputs but lack the flexibility to capture observed variability in responses. The origins and structure of this variability have significant implications for how information is encoded and processed in the nervous system, both by limiting information that can be conveyed and by determining processing strategies that are favorable for minimizing its negative effects. Here, we present a new modeling framework that incorporates multiple sources of noise to better capture observed features of neural response variability across stimulus conditions. We apply this model to retinal ganglion cells at two different ambient light levels and demonstrate that it captures the full distribution of responses. Further, the model reveals light level-dependent changes that could not be seen with previous models, showing both large changes in rectification of nonlinear circuit elements and systematic differences in the contributions of different noise sources under different conditions.

Matching visual induction effects on screens of different size

In the film industry, the same movie is expected to be watched on displays of vastly different sizes, from cinema screens to mobile phones. But visual induction, the perceptual phenomenon by which the appearance of a scene region is affected by its surroundings, will be different for the same image shown on two displays of different dimensions. This phenomenon presents a practical challenge for the preservation of the artistic intentions of filmmakers, because it can lead to shifts in image appearance between viewing destinations. In this work, we show that a neural field model based on the efficient representation principle is able to predict induction effects and how, by regularizing its associated energy functional, the model is still able to represent induction but is now invertible. From this finding, we propose a method to preprocess an image in a screen-size dependent way so that its perception, in terms of visual induction, may remain constant across displays of different size. The potential of the method is demonstrated through psychophysical experiments on synthetic images and qualitative examples on natural images.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

A quantitative description of macaque ganglion cell responses to natural scenes: the interplay of time and space

KEY POINTS

Responses to natural scenes are the business of the retina. We find primate ganglion cell responses to such scenes consistent with those to simpler stimuli. A biophysical model confirmed this and predicted ganglion cell responses with close to retinal reliability. Primate ganglion cell responses to natural scenes were driven by temporal variations in colour and luminance over the receptive field centre caused by eye movements, and little influenced by interaction of centre and surround with structure in the scene. We discuss implications in the context of efficient coding of the visual environment. Much information in a higher spatiotemporal frequency band is concentrated in the magnocellular pathway.

ABSTRACT

Responses of visual neurons to natural scenes provide a link between classical descriptions of receptive field structure and visual perception of the natural environment. A natural scene video with a movement pattern resembling that of primate eye movements was used to evoke responses from macaque ganglion cells. Cell responses were well described through known properties of cell receptive fields. Different analyses converge to show that responses primarily derive from the temporal pattern of stimulation derived from eye movements, rather than spatial receptive field structure beyond centre size and position. This was confirmed using a model that predicted ganglion cell responses close to retinal reliability, with only a small contribution of the surround relative to the centre. We also found that the spatiotemporal spectrum of the stimulus is modified in ganglion cell responses, and this can reduce redundancy in the retinal signal. This is more pronounced in the magnocellular pathway, which is much better suited to transmit the detailed structure of natural scenes than the parvocellular pathway. Whitening is less important for chromatic channels. Taken together, this shows how a complex interplay across space, time and spectral content sculpts ganglion cell responses.

Nonlinear spatial integration in retinal bipolar cells shapes the encoding of artificial and natural stimuli

The retina dissects the visual scene into parallel information channels, which extract specific visual features through nonlinear processing. The first nonlinear stage is typically considered to occur at the output of bipolar cells, resulting from nonlinear transmitter release from synaptic terminals. In contrast, we show here that bipolar cells themselves can act as nonlinear processing elements at the level of their somatic membrane potential. Intracellular recordings from bipolar cells in the salamander retina revealed frequent nonlinear integration of visual signals within bipolar cell receptive field centers, affecting the encoding of artificial and natural stimuli. These nonlinearities provide sensitivity to spatial structure below the scale of bipolar cell receptive fields in both bipolar and downstream ganglion cells and appear to arise at the excitatory input into bipolar cells. Thus, our data suggest that nonlinear signal pooling starts earlier than previously thought: that is, at the input stage of bipolar cells.

•

Some retinal bipolar cells represent visual contrast in a nonlinear fashion

•

These bipolar cells also nonlinearly integrate visual signals over space

•

The spatial nonlinearity affects the encoding of natural stimuli by bipolar cells

•

The nonlinearity results from feedforward input, not from feedback inhibition

Linear and nonlinear chromatic integration in the mouse retina

The computations performed by a neural circuit depend on how it integrates its input signals into an output of its own. In the retina, ganglion cells integrate visual information over time, space, and chromatic channels. Unlike the former two, chromatic integration is largely unexplored. Analogous to classical studies of spatial integration, we here study chromatic integration in mouse retina by identifying chromatic stimuli for which activation from the green or UV color channel is maximally balanced by deactivation through the other color channel. This reveals nonlinear chromatic integration in subsets of On, Off, and On-Off ganglion cells. Unlike the latter two, nonlinear On cells display response suppression rather than activation under balanced chromatic stimulation. Furthermore, nonlinear chromatic integration occurs independently of nonlinear spatial integration, depends on contributions from the rod pathway and on surround inhibition, and may provide information about chromatic boundaries, such as the skyline in natural scenes.

Nonlinear Spatial Integration Underlies the Diversity of Retinal Ganglion Cell Responses to Natural Images

How neurons encode natural stimuli is a fundamental question for sensory neuroscience. In the early visual system, standard encoding models assume that neurons linearly filter incoming stimuli through their receptive fields, but artificial stimuli, such as contrast-reversing gratings, often reveal nonlinear spatial processing. We investigated to what extent such nonlinear processing is relevant for the encoding of natural images in retinal ganglion cells in mice of either sex.

How neurons encode natural stimuli is a fundamental question for sensory neuroscience. In the early visual system, standard encoding models assume that neurons linearly filter incoming stimuli through their receptive fields, but artificial stimuli, such as contrast-reversing gratings, often reveal nonlinear spatial processing. We investigated to what extent such nonlinear processing is relevant for the encoding of natural images in retinal ganglion cells in mice of either sex. We found that standard linear receptive field models yielded good predictions of responses to flashed natural images for a subset of cells but failed to capture the spiking activity for many others. Cells with poor model performance displayed pronounced sensitivity to fine spatial contrast and local signal rectification as the dominant nonlinearity. By contrast, sensitivity to high-frequency contrast-reversing gratings, a classical test for nonlinear spatial integration, was not a good predictor of model performance and thus did not capture the variability of nonlinear spatial integration under natural images. In addition, we also observed a class of nonlinear ganglion cells with inverse tuning for spatial contrast, responding more strongly to spatially homogeneous than to spatially structured stimuli. These findings highlight the diversity of receptive field nonlinearities as a crucial component for understanding early sensory encoding in the context of natural stimuli.

SIGNIFICANCE STATEMENT Experiments with artificial visual stimuli have revealed that many types of retinal ganglion cells pool spatial input signals nonlinearly. However, it is still unclear how relevant this nonlinear spatial integration is when the input signals are natural images. Here we analyze retinal responses to natural scenes in large populations of mouse ganglion cells. We show that nonlinear spatial integration strongly influences responses to natural images for some ganglion cells, but not for others. Cells with nonlinear spatial integration were sensitive to spatial structure inside their receptive fields, and a small group of cells displayed a surprising sensitivity to spatially homogeneous stimuli. Traditional analyses with contrast-reversing gratings did not predict this variability of nonlinear spatial integration under natural images.

Nonlinear convergence boosts information coding in circuits with parallel outputs

Neural circuits are structured with layers of converging and diverging connectivity and selectivity-inducing nonlinearities at neurons and synapses. These components have the potential to hamper an accurate encoding of the circuit inputs. Past computational studies have optimized the nonlinearities of single neurons, or connection weights in networks, to maximize encoded information, but have not grappled with the simultaneous impact of convergent circuit structure and nonlinear response functions for efficient coding. Our approach is to compare model circuits with different combinations of convergence, divergence, and nonlinear neurons to discover how interactions between these components affect coding efficiency. We find that a convergent circuit with divergent parallel pathways can encode more information with nonlinear subunits than with linear subunits, despite the compressive loss induced by the convergence and the nonlinearities when considered separately.

Retinal ganglion cells and the magnocellular, parvocellular, and koniocellular subcortical visual pathways from the eye to the brain

In primates including humans, most retinal ganglion cells send signals to the lateral geniculate nucleus (LGN) of the thalamus. The anatomical and functional properties of the two major pathways through the LGN, the parvocellular (P) and magnocellular (M) pathways, are now well understood. Neurones in these pathways appear to convey a filtered version of the retinal image to primary visual cortex for further analysis. The properties of the P-pathway suggest it is important for high spatial acuity and red-green color vision, while those of the M-pathway suggest it is important for achromatic visual sensitivity and motion vision. Recent work has sharpened our understanding of how these properties are built in the retina, and described subtle but important nonlinearities that shape the signals that cortex receives. In addition to the P- and M-pathways, other retinal ganglion cells also project to the LGN. These ganglion cells are larger than those in the P- and M-pathways, have different retinal connectivity, and project to distinct regions of the LGN, together forming heterogenous koniocellular (K) pathways. Recent work has started to reveal the properties of these K-pathways, in the retina and in the LGN. The functional properties of K-pathways are more complex than those in the P- and M-pathways, and the K-pathways are likely to have a distinct contribution to vision. They provide a complementary pathway to the primary visual cortex, but can also send signals directly to extrastriate visual cortex. At the level of the LGN, many neurones in the K-pathways seem to integrate retinal with non-retinal inputs, and some may provide an early site of binocular convergence.

Reconstruction of natural images from responses of primate retinal ganglion cells

The visual message conveyed by a retinal ganglion cell (RGC) is often summarized by its spatial receptive field, but in principle also depends on the responses of other RGCs and natural image statistics. This possibility was explored by linear reconstruction of natural images from responses of the four numerically-dominant macaque RGC types. Reconstructions were highly consistent across retinas. The optimal reconstruction filter for each RGC - its visual message - reflected natural image statistics, and resembled the receptive field only when nearby, same-type cells were included. ON and OFF cells conveyed largely independent, complementary representations, and parasol and midget cells conveyed distinct features. Correlated activity and nonlinearities had statistically significant but minor effects on reconstruction. Simulated reconstructions, using linear-nonlinear cascade models of RGC light responses that incorporated measured spatial properties and nonlinearities, produced similar results. Spatiotemporal reconstructions exhibited similar spatial properties, suggesting that the results are relevant for natural vision.

Computational challenges and opportunities for a bi-directional artificial retina

A future artificial retina that can restore high acuity vision in blind people will rely on the capability to both read (observe) and write (control) the spiking activity of neurons using an adaptive, bi-directional and high-resolution device. Although current research is focused on overcoming the technical challenges of building and implanting such a device, exploiting its capabilities to achieve more acute visual perception will also require substantial computational advances. Using high-density large-scale recording and stimulation in the primate retina with an ex vivo multi-electrode array lab prototype, we frame several of the major computational problems, and describe current progress and future opportunities in solving them. First, we identify cell types and locations from spontaneous activity in the blind retina, and then efficiently estimate their visual response properties by using a low-dimensional manifold of inter-retina variability learned from a large experimental dataset. Second, we estimate retinal responses to a large collection of relevant electrical stimuli by passing current patterns through an electrode array, spike sorting the resulting recordings and using the results to develop a model of evoked responses. Third, we reproduce the desired responses for a given visual target by temporally dithering a diverse collection of electrical stimuli within the integration time of the visual system. Together, these novel approaches may substantially enhance artificial vision in a next-generation device.

Wide-field amacrine cell inputs to ON parasol ganglion cells in macaque retina

Parasol cells are one of the major types of primate retinal ganglion cells. The goal of this study was to describe the synaptic inputs that shape the light responses of the ON type of parasol cells, which are excited by increments in light intensity. A connectome from central macaque retina was generated by serial blockface scanning electron microscopy. Six neighboring ON parasol cells were reconstructed, and their synaptic inputs were analyzed. On average, they received 21% of their input from bipolar cells, excitatory local circuit neurons receiving input from cones. The majority of their input was from amacrine cells, local circuit neurons of the inner retina that are typically inhibitory. Their contributions to the neural circuit providing input to parasol cells are not well-understood, and the focus of this study was on the presynaptic wide-field amacrine cells, which provided 17% of the input to ON parasol cells. These are GABAergic amacrine cells with long, relatively straight dendrites, and sometimes also axons, that run in a single, narrow stratum of the inner plexiform layer. The presynaptic wide-field amacrine cells were reconstructed, and two types were identified based on their characteristic morphology. One presynaptic amacrine cell was identified as semilunar type 2, a polyaxonal cell that is electrically coupled to ON parasol cells. A second amacrine was identified as wiry type 2, a type known to be sensitive to motion. These inputs likely make ON parasol cells more sensitive to stimuli that are rapidly changing outside their classical receptive fields.

Efficient Coding by Midget and Parasol Ganglion Cells in the Human Retina

In humans, midget and parasol ganglion cells account for most of the input from the eyes to the brain. Yet, how they encode visual information is unknown. Here, we perform large-scale multi-electrode array recordings from retinas of treatment-naive patients who underwent enucleation surgery for choroidal malignant melanomas. We identify robust differences in the function of midget and parasol ganglion cells, consistent asymmetries between their ON and OFF types (that signal light increments and decrements, respectively) and divergence in the function of human versus non-human primate retinas. Our computational analyses reveal that the receptive fields of human midget and parasol ganglion cells divide naturalistic movies into adjacent spatiotemporal frequency domains with equal stimulus power, while the asymmetric response functions of their ON and OFF types simultaneously maximize stimulus coverage and information transmission and minimize metabolic cost. Thus, midget and parasol ganglion cells in the human retina efficiently encode our visual environment.

Inference of nonlinear receptive field subunits with spike-triggered clustering

Responses of sensory neurons are often modeled using a weighted combination of rectified linear subunits. Since these subunits often cannot be measured directly, a flexible method is needed to infer their properties from the responses of downstream neurons. We present a method for maximum likelihood estimation of subunits by soft-clustering spike-triggered stimuli, and demonstrate its effectiveness in visual neurons. For parasol retinal ganglion cells in macaque retina, estimated subunits partitioned the receptive field into compact regions, likely representing aggregated bipolar cell inputs. Joint clustering revealed shared subunits between neighboring cells, producing a parsimonious population model. Closed-loop validation, using stimuli lying in the null space of the linear receptive field, revealed stronger nonlinearities in OFF cells than ON cells. Responses to natural images, jittered to emulate fixational eye movements, were accurately predicted by the subunit model. Finally, the generality of the approach was demonstrated in macaque V1 neurons.

Selectivity to approaching motion in retinal inputs to the dorsal visual pathway

To efficiently navigate through the environment and avoid potential threats, an animal must quickly detect the motion of approaching objects. Current models of primate vision place the origins of this complex computation in the visual cortex. Here, we report that detection of approaching motion begins in the retina. Several ganglion cell types, the retinal output neurons, show selectivity to approaching motion. Synaptic current recordings from these cells further reveal that this preference for approaching motion arises in the interplay between presynaptic excitatory and inhibitory circuit elements. These findings demonstrate how excitatory and inhibitory circuits interact to mediate an ethologically relevant neural function. Moreover, the elementary computations that detect approaching motion begin early in the visual stream of primates.

Inferring synaptic inputs from spikes with a conductance-based neural encoding model

Descriptive statistical models of neural responses generally aim to characterize the mapping from stimuli to spike responses while ignoring biophysical details of the encoding process. Here, we introduce an alternative approach, the conductance-based encoding model (CBEM), which describes a mapping from stimuli to excitatory and inhibitory synaptic conductances governing the dynamics of sub-threshold membrane potential. Remarkably, we show that the CBEM can be fit to extracellular spike train data and then used to predict excitatory and inhibitory synaptic currents. We validate these predictions with intracellular recordings from macaque retinal ganglion cells. Moreover, we offer a novel quasi-biophysical interpretation of the Poisson generalized linear model (GLM) as a special case of the CBEM in which excitation and inhibition are perfectly balanced. This work forges a new link between statistical and biophysical models of neural encoding and sheds new light on the biophysical variables that underlie spiking in the early visual pathway.

Understanding the retinal basis of vision across species

The vertebrate retina first evolved some 500 million years ago in ancestral marine chordates. Since then, the eyes of different species have been tuned to best support their unique visuoecological lifestyles. Visual specializations in eye designs, large-scale inhomogeneities across the retinal surface and local circuit motifs mean that all species' retinas are unique. Computational theories, such as the efficient coding hypothesis, have come a long way towards an explanation of the basic features of retinal organization and function; however, they cannot explain the full extent of retinal diversity within and across species. To build a truly general understanding of vertebrate vision and the retina's computational purpose, it is therefore important to more quantitatively relate different species' retinal functions to their specific natural environments and behavioural requirements. Ultimately, the goal of such efforts should be to build up to a more general theory of vision.

Data-Driven Approaches to Understanding Visual Neuron Activity

With modern neurophysiological methods able to record neural activity throughout the visual pathway in the context of arbitrarily complex visual stimulation, our understanding of visual system function is becoming limited by the available models of visual neurons that can be directly related to such data. Different forms of statistical models are now being used to probe the cellular and circuit mechanisms shaping neural activity, understand how neural selectivity to complex visual features is computed, and derive the ways in which neurons contribute to systems-level visual processing. However, models that are able to more accurately reproduce observed neural activity often defy simple interpretations. As a result, rather than being used solely to connect with existing theories of visual processing, statistical modeling will increasingly drive the evolution of more sophisticated theories.

Neural sensitization improves encoding fidelity in the primate retina

An animal’s motion through the environment can induce large and frequent fluctuations in light intensity on the retina. These fluctuations pose a major challenge to neural circuits tasked with encoding visual information, as they can cause cells to adapt and lose sensitivity. Here, we report that sensitization, a short-term plasticity mechanism, solves this difficult computational problem by maintaining neuronal sensitivity in the face of these fluctuations. The numerically dominant output pathway in the macaque monkey retina, the midget (parvocellular-projecting) pathway, undergoes sensitization under specific conditions, including simulated eye movements. Sensitization is present in the excitatory synaptic inputs from midget bipolar cells and is mediated by presynaptic disinhibition from a wide-field mechanism extending >0.5 mm along the retinal surface. Direct physiological recordings and a computational model indicate that sensitization in the midget pathway supports accurate sensory encoding and prevents a loss of responsiveness during dynamic visual processing.

Light intensity on the retina can fluctuate rapidly during natural vision, posing a challenge for encoding visual information. Here, the authors report that mechanisms of sensitization/facilitation maintain the sensitivity of the numerically dominant neural pathway in the primate retina during dynamic vision.

An S-cone circuit for edge detection in the primate retina

Midget retinal ganglion cells (RGCs) are the most common RGC type in the primate retina. Their responses have been proposed to mediate both color and spatial vision, yet the specific links between midget RGC responses and visual perception are unclear. Previous research on the dual roles of midget RGCs has focused on those comparing long (L) vs. middle (M) wavelength sensitive cones. However, there is evidence for several other rare midget RGC subtypes receiving S-cone input, but their role in color and spatial vision is uncertain. Here, we confirm the existence of the single S-cone center OFF midget RGC circuit in the central retina of macaque monkey both structurally and functionally. We investigated the receptive field properties of the S-OFF midget circuit with single cell electrophysiology and 3D electron microscopy reconstructions of the upstream circuitry. Like the well-studied L vs. M midget RGCs, the S-OFF midget RGCs have a center-surround receptive field consistent with a role in spatial vision. While spectral opponency in a primate RGC is classically assumed to contribute to hue perception, a role supporting edge detection is more consistent with the S-OFF midget RGC receptive field structure and studies of hue perception.

Coding Principles in Adaptation

Adaptation is a common principle that recurs throughout the nervous system at all stages of processing. This principle manifests in a variety of phenomena, from spike frequency adaptation, to apparent changes in receptive fields with changes in stimulus statistics, to enhanced responses to unexpected stimuli. The ubiquity of adaptation leads naturally to the question: What purpose do these different types of adaptation serve? A diverse set of theories, often highly overlapping, has been proposed to explain the functional role of adaptive phenomena. In this review, we discuss several of these theoretical frameworks, highlighting relationships among them and clarifying distinctions. We summarize observations of the varied manifestations of adaptation, particularly as they relate to these theoretical frameworks, focusing throughout on the visual system and making connections to other sensory systems.

A Retinal Circuit Generating a Dynamic Predictive Code for Oriented Features

Sensory systems must reduce the transmission of redundant information to function efficiently. One strategy is to continuously adjust the sensitivity of neurons to suppress responses to common features of the input while enhancing responses to new ones. Here we image the excitatory synaptic inputs and outputs of retinal ganglion cells to understand how such dynamic predictive coding is implemented in the analysis of spatial patterns. Synapses of bipolar cells become tuned to orientation through presynaptic inhibition, generating lateral antagonism in the orientation domain. Individual ganglion cells receive excitatory synapses tuned to different orientations, but feedforward inhibition generates a high-pass filter that only transmits the initial activation of these inputs, removing redundancy. These results demonstrate how a dynamic predictive code can be implemented by circuit motifs common to many parts of the brain.

Probing Computation in the Primate Visual System at Single-Cone Resolution

Daylight vision begins when light activates cone photoreceptors in the retina, creating spatial patterns of neural activity. These cone signals are then combined and processed in downstream neural circuits, ultimately producing visual perception. Recent technical advances have made it possible to deliver visual stimuli to the retina that probe this processing by the visual system at its elementary resolution of individual cones. Physiological recordings from nonhuman primate retinas reveal the spatial organization of cone signals in retinal ganglion cells, including how signals from cones of different types are combined to support both spatial and color vision. Psychophysical experiments with human subjects characterize the visual sensations evoked by stimulating a single cone, including the perception of color. Future combined physiological and psychophysical experiments focusing on probing the elementary visual inputs are likely to clarify how neural processing generates our perception of the visual world.

Stimulus- and goal-oriented frameworks for understanding natural vision

Our knowledge of sensory processing has advanced dramatically in the last few decades, but this understanding remains far from complete, especially for stimuli with the large dynamic range and strong temporal and spatial correlations characteristic of natural visual inputs. Here we describe some of the issues that make understanding the encoding of natural images a challenge. We highlight two broad strategies for approaching this problem: a stimulus-oriented framework and a goal-oriented one. Different contexts can call for one framework or the other. Looking forward, recent advances, particularly those based in machine learning, show promise in borrowing key strengths of both frameworks and by doing so illuminating a path to a more comprehensive understanding of the encoding of natural stimuli.

The dynamic receptive fields of retinal ganglion cells

Retinal ganglion cells (RGCs) were one of the first classes of sensory neurons to be described in terms of a receptive field (RF). Over the last six decades, our understanding of the diversity of RGC types and the nuances of their response properties has grown exponentially. We will review the current understanding of RGC RFs mostly from studies in mammals, but including work from other vertebrates as well. We will argue for a new paradigm that embraces the fluidity of RGC RFs with an eye toward the neuroethology of vision. Specifically, we will focus on (1) different methods for measuring RGC RFs, (2) RF models, (3) feature selectivity and the distinction between fluid and stable RF properties, and (4) ideas about the future of understanding RGC RFs.

How we see the forest and the trees

Signaling pathways in the retina help us see spatial detail in our visual world.

Pathway-Specific Asymmetries between ON and OFF Visual Signals

Visual processing is largely organized into ON and OFF pathways that signal stimulus increments and decrements, respectively. These pathways exhibit natural pairings based on morphological and physiological similarities, such as ON and OFF α-ganglion cells in the mammalian retina. Several studies have noted asymmetries in the properties of ON and OFF pathways. For example, the spatial receptive fields (RFs) of OFF α-cells are systematically smaller than ON α-cells. Analysis of natural scenes suggests that these asymmetries are optimal for visual encoding. To test the generality of ON/OFF asymmetries, we measured the spatiotemporal RF properties of multiple RGC types in rat retina. Through a quantitative and serial classification, we identified three functional pairs of ON and OFF RGCs. We analyzed the structure of their RFs and compared spatial integration, temporal integration, and gain across ON and OFF pairs. Similar to previous results from the cat and primate, RGC types with larger spatial RFs exhibited briefer temporal integration and higher gain. However, each pair of ON and OFF RGC types exhibited distinct asymmetric relationships between RF properties, some of which were opposite to the findings of previous reports. These results reveal the functional organization of six RGC types in the rodent retina and indicate that ON/OFF asymmetries are pathway specific.SIGNIFICANCE STATEMENT Circuits that process sensory input frequently process increments separately from decrements, so-called ON and OFF responses. Theoretical studies indicate that this separation, and associated asymmetries in ON and OFF pathways, may be beneficial for encoding natural stimuli. However, the generality of ON and OFF pathway asymmetries has not been tested. Here we compare the functional properties of three distinct pairs of ON and OFF pathways in the rodent retina and show that their asymmetries are pathway specific. These results provide a new view on the partitioning of vision across diverse ON and OFF signaling pathways.

Neural Mechanisms of Motion Processing in the Mammalian Retina

Visual motion on the retina activates a cohort of retinal ganglion cells (RGCs). This population activity encodes multiple streams of information extracted by parallel retinal circuits. Motion processing in the retina is best studied in the direction-selective circuit. The main focus of this review is the neural basis of direction selectivity, which has been investigated in unprecedented detail using state-of-the-art functional, connectomic, and modeling methods. Mechanisms underlying the encoding of other motion features by broader RGC populations are also discussed. Recent discoveries at both single-cell and population levels highlight the dynamic and stimulus-dependent engagement of multiple mechanisms that collectively implement robust motion detection under diverse visual conditions.