Ignoring correlated activity causes a failure of retinal population codes

Tackling visual impairment: emerging avenues in ophthalmology

Visual impairment, stemming from genetic, degenerative, and traumatic causes, affects millions globally. Recent advancements in ophthalmology present novel strategies for managing and potentially reversing these conditions. Here, we explore 10 emerging avenues-including gene therapy, stem cell therapy, advanced imaging, novel therapeutics, nanotechnology, artificial intelligence (AI) and machine learning, teleophthalmology, optogenetics, bionics, and neuro-ophthalmology-all making strides to improve diagnosis, treatment, and vision restoration. Among these, gene therapy and stem cell therapy are revolutionizing the treatment of retinal degenerative diseases, while advanced imaging technologies enable early detection and personalized care. Therapeutic advancements like anti-vascular endothelial growth factor therapies and neuroprotective agents, along with nanotechnology, have improved clinical outcomes for multiple ocular conditions. AI, especially machine learning, is enhancing diagnostic accuracy, facilitating early detection, and personalized treatment strategies, particularly when integrated with advanced imaging technologies. Teleophthalmology, further strengthened by AI, is expanding access to care, particularly in underserved regions, whereas emerging technologies like optogenetics, bionics, and neuro-ophthalmology offer new hope for patients with severe vision impairment. In light of ongoing research, we summarize the current clinical landscape and the potential advantages of these innovations to revolutionize the management of visual impairments. Additionally, we address the challenges and limitations associated with these emerging avenues in ophthalmology, providing insights into their future trajectories in clinical practice. Continued advancements in these fields promise to reshape the landscape of ophthalmic care, ultimately improving the quality of life for individuals with visual impairments.

Luminance invariant encoding in mouse primary visual cortex

The visual system adapts to maintain sensitivity and selectivity over a large range of luminance intensities. One way that the retina maintains sensitivity across night and day is by switching between rod and cone photoreceptors, which alters the receptive fields and interneuronal correlations of retinal ganglion cells (RGCs). While these adaptations allow the retina to transmit visual information to the brain across environmental conditions, the code used for that transmission varies. To determine how downstream targets encode visual scenes across light levels, we measured the effects of luminance adaptation on thalamic and cortical population activity. While changes in the retinal output are evident in the lateral geniculate nucleus (LGN), selectivity in the primary visual cortex (V1) is largely invariant to the changes in luminance. We show that the visual system could maintain sensitivity across environmental conditions without altering cortical selectivity through the convergence of parallel functional pathways from the thalamus to the cortex.

The geometry of correlated variability leads to highly suboptimal discriminative sensory coding

The brain represents the world through the activity of neural populations; however, whether the computational goal of sensory coding is to support discrimination of sensory stimuli or to generate an internal model of the sensory world is unclear. Correlated variability across a neural population (noise correlations) is commonly observed experimentally, and many studies demonstrate that correlated variability improves discriminative sensory coding compared to a null model with no correlations. However, such results do not address whether correlated variability is optimal for discriminative sensory coding. If the computational goal of sensory coding is discriminative, than correlated variability should be optimized to support that goal. We assessed optimality of noise correlations for discriminative sensory coding in diverse datasets by developing two novel null models, each with a biological interpretation. Across datasets, we found that correlated variability in neural populations leads to highly suboptimal discriminative sensory coding according to both null models. Furthermore, biological constraints prevent many subsets of the neural populations from achieving optimality, and subselecting based on biological criteria leaves red discriminative coding performance suboptimal. Finally, we show that optimal subpopulations are exponentially small as the population size grows. Together, these results demonstrate that the geometry of correlated variability leads to highly suboptimal discriminative sensory coding.NEW & NOTEWORTHY The brain represents the world through the activity of neural populations that exhibit correlated variability. We assessed optimality of correlated variability for discriminative sensory coding in diverse datasets by developing two novel null models. Across datasets, correlated variability in neural populations leads to highly suboptimal discriminative sensory coding according to both null models. Biological constraints prevent the neural populations from achieving optimality. Together, these results demonstrate that the geometry of correlated variability leads to highly suboptimal discriminative sensory coding.

Nonlinear receptive fields evoke redundant retinal coding of natural scenes

The role of the vertebrate retina in early vision is generally described by the efficient coding hypothesis1,2, which predicts that the retina reduces the redundancy inherent in natural scenes3 by discarding spatiotemporal correlations while preserving stimulus information4. It is unclear, however, whether the predicted decorrelation and redundancy reduction in the activity of ganglion cells, the retina's output neurons, hold under gaze shifts, which dominate the dynamics of the natural visual input5. We show here that species-specific gaze patterns in natural stimuli can drive correlated spiking responses both in and across distinct types of ganglion cells in marmoset as well as mouse retina. These concerted responses disrupt redundancy reduction to signal fixation periods with locally high spatial contrast. Model-based analyses of ganglion cell responses to natural stimuli show that the observed response correlations follow from nonlinear pooling of ganglion cell inputs. Our results indicate cell-type-specific deviations from efficient coding in retinal processing of natural gaze shifts.

Fixational eye movements enhance the precision of visual information transmitted by the primate retina

Fixational eye movements alter the number and timing of spikes transmitted from the retina to the brain, but whether these changes enhance or degrade the retinal signal is unclear. To quantify this, we developed a Bayesian method for reconstructing natural images from the recorded spikes of hundreds of retinal ganglion cells (RGCs) in the macaque retina (male), combining a likelihood model for RGC light responses with the natural image prior implicitly embedded in an artificial neural network optimized for denoising. The method matched or surpassed the performance of previous reconstruction algorithms, and provides an interpretable framework for characterizing the retinal signal. Reconstructions were improved with artificial stimulus jitter that emulated fixational eye movements, even when the eye movement trajectory was assumed to be unknown and had to be inferred from retinal spikes. Reconstructions were degraded by small artificial perturbations of spike times, revealing more precise temporal encoding than suggested by previous studies. Finally, reconstructions were substantially degraded when derived from a model that ignored cell-to-cell interactions, indicating the importance of stimulus-evoked correlations. Thus, fixational eye movements enhance the precision of the retinal representation.

Biophysical neural adaptation mechanisms enable artificial neural networks to capture dynamic retinal computation

Adaptation is a universal aspect of neural systems that changes circuit computations to match prevailing inputs. These changes facilitate efficient encoding of sensory inputs while avoiding saturation. Conventional artificial neural networks (ANNs) have limited adaptive capabilities, hindering their ability to reliably predict neural output under dynamic input conditions. Can embedding neural adaptive mechanisms in ANNs improve their performance? To answer this question, we develop a new deep learning model of the retina that incorporates the biophysics of photoreceptor adaptation at the front-end of conventional convolutional neural networks (CNNs). These conventional CNNs build on 'Deep Retina,' a previously developed model of retinal ganglion cell (RGC) activity. CNNs that include this new photoreceptor layer outperform conventional CNN models at predicting male and female primate and rat RGC responses to naturalistic stimuli that include dynamic local intensity changes and large changes in the ambient illumination. These improved predictions result directly from adaptation within the phototransduction cascade. This research underscores the potential of embedding models of neural adaptation in ANNs and using them to determine how neural circuits manage the complexities of encoding natural inputs that are dynamic and span a large range of light levels.

Advancements in brain-machine interfaces for application in the metaverse

In recent years, with the shift of focus in metaverse research toward content exchange and social interaction, breaking through the current bottleneck of audio-visual media interaction has become an urgent issue. The use of brain-machine interfaces for sensory simulation is one of the proposed solutions. Currently, brain-machine interfaces have demonstrated irreplaceable potential as physiological signal acquisition tools in various fields within the metaverse. This study explores three application scenarios: generative art in the metaverse, serious gaming for healthcare in metaverse medicine, and brain-machine interface applications for facial expression synthesis in the virtual society of the metaverse. It investigates existing commercial products and patents (such as MindWave Mobile, GVS, and Galea), draws analogies with the development processes of network security and neurosecurity, bioethics and neuroethics, and discusses the challenges and potential issues that may arise when brain-machine interfaces mature and are widely applied. Furthermore, it looks ahead to the diverse possibilities of deep and varied applications of brain-machine interfaces in the metaverse in the future.

Calibrating Bayesian Decoders of Neural Spiking Activity

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Small-correlation expansion to quantify information in noisy sensory systems

Neural networks encode information through their collective spiking activity in response to external stimuli. This population response is noisy and strongly correlated, with a complex interplay between correlations induced by the stimulus, and correlations caused by shared noise. Understanding how these correlations affect information transmission has so far been limited to pairs or small groups of neurons, because the curse of dimensionality impedes the evaluation of mutual information in larger populations. Here, we develop a small-correlation expansion to compute the stimulus information carried by a large population of neurons, yielding interpretable analytical expressions in terms of the neurons' firing rates and pairwise correlations. We validate the approximation on synthetic data and demonstrate its applicability to electrophysiological recordings in the vertebrate retina, allowing us to quantify the effects of noise correlations between neurons and of memory in single neurons.

Dynamic structure of motor cortical neuron coactivity carries behaviorally relevant information

Skillful, voluntary movements are underpinned by computations performed by networks of interconnected neurons in the primary motor cortex (M1). Computations are reflected by patterns of coactivity between neurons. Using pairwise spike time statistics, coactivity can be summarized as a functional network (FN). Here, we show that the structure of FNs constructed from an instructed-delay reach task in nonhuman primates is behaviorally specific: Low-dimensional embedding and graph alignment scores show that FNs constructed from closer target reach directions are also closer in network space. Using short intervals across a trial, we constructed temporal FNs and found that temporal FNs traverse a low-dimensional subspace in a reach-specific trajectory. Alignment scores show that FNs become separable and correspondingly decodable shortly after the Instruction cue. Finally, we observe that reciprocal connections in FNs transiently decrease following the Instruction cue, consistent with the hypothesis that information external to the recorded population temporarily alters the structure of the network at this moment.

Large-scale interrogation of retinal cell functions by 1-photon light-sheet microscopy

Visual processing in the retina depends on the collective activity of large ensembles of neurons organized in different layers. Current techniques for measuring activity of layer-specific neural ensembles rely on expensive pulsed infrared lasers to drive 2-photon activation of calcium-dependent fluorescent reporters. We present a 1-photon light-sheet imaging system that can measure the activity in hundreds of neurons in the ex vivo retina over a large field of view while presenting visual stimuli. This allows for a reliable functional classification of different retinal cell types. We also demonstrate that the system has sufficient resolution to image calcium entry at individual synaptic release sites across the axon terminals of dozens of simultaneously imaged bipolar cells. The simple design, large field of view, and fast image acquisition make this a powerful system for high-throughput and high-resolution measurements of retinal processing at a fraction of the cost of alternative approaches.

Differences in the spatial fidelity of evoked and spontaneous signals in the degenerating retina

Vision restoration strategies aim to reestablish vision by replacing the function of lost photoreceptors with optoelectronic hardware or through gene therapy. One complication to these approaches is that retinal circuitry undergoes remodeling after photoreceptor loss. Circuit remodeling following perturbation is ubiquitous in the nervous system and understanding these changes is crucial for treating neurodegeneration. Spontaneous oscillations that arise during retinal degeneration have been well-studied, however, other changes in the spatiotemporal processing of evoked and spontaneous activity have received less attention. Here we use subretinal electrical stimulation to measure the spatial and temporal spread of both spontaneous and evoked activity during retinal degeneration. We found that electrical stimulation synchronizes spontaneous oscillatory activity, over space and through time, thus leading to increased correlations in ganglion cell activity. Intriguingly, we found that spatial selectivity was maintained in rd10 retina for evoked responses, with spatial receptive fields comparable to wt retina. These findings indicate that different biophysical mechanisms are involved in mediating feed forward excitation, and the lateral spread of spontaneous activity in the rd10 retina, lending support toward the possibility of high-resolution vision restoration.

Retinal Processing: Insights from Mathematical Modelling

The retina is the entrance of the visual system. Although based on common biophysical principles, the dynamics of retinal neurons are quite different from their cortical counterparts, raising interesting problems for modellers. In this paper, I address some mathematically stated questions in this spirit, discussing, in particular: (1) How could lateral amacrine cell connectivity shape the spatio-temporal spike response of retinal ganglion cells? (2) How could spatio-temporal stimuli correlations and retinal network dynamics shape the spike train correlations at the output of the retina? These questions are addressed, first, introducing a mathematically tractable model of the layered retina, integrating amacrine cells' lateral connectivity and piecewise linear rectification, allowing for computing the retinal ganglion cells receptive field together with the voltage and spike correlations of retinal ganglion cells resulting from the amacrine cells networks. Then, I review some recent results showing how the concept of spatio-temporal Gibbs distributions and linear response theory can be used to characterize the collective spike response to a spatio-temporal stimulus of a set of retinal ganglion cells, coupled via effective interactions corresponding to the amacrine cells network. On these bases, I briefly discuss several potential consequences of these results at the cortical level.

Modelling the neural code in large populations of correlated neurons

Neurons respond selectively to stimuli, and thereby define a code that associates stimuli with population response patterns. Certain correlations within population responses (noise correlations) significantly impact the information content of the code, especially in large populations. Understanding the neural code thus necessitates response models that quantify the coding properties of modelled populations, while fitting large-scale neural recordings and capturing noise correlations. In this paper, we propose a class of response model based on mixture models and exponential families. We show how to fit our models with expectation-maximization, and that they capture diverse variability and covariability in recordings of macaque primary visual cortex. We also show how they facilitate accurate Bayesian decoding, provide a closed-form expression for the Fisher information, and are compatible with theories of probabilistic population coding. Our framework could allow researchers to quantitatively validate the predictions of neural coding theories against both large-scale neural recordings and cognitive performance.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Variations in photoreceptor throughput to mouse visual cortex and the unique effects on tuning

Visual input to primary visual cortex (V1) depends on highly adaptive filtering in the retina. In turn, isolation of V1 computations requires experimental control of retinal adaptation to infer its spatio-temporal-chromatic output. Here, we measure the balance of input to mouse V1, in the anesthetized setup, from the three main photoreceptor opsins-M-opsin, S-opsin, and rhodopsin-as a function of two stimulus dimensions. The first dimension is the level of light adaptation within the mesopic range, which governs the balance of rod and cone inputs to cortex. The second stimulus dimension is retinotopic position, which governs the balance of S- and M-cone opsin input due to the opsin expression gradient in the retina. The fitted model predicts opsin input under arbitrary lighting environments, which provides a much-needed handle on in-vivo studies of the mouse visual system. We use it here to reveal that V1 is rod-mediated in common laboratory settings yet cone-mediated in natural daylight. Next, we compare functional properties of V1 under rod and cone-mediated inputs. The results show that cone-mediated V1 responds to 2.5-fold higher temporal frequencies than rod-mediated V1. Furthermore, cone-mediated V1 has smaller receptive fields, yet similar spatial frequency tuning. V1 responses in rod-deficient (Gnat1-/-) mice confirm that the effects are due to differences in photoreceptor opsin contribution.

Reconstruction of natural images from responses of primate retinal ganglion cells

The visual message conveyed by a retinal ganglion cell (RGC) is often summarized by its spatial receptive field, but in principle also depends on the responses of other RGCs and natural image statistics. This possibility was explored by linear reconstruction of natural images from responses of the four numerically-dominant macaque RGC types. Reconstructions were highly consistent across retinas. The optimal reconstruction filter for each RGC - its visual message - reflected natural image statistics, and resembled the receptive field only when nearby, same-type cells were included. ON and OFF cells conveyed largely independent, complementary representations, and parasol and midget cells conveyed distinct features. Correlated activity and nonlinearities had statistically significant but minor effects on reconstruction. Simulated reconstructions, using linear-nonlinear cascade models of RGC light responses that incorporated measured spatial properties and nonlinearities, produced similar results. Spatiotemporal reconstructions exhibited similar spatial properties, suggesting that the results are relevant for natural vision.