Rank Order Coding: a Retinal Information Decoding Strategy Revealed by Large-Scale Multielectrode Array Retinal Recordings

A neurocomputational model of decision and confidence in object recognition task

How does the brain process natural visual stimuli to make a decision? Imagine driving through fog. An object looms ahead. What do you do? This decision requires not only identifying the object but also choosing an action based on your decision confidence. In this circumstance, confidence is making a bridge between seeing and believing. Our study unveils how the brain processes visual information to make such decisions with an assessment of confidence, using a model inspired by the visual cortex. To computationally model the process, this study uses a spiking neural network inspired by the hierarchy of the visual cortex in mammals to investigate the dynamics of feedforward object recognition and decision-making in the brain. The model consists of two modules: a temporal dynamic object representation module and an attractor neural network-based decision-making module. Unlike traditional models, ours captures the evolution of evidence within the visual cortex, mimicking how confidence forms in the brain. This offers a more biologically plausible approach to decision-making when encountering real-world stimuli. We conducted experiments using natural stimuli and measured accuracy, reaction time, and confidence. The model's estimated confidence aligns remarkably well with human-reported confidence. Furthermore, the model can simulate the human change-of-mind phenomenon, reflecting the ongoing evaluation of evidence in the brain. Also, this finding offers decision-making and confidence encoding share the same neural circuit.

A fully spiking coupled model of a deep neural network and a recurrent attractor explains dynamics of decision making in an object recognition task

Objective.Object recognition and making a choice regarding the recognized object is pivotal for most animals. This process in the brain contains information representation and decision making steps which both take different amount of times for different objects. While dynamics of object recognition and decision making are usually ignored in object recognition models, here we proposed a fully spiking hierarchical model, explaining the process of object recognition from information representation to making decision.Approach.Coupling a deep neural network and a recurrent attractor based decision making model beside using spike time dependent plasticity learning rules in several convolutional and pooling layers, we proposed a model which can resemble brain behaviors during an object recognition task. We also measured human choices and reaction times in a psychophysical object recognition task and used it as a reference to evaluate the model.Main results.The proposed model explains not only the probability of making a correct decision but also the time that it takes to make a decision. Importantly, neural firing rates in both feature representation and decision making levels mimic the observed patterns in animal studies (number of spikes (p-value < 10-173) and the time of the peak response (p-value < 10-31) are significantly modulated with the strength of the stimulus). Moreover, the speed-accuracy trade-off as a well-known characteristic of decision making process in the brain is also observed in the model (changing the decision bound significantly affect the reaction time (p-value < 10-59) and accuracy (p-value < 10-165)).Significance.We proposed a fully spiking deep neural network which can explain dynamics of making decision about an object in both neural and behavioral level. Results showed that there is a strong and significant correlation (r= 0.57) between the reaction time of the model and of human participants in the psychophysical object recognition task.

Spike sorting in the presence of stimulation artifacts: a dynamical control systems approach

Objective. Bi-directional electronic neural interfaces, capable of both electrical recording and stimulation, communicate with the nervous system to permit precise calibration of electrical inputs by capturing the evoked neural responses. However, one significant challenge is that stimulation artifacts often mask the actual neural signals. To address this issue, we introduce a novel approach that employs dynamical control systems to detect and decipher electrically evoked neural activity despite the presence of electrical artifacts.Approach. Our proposed method leverages the unique spatiotemporal patterns of neural activity and electrical artifacts to distinguish and identify individual neural spikes. We designed distinctive dynamical models for both the stimulation artifact and each neuron observed during spontaneous neural activity. We can estimate which neurons were active by analyzing the recorded voltage responses across multiple electrodes post-stimulation. This technique also allows us to exclude signals from electrodes heavily affected by stimulation artifacts, such as the stimulating electrode itself, yet still accurately differentiate between evoked spikes and electrical artifacts.Main results. We applied our method to high-density multi-electrode recordings from the primate retina in anex vivosetup, using a grid of 512 electrodes. Through repeated electrical stimulations at varying amplitudes, we were able to construct activation curves for each neuron. The curves obtained with our method closely resembled those derived from manual spike sorting. Additionally, the stimulation thresholds we estimated strongly agreed with those determined through manual analysis, demonstrating high reliability (R2=0.951for human 1 andR2=0.944for human 2).Significance. Our method can effectively separate evoked neural spikes from stimulation artifacts by exploiting the distinct spatiotemporal propagation patterns captured by a dense, large-scale multi-electrode array. This technique holds promise for future applications in real-time closed-loop stimulation systems and for managing multi-channel stimulation strategies.

Sparse-firing regularization methods for spiking neural networks with time-to-first-spike coding

The training of multilayer spiking neural networks (SNNs) using the error backpropagation algorithm has made significant progress in recent years. Among the various training schemes, the error backpropagation method that directly uses the firing time of neurons has attracted considerable attention because it can realize ideal temporal coding. This method uses time-to-first-spike (TTFS) coding, in which each neuron fires at most once, and this restriction on the number of firings enables information to be processed at a very low firing frequency. This low firing frequency increases the energy efficiency of information processing in SNNs. However, only an upper limit has been provided for TTFS-coded SNNs, and the information-processing capability of SNNs at lower firing frequencies has not been fully investigated. In this paper, we propose two spike-timing-based sparse-firing (SSR) regularization methods to further reduce the firing frequency of TTFS-coded SNNs. Both methods are characterized by the fact that they only require information about the firing timing and associated weights. The effects of these regularization methods were investigated on the MNIST, Fashion-MNIST, and CIFAR-10 datasets using multilayer perceptron networks and convolutional neural network structures.

Receptive field estimation in large visual neuron assemblies using a super-resolution approach

Computing the spike-triggered average (STA) is a simple method to estimate the sensory neurons' linear receptive fields (RFs). For random, uncorrelated stimuli the STA provides an unbiased RF estimate, but in practice, white noise is not a feasible stimulus as it usually evokes only weak responses. Therefore, for a visual stimulus, it is often used images of randomly modulated blocks of pixels. This solution naturally limits the resolution at which an RF can be obtained. Here we show that this limitation can be overcome by using a simple super-resolution technique. We define a novel type of stimulus, the Shifted White Noise (SWN), by introducing random spatial shifts in the usual stimulus in order to increase the resolution of the measurements. In simulated data we show that the average error using the SWN was 1.7 times smaller than when using the classical stimulus, with successful mapping of 2.3 times more neurons, covering a broader range of RF sizes. Moreover, successful RF mapping was achieved with short recordings of about one minute of activity, more than 10 times more efficient than the classical white noise stimulus. In recordings from mouse retinal ganglion cells with large scale microelectrode arrays, we could map 18 times more RFs covering a broader range of sizes. In summary, here we show that randomly shifting the usual white noise stimulus significantly improves RFs estimation, and requires only short recordings. It is straight forward to extend this method into the time dimension and adapt it to other sensory modalities.

A novel approach to the functional classification of retinal ganglion cells

Retinal neurons are remarkedly diverse based on structure, function and genetic identity. Classifying these cells is a challenging task, requiring multimodal methodology. Here, we introduce a novel approach for retinal ganglion cell (RGC) classification, based on pharmacogenetics combined with immunohistochemistry and large-scale retinal electrophysiology. Our novel strategy allows grouping of cells sharing gene expression and understanding how these cell classes respond to basic and complex visual scenes. Our approach consists of several consecutive steps. First, the spike firing frequency is increased in RGCs co-expressing a certain gene (Scnn1a or Grik4) using excitatory DREADDs (designer receptors exclusively activated by designer drugs) in order to single out activity originating specifically from these cells. Their spike location is then combined with post hoc immunostaining, to unequivocally characterize their anatomical and functional features. We grouped these isolated RGCs into multiple clusters based on spike train similarities. Using this novel approach, we were able to extend the pre-existing list of Grik4-expressing RGC types to a total of eight and, for the first time, we provide a phenotypical description of 13 Scnn1a-expressing RGCs. The insights and methods gained here can guide not only RGC classification but neuronal classification challenges in other brain regions as well.

A Survey of Encoding Techniques for Signal Processing in Spiking Neural Networks

Biologically inspired spiking neural networks are increasingly popular in the field of artificial intelligence due to their ability to solve complex problems while being power efficient. They do so by leveraging the timing of discrete spikes as main information carrier. Though, industrial applications are still lacking, partially because the question of how to encode incoming data into discrete spike events cannot be uniformly answered. In this paper, we summarise the signal encoding schemes presented in the literature and propose a uniform nomenclature to prevent the vague usage of ambiguous definitions. Therefore we survey both, the theoretical foundations as well as applications of the encoding schemes. This work provides a foundation in spiking signal encoding and gives an overview over different application-oriented implementations which utilise the schemes.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Comparison of Artificial and Spiking Neural Networks on Digital Hardware

Despite the success of Deep Neural Networks-a type of Artificial Neural Network (ANN)-in problem domains such as image recognition and speech processing, the energy and processing demands during both training and deployment are growing at an unsustainable rate in the push for greater accuracy. There is a temptation to look for radical new approaches to these applications, and one such approach is the notion that replacing the abstract neuron used in most deep networks with a more biologically-plausible spiking neuron might lead to savings in both energy and resource cost. The most common spiking networks use rate-coded neurons for which a simple translation from a pre-trained ANN to an equivalent spike-based network (SNN) is readily achievable. But does the spike-based network offer an improvement of energy efficiency over the original deep network? In this work, we consider the digital implementations of the core steps in an ANN and the equivalent steps in a rate-coded spiking neural network. We establish a simple method of assessing the relative advantages of rate-based spike encoding over a conventional ANN model. Assuming identical underlying silicon technology we show that most rate-coded spiking network implementations will not be more energy or resource efficient than the original ANN, concluding that more imaginative uses of spikes are required to displace conventional ANNs as the dominant computing framework for neural computation.

Reconstruction of natural images from responses of primate retinal ganglion cells

The visual message conveyed by a retinal ganglion cell (RGC) is often summarized by its spatial receptive field, but in principle also depends on the responses of other RGCs and natural image statistics. This possibility was explored by linear reconstruction of natural images from responses of the four numerically-dominant macaque RGC types. Reconstructions were highly consistent across retinas. The optimal reconstruction filter for each RGC - its visual message - reflected natural image statistics, and resembled the receptive field only when nearby, same-type cells were included. ON and OFF cells conveyed largely independent, complementary representations, and parasol and midget cells conveyed distinct features. Correlated activity and nonlinearities had statistically significant but minor effects on reconstruction. Simulated reconstructions, using linear-nonlinear cascade models of RGC light responses that incorporated measured spatial properties and nonlinearities, produced similar results. Spatiotemporal reconstructions exhibited similar spatial properties, suggesting that the results are relevant for natural vision.

Response Latency Tuning by Retinal Circuits Modulates Signal Efficiency

In the visual system, retinal ganglion cells (RGCs) of various subtypes encode preprocessed photoreceptor signals into a spike output which is then transmitted towards the brain through parallel feature pathways. Spike timing determines how each feature signal contributes to the output of downstream neurons in visual brain centers, thereby influencing efficiency in visual perception. In this study, we demonstrate a marked population-wide variability in RGC response latency that is independent of trial-to-trial variability and recording approach. RGC response latencies to simple visual stimuli vary considerably in a heterogenous cell population but remain reliable when RGCs of a single subtype are compared. This subtype specificity, however, vanishes when the retinal circuitry is bypassed via direct RGC electrical stimulation. This suggests that latency is primarily determined by the signaling speed through retinal pathways that provide subtype specific inputs to RGCs. In addition, response latency is significantly altered when GABA inhibition or gap junction signaling is disturbed, which further supports the key role of retinal microcircuits in latency tuning. Finally, modulation of stimulus parameters affects individual RGC response delays considerably. Based on these findings, we hypothesize that retinal microcircuits fine-tune RGC response latency, which in turn determines the context-dependent weighing of each signal and its contribution to visual perception.

New Drug Discovery Paradigms for Retinal Diseases: A Focus on Retinal Organoids

Retinal disease represents a growing global problem, both in terms of quality of life and economic impact, yet new therapies are not being developed at a sufficient rate to meet this mounting need. In this context, retinal organoids derived from human induced pluripotent stem cells hold significant promise for improving upon the current drug development process, increasing the speed and efficiency of moving potential therapeutic agents from bench to bedside. These organoid systems display the cell–cell and cell–matrix interactions, cellular heterogeneity, and physiological responses reflective of human biology and, thus, have the ability to replicate retinal disease pathology in a way that 2-dimensional cell cultures and animal models have been heretofore unable to achieve. However, organoid technology is not yet mature enough to meet the high-throughput demands of the first stages of drug screening. Hence, the augmentation of the existing drug development pipeline with retinal organoids, rather than the replacement of existing pathway components, may provide a way to harness the benefits of this improved pathological modeling. In this study, we outline the possible benefits of such a symbiosis, discuss other potential uses, and highlight barriers that remain to be overcome.

Non-parametric Physiological Classification of Retinal Ganglion Cells in the Mouse Retina

Retinal ganglion cells, the sole output neurons of the retina, exhibit surprising diversity. A recent study reported over 30 distinct types in the mouse retina, indicating that the processing of visual information is highly parallelised in the brain. The advent of high density multi-electrode arrays now enables recording from many hundreds to thousands of neurons from a single retina. Here we describe a method for the automatic classification of large-scale retinal recordings using a simple stimulus paradigm and a spike train distance measure as a clustering metric. We evaluate our approach using synthetic spike trains, and demonstrate that major known cell types are identified in high-density recording sessions from the mouse retina with around 1,000 retinal ganglion cells. A comparison across different retinas reveals substantial variability between preparations, suggesting pooling data across retinas should be approached with caution. As a parameter-free method, our approach is broadly applicable for cellular physiological classification in all sensory modalities.

SiNAPS: An implantable active pixel sensor CMOS-probe for simultaneous large-scale neural recordings

Large-scale neural recordings with high spatial and temporal accuracy are instrumental to understand how the brain works. To this end, it is of key importance to develop probes that can be conveniently scaled up to a high number of recording channels. Despite recent achievements in complementary metal-oxide semiconductor (CMOS) multi-electrode arrays probes, in current circuit architectures an increase in the number of simultaneously recording channels would significantly increase the total chip area. A promising approach for overcoming this scaling issue consists in the use of the modular Active Pixel Sensor (APS) concept, in which a small front-end circuit is located beneath each electrode. However, this approach imposes challenging constraints on the area of the in-pixel circuit, power consumption and noise. Here, we present an APS CMOS-probe technology for Simultaneous Neural recording that successfully addresses all these issues for whole-array read-outs at 25 kHz/channel from up to 1024 electrode-pixels. To assess the circuit performances, we realized in a 0.18 μm CMOS technology an implantable single-shaft probe with a regular array of 512 electrode-pixels with a pitch of 28 μm. Extensive bench tests showed an in-pixel gain of 45.4 ± 0.4 dB (low pass, F_-3 dB = 4 kHz), an input referred noise of 7.5 ± 0.67 μV_RMS (300 Hz to 7.5 kHz) and a power consumption <6 μW/pixel. In vivo acute recordings demonstrate that our SiNAPS CMOS-probe can sample full-band bioelectrical signals from each electrode, with the ability to resolve and discriminate activity from several packed neurons both at the spatial and temporal scale. These results pave the way to new generations of compact and scalable active single/multi-shaft brain recording systems.

Spiking neural networks for computer vision

State-of-the-art computer vision systems use frame-based cameras that sample the visual scene as a series of high-resolution images. These are then processed using convolutional neural networks using neurons with continuous outputs. Biological vision systems use a quite different approach, where the eyes (cameras) sample the visual scene continuously, often with a non-uniform resolution, and generate neural spike events in response to changes in the scene. The resulting spatio-temporal patterns of events are then processed through networks of spiking neurons. Such event-based processing offers advantages in terms of focusing constrained resources on the most salient features of the perceived scene, and those advantages should also accrue to engineered vision systems based upon similar principles. Event-based vision sensors, and event-based processing exemplified by the SpiNNaker (Spiking Neural Network Architecture) machine, can be used to model the biological vision pathway at various levels of detail. Here we use this approach to explore structural synaptic plasticity as a possible mechanism whereby biological vision systems may learn the statistics of their inputs without supervision, pointing the way to engineered vision systems with similar online learning capabilities.

Unsupervised Spike Sorting for Large-Scale, High-Density Multielectrode Arrays

We present a method for automated spike sorting for recordings with high-density, large-scale multielectrode arrays. Exploiting the dense sampling of single neurons by multiple electrodes, an efficient, low-dimensional representation of detected spikes consisting of estimated spatial spike locations and dominant spike shape features is exploited for fast and reliable clustering into single units. Millions of events can be sorted in minutes, and the method is parallelized and scales better than quadratically with the number of detected spikes. Performance is demonstrated using recordings with a 4,096-channel array and validated using anatomical imaging, optogenetic stimulation, and model-based quality control. A comparison with semi-automated, shape-based spike sorting exposes significant limitations of conventional methods. Our approach demonstrates that it is feasible to reliably isolate the activity of up to thousands of neurons and that dense, multi-channel probes substantially aid reliable spike sorting.

Stimulation triggers endogenous activity patterns in cultured cortical networks

Cultures of dissociated cortical neurons represent a powerful trade-off between more realistic experimental models and abstract modeling approaches, allowing to investigate mechanisms of synchronized activity generation. These networks spontaneously alternate periods of high activity (i.e. network bursts) with periods of quiescence in a dynamic state which recalls the fluctuation of in vivo UP and DOWN states. Network bursts can also be elicited by external stimulation and their spatial propagation patterns tracked by means of multi-channel micro-electrode arrays. In this study, we used rat cortical cultures coupled to micro-electrode arrays to investigate the similarity between spontaneous and evoked activity patterns. We performed experiments by applying electrical stimulation to different network locations and demonstrated that the rank orders of electrodes during evoked and spontaneous events are remarkably similar independently from the stimulation source. We linked this result to the capability of stimulation to evoke firing in highly active and “leader” sites of the network, reliably and rapidly recruited within both spontaneous and evoked bursts. Our study provides the first evidence that spontaneous and evoked activity similarity is reliably observed also in dissociated cortical networks.

Pan-retinal characterisation of Light Responses from Ganglion Cells in the Developing Mouse Retina

We have investigated the ontogeny of light-driven responses in mouse retinal ganglion cells (RGCs). Using a large-scale, high-density multielectrode array, we recorded from hundreds to thousands of RGCs simultaneously at pan-retinal level, including dorsal and ventral locations. Responses to different contrasts not only revealed a complex developmental profile for ON, OFF and ON-OFF responses, but also unveiled differences between dorsal and ventral RGC responses. At eye-opening, dorsal RGCs of all types were more responsive to light, perhaps indicating an environmental priority to nest viewing for pre-weaning pups. The developmental profile of ON and OFF responses exhibited antagonistic behaviour, with the strongest ON responses shortly after eye-opening, followed by an increase in the strength of OFF responses later on. Further, we found that with maturation receptive field (RF) center sizes decrease, spike-triggered averaged responses to white noise become stronger, and centers become more circular while maintaining differences between RGC types. We conclude that the maturation of retinal functionality is not spatially homogeneous, likely reflecting ecological requirements that favour earlier maturation of the dorsal retina.

Decision making under uncertainty in a spiking neural network model of the basal ganglia

The mechanisms of decision-making and action selection are generally thought to be under the control of parallel cortico-subcortical loops connecting back to distinct areas of cortex through the basal ganglia and processing motor, cognitive and limbic modalities of decision-making. We have used these properties to develop and extend a connectionist model at a spiking neuron level based on a previous rate model approach. This model is demonstrated on decision-making tasks that have been studied in primates and the electrophysiology interpreted to show that the decision is made in two steps. To model this, we have used two parallel loops, each of which performs decision-making based on interactions between positive and negative feedback pathways. This model is able to perform two-level decision-making as in primates. We show here that, before learning, synaptic noise is sufficient to drive the decision-making process and that, after learning, the decision is based on the choice that has proven most likely to be rewarded. The model is then submitted to lesion tests, reversal learning and extinction protocols. We show that, under these conditions, it behaves in a consistent manner and provides predictions in accordance with observed experimental data.

Dampening Spontaneous Activity Improves the Light Sensitivity and Spatial Acuity of Optogenetic Retinal Prosthetic Responses

Retinitis pigmentosa is a progressive retinal dystrophy that causes irreversible visual impairment and blindness. Retinal prostheses currently represent the only clinically available vision-restoring treatment, but the quality of vision returned remains poor. Recently, it has been suggested that the pathological spontaneous hyperactivity present in dystrophic retinas may contribute to the poor quality of vision returned by retinal prosthetics by reducing the signal-to-noise ratio of prosthetic responses. Here, we investigated to what extent blocking this hyperactivity can improve optogenetic retinal prosthetic responses. We recorded activity from channelrhodopsin-expressing retinal ganglion cells in retinal wholemounts in a mouse model of retinitis pigmentosa. Sophisticated stimuli, inspired by those used in clinical visual assessment, were used to assess light sensitivity, contrast sensitivity and spatial acuity of optogenetic responses; in all cases these were improved after blocking spontaneous hyperactivity using meclofenamic acid, a gap junction blocker. Our results suggest that this approach significantly improves the quality of vision returned by retinal prosthetics, paving the way to novel clinical applications. Moreover, the improvements in sensitivity achieved by blocking spontaneous hyperactivity may extend the dynamic range of optogenetic retinal prostheses, allowing them to be used at lower light intensities such as those encountered in everyday life.