Natural images are reliably represented by sparse and variable populations of neurons in visual cortex

The cortical critical power law balances energy and information in an optimal fashion

A recent study has suggested that the stimulus responses of cortical neural populations follow a critical power law. More precisely, the power spectrum of the covariance matrix of neural responses follows a power law with an exponent indicating that the neural manifold lies on the edge of differentiability. This criticality is hypothesized to balance expressivity and robustness in neural encoding, as population responses on a nondifferential fractal manifold are thought to be overly sensitive to perturbations. However, contrary to this hypothesis, we prove that neural coding is far more robust than previously assumed. We develop a theoretical framework that provides an analytical expression for the Fisher information of population coding under the small noise assumption. Our results reveal that, due to its intrinsic high dimensionality, population coding maintains reliability even on a nondifferentiable fractal manifold, despite its sensitivity to perturbations. Furthermore, the theory reveals that the trade-off between energetic cost and information makes the critical power-law coding the optimal neural encoding of sensory information for a wide range of conditions. In this derivation, we highlight the essential role of a neural correlation, known as differential correlation, in power-law population coding. By uncovering the nontrivial nature of high-dimensional information coding, this work deepens our understanding of criticality and power laws in both biological and artificial neural computation.

Tensor-powered insights into neural dynamics

The complex spatiotemporal dynamics of neurons encompass a wealth of information relevant to perception and decision-making, making the decoding of neural activity a central focus in neuroscience research. Traditional machine learning or deep learning-based neural information modeling approaches have achieved significant results in decoding. Nevertheless, such methodologies require the vectorization of data, a process that disrupts the intrinsic relationships inherent in high-dimensional spaces, consequently impeding their capability to effectively process information in high-order tensor domains. In this paper, we introduce a novel decoding approach, namely the Least Squares Sport Tensor Machine (LS-STM), which is based on tensor space and represents a tensorized improvement over traditional vector learning frameworks. In extensive evaluations using human and mouse data, our results demonstrate that LS-STM exhibits superior performance in neural signal decoding tasks compared to traditional vectorization-based decoding methods. Furthermore, LS-STM demonstrates better performance in decoding neural signals with limited samples and the tensor weights of the LS-STM decoder enable the retrospective identification of key neurons during the neural encoding process. This study introduces a novel tensor computing approach and perspective for decoding high-dimensional neural information in the field.

Coordinating multiple mental faculties during learning

Complex behavior is supported by the coordination of multiple brain regions. How do brain regions coordinate absent a homunculus? We propose coordination is achieved by a controller-peripheral architecture in which peripherals (e.g., the ventral visual stream) aim to supply needed inputs to their controllers (e.g., the hippocampus and prefrontal cortex) while expending minimal resources. We developed a formal model within this framework to address how multiple brain regions coordinate to support rapid learning from a few example images. The model captured how higher-level activity in the controller shaped lower-level visual representations, affecting their precision and sparsity in a manner that paralleled brain measures. In particular, the peripheral encoded visual information to the extent needed to support the smooth operation of the controller. Alternative models optimized by gradient descent irrespective of architectural constraints could not account for human behavior or brain responses, and, typical of standard deep learning approaches, were unstable trial-by-trial learners. While previous work offered accounts of specific faculties, such as perception, attention, and learning, the controller-peripheral approach is a step toward addressing next generation questions concerning how multiple faculties coordinate.

Robust Decoding of Rich Dynamical Visual Scenes With Retinal Spikes

Sensory information transmitted to the brain activates neurons to create a series of coping behaviors. Understanding the mechanisms of neural computation and reverse engineering the brain to build intelligent machines requires establishing a robust relationship between stimuli and neural responses. Neural decoding aims to reconstruct the original stimuli that trigger neural responses. With the recent upsurge of artificial intelligence, neural decoding provides an insightful perspective for designing novel algorithms of brain-machine interface. For humans, vision is the dominant contributor to the interaction between the external environment and the brain. In this study, utilizing the retinal neural spike data collected over multi trials with visual stimuli of two movies with different levels of scene complexity, we used a neural network decoder to quantify the decoded visual stimuli with six different metrics for image quality assessment establishing comprehensive inspection of decoding. With the detailed and systematical study of the effect and single and multiple trials of data, different noise in spikes, and blurred images, our results provide an in-depth investigation of decoding dynamical visual scenes using retinal spikes. These results provide insights into the neural coding of visual scenes and services as a guideline for designing next-generation decoding algorithms of neuroprosthesis and other devices of brain-machine interface.

A novel DRL-guided sparse voxel decoding model for reconstructing perceived images from brain activity

BACKGROUND

Due to the sparse encoding character of the human visual cortex and the scarcity of paired training samples for {images, fMRIs}, voxel selection is an effective means of reconstructing perceived images from fMRI. However, the existing data-driven voxel selection methods have not achieved satisfactory results.

NEW METHOD

Here, a novel deep reinforcement learning-guided sparse voxel (DRL-SV) decoding model is proposed to reconstruct perceived images from fMRI. We innovatively describe voxel selection as a Markov decision process (MDP), training agents to select voxels that are highly involved in specific visual encoding.

RESULTS

Experimental results on two public datasets verify the effectiveness of the proposed DRL-SV, which can accurately select voxels highly involved in neural encoding, thereby improving the quality of visual image reconstruction.

COMPARISON WITH EXISTING METHODS

We qualitatively and quantitatively compared our results with the state-of-the-art (SOTA) methods, getting better reconstruction results. We compared the proposed DRL-SV with traditional data-driven baseline methods, obtaining sparser voxel selection results, but better reconstruction performance.

CONCLUSIONS

DRL-SV can accurately select voxels involved in visual encoding on few-shot, compared to data-driven voxel selection methods. The proposed decoding model provides a new avenue to improving the image reconstruction quality of the primary visual cortex.

Identification of movie encoding neurons enables movie recognition AI

Natural visual scenes are dominated by spatiotemporal image dynamics, but how the visual system integrates "movie" information over time is unclear. We characterized optic tectal neuronal receptive fields using sparse noise stimuli and reverse correlation analysis. Neurons recognized movies of ~200-600 ms durations with defined start and stop stimuli. Movie durations from start to stop responses were tuned by sensory experience though a hierarchical algorithm. Neurons encoded families of image sequences following trigonometric functions. Spike sequence and information flow suggest that repetitive circuit motifs underlie movie detection. Principles of frog topographic retinotectal plasticity and cortical simple cells are employed in machine learning networks for static image recognition, suggesting that discoveries of principles of movie encoding in the brain, such as how image sequences and duration are encoded, may benefit movie recognition technology. We built and trained a machine learning network that mimicked neural principles of visual system movie encoders. The network, named MovieNet, outperformed current machine learning image recognition networks in classifying natural movie scenes, while reducing data size and steps to complete the classification task. This study reveals how movie sequences and time are encoded in the brain and demonstrates that brain-based movie processing principles enable efficient machine learning.

Mesoscale neuronal granular trial variability in vivo illustrated by nonlinear recurrent network in silico

Large-scale neural recording with single-neuron resolution has revealed the functional complexity of the neural systems. However, even under well-designed task conditions, the cortex-wide network exhibits highly dynamic trial variability, posing challenges to the conventional trial-averaged analysis. To study mesoscale trial variability, we conducted a comparative study between fluorescence imaging of layer-2/3 neurons in vivo and network simulation in silico. We imaged up to 40,000 cortical neurons' triggered responses by deep brain stimulus (DBS). And we build an in silico network to reproduce the biological phenomena we observed in vivo. We proved the existence of ineluctable trial variability and found it influenced by input amplitude and range. Moreover, we demonstrated that a spatially heterogeneous coding community accounts for more reliable inter-trial coding despite single-unit trial variability. A deeper understanding of trial variability from the perspective of a dynamical system may lead to uncovering intellectual abilities such as parallel coding and creativity.

Robust entropy rate estimation for nonstationary neuronal calcium spike trains based on empirical probabilities

Objective. Temporal patterns in neuronal spiking encode stimulus uncertainty, and convey information about high-level functions such as memory and cognition. Estimating the associated information content and understanding how that evolves with time assume significance in the investigation of neuronal coding mechanisms and abnormal signaling. However, existing estimators of the entropy rate, a measure of information content, either ignore the inherent nonstationarity, or employ dictionary-based Lempel-Ziv (LZ) methods that converge too slowly for one to study temporal variations in sufficient detail. Against this backdrop, we seek estimates that handle nonstationarity, are fast converging, and hence allow meaningful temporal investigations.Approach. We proposed a homogeneous Markov model approximation of spike trains within windows of suitably chosen length and an entropy rate estimator based on empirical probabilities that converges quickly.Main results. We constructed mathematical families of nonstationary Markov processes with certain bi/multi-level properties (inspired by neuronal responses) with known entropy rates, and validated the proposed estimator against those. Further statistical validations were presented on data collected from hippocampal (and primary visual cortex) neuron populations in terms of single neuron behavior as well as population heterogeneity. Our estimator appears to be statistically more accurate and converges faster than existing LZ estimators, and hence well suited for temporal studies.Significance. The entropy rate analysis revealed not only informational and process memory heterogeneity among neurons, but distinct statistical patterns in neuronal populations (from two different brain regions) under basal and post-stimulus conditions. Taking inspiration, we envision future large-scale studies of different brain regions enabled by the proposed tool (estimator), potentially contributing to improved functional modeling of the brain and identification of statistical signatures of neurodegenerative diseases.

A topological deep learning framework for neural spike decoding

The brain's spatial orientation system uses different neuron ensembles to aid in environment-based navigation. Two of the ways brains encode spatial information are through head direction cells and grid cells. Brains use head direction cells to determine orientation, whereas grid cells consist of layers of decked neurons that overlay to provide environment-based navigation. These neurons fire in ensembles where several neurons fire at once to activate a single head direction or grid. We want to capture this firing structure and use it to decode head direction and animal location from head direction and grid cell activity. Understanding, representing, and decoding these neural structures require models that encompass higher-order connectivity, more than the one-dimensional connectivity that traditional graph-based models provide. To that end, in this work, we develop a topological deep learning framework for neural spike train decoding. Our framework combines unsupervised simplicial complex discovery with the power of deep learning via a new architecture we develop herein called a simplicial convolutional recurrent neural network. Simplicial complexes, topological spaces that use not only vertices and edges but also higher-dimensional objects, naturally generalize graphs and capture more than just pairwise relationships. Additionally, this approach does not require prior knowledge of the neural activity beyond spike counts, which removes the need for similarity measurements. The effectiveness and versatility of the simplicial convolutional neural network is demonstrated on head direction and trajectory prediction via head direction and grid cell datasets.

Decoding dynamic visual scenes across the brain hierarchy

Understanding the computational mechanisms that underlie the encoding and decoding of environmental stimuli is a crucial investigation in neuroscience. Central to this pursuit is the exploration of how the brain represents visual information across its hierarchical architecture. A prominent challenge resides in discerning the neural underpinnings of the processing of dynamic natural visual scenes. Although considerable research efforts have been made to characterize individual components of the visual pathway, a systematic understanding of the distinctive neural coding associated with visual stimuli, as they traverse this hierarchical landscape, remains elusive. In this study, we leverage the comprehensive Allen Visual Coding-Neuropixels dataset and utilize the capabilities of deep learning neural network models to study neural coding in response to dynamic natural visual scenes across an expansive array of brain regions. Our study reveals that our decoding model adeptly deciphers visual scenes from neural spiking patterns exhibited within each distinct brain area. A compelling observation arises from the comparative analysis of decoding performances, which manifests as a notable encoding proficiency within the visual cortex and subcortical nuclei, in contrast to a relatively reduced encoding activity within hippocampal neurons. Strikingly, our results unveil a robust correlation between our decoding metrics and well-established anatomical and functional hierarchy indexes. These findings corroborate existing knowledge in visual coding related to artificial visual stimuli and illuminate the functional role of these deeper brain regions using dynamic stimuli. Consequently, our results suggest a novel perspective on the utility of decoding neural network models as a metric for quantifying the encoding quality of dynamic natural visual scenes represented by neural responses, thereby advancing our comprehension of visual coding within the complex hierarchy of the brain.

Basic Properties of Coordinated Neuronal Ensembles in the Auditory Thalamus

Coordinated neuronal activity has been identified to play an important role in information processing and transmission in the brain. However, current research predominantly focuses on understanding the properties and functions of neuronal coordination in hippocampal and cortical areas, leaving subcortical regions relatively unexplored. In this study, we use single-unit recordings in female Sprague Dawley rats to investigate the properties and functions of groups of neurons exhibiting coordinated activity in the auditory thalamus-the medial geniculate body (MGB). We reliably identify coordinated neuronal ensembles (cNEs), which are groups of neurons that fire synchronously, in the MGB. cNEs are shown not to be the result of false-positive detections or by-products of slow-state oscillations in anesthetized animals. We demonstrate that cNEs in the MGB have enhanced information-encoding properties over individual neurons. Their neuronal composition is stable between spontaneous and evoked activity, suggesting limited stimulus-induced ensemble dynamics. These MGB cNE properties are similar to what is observed in cNEs in the primary auditory cortex (A1), suggesting that ensembles serve as a ubiquitous mechanism for organizing local networks and play a fundamental role in sensory processing within the brain.

Population coding of time-varying sounds in the nonlemniscal inferior colliculus

The inferior colliculus (IC) of the midbrain is important for complex sound processing, such as discriminating conspecific vocalizations and human speech. The IC's nonlemniscal, dorsal "shell" region is likely important for this process, as neurons in these layers project to higher-order thalamic nuclei that subsequently funnel acoustic signals to the amygdala and nonprimary auditory cortices, forebrain circuits important for vocalization coding in a variety of mammals, including humans. However, the extent to which shell IC neurons transmit acoustic features necessary to discern vocalizations is less clear, owing to the technical difficulty of recording from neurons in the IC's superficial layers via traditional approaches. Here, we use two-photon Ca2+ imaging in mice of either sex to test how shell IC neuron populations encode the rate and depth of amplitude modulation, important sound cues for speech perception. Most shell IC neurons were broadly tuned, with a low neurometric discrimination of amplitude modulation rate; only a subset was highly selective to specific modulation rates. Nevertheless, neural network classifier trained on fluorescence data from shell IC neuron populations accurately classified amplitude modulation rate, and decoding accuracy was only marginally reduced when highly tuned neurons were omitted from training data. Rather, classifier accuracy increased monotonically with the modulation depth of the training data, such that classifiers trained on full-depth modulated sounds had median decoding errors of ∼0.2 octaves. Thus, shell IC neurons may transmit time-varying signals via a population code, with perhaps limited reliance on the discriminative capacity of any individual neuron.NEW & NOTEWORTHY The IC's shell layers originate a "nonlemniscal" pathway important for perceiving vocalization sounds. However, prior studies suggest that individual shell IC neurons are broadly tuned and have high response thresholds, implying a limited reliability of efferent signals. Using Ca2+ imaging, we show that amplitude modulation is accurately represented in the population activity of shell IC neurons. Thus, downstream targets can read out sounds' temporal envelopes from distributed rate codes transmitted by populations of broadly tuned neurons.

Neural populations within macaque early vestibular pathways are adapted to encode natural self-motion

How the activities of large neural populations are integrated in the brain to ensure accurate perception and behavior remains a central problem in systems neuroscience. Here, we investigated population coding of naturalistic self-motion by neurons within early vestibular pathways in rhesus macaques (Macacca mulatta). While vestibular neurons displayed similar dynamic tuning to self-motion, inspection of their spike trains revealed significant heterogeneity. Further analysis revealed that, during natural but not artificial stimulation, heterogeneity resulted primarily from variability across neurons as opposed to trial-to-trial variability. Interestingly, vestibular neurons displayed different correlation structures during naturalistic and artificial self-motion. Specifically, while correlations due to the stimulus (i.e., signal correlations) did not differ, correlations between the trial-to-trial variabilities of neural responses (i.e., noise correlations) were instead significantly positive during naturalistic but not artificial stimulation. Using computational modeling, we show that positive noise correlations during naturalistic stimulation benefits information transmission by heterogeneous vestibular neural populations. Taken together, our results provide evidence that neurons within early vestibular pathways are adapted to the statistics of natural self-motion stimuli at the population level. We suggest that similar adaptations will be found in other systems and species.

The lncRNA lnc_AABR07044470.1 promotes the mitochondrial-damaged inflammatory response to neuronal injury via miR-214-3p/PERM1 axis in acute ischemic stroke

PURPOSE

We investigated the role of lnc_AABR07044470.1 on the occurrence and development of acute ischemic stroke (AIS) and neuronal injury by targeting the miR-214-3p/PERM1 axis to find a novel clinical drug target and prediction and treatment of AIS.

METHODS

The mouse AIS animal model was used in vivo experiments and hypoxia/reoxygenation cell model in vitro was established. Firstly, infarction volume and pathological changes of mouse hippocampal neurons were detected using HE staining. Secondly, rat primary neuron apoptosis was detected by flow cytometry assay. The numbers of neuron, microglia and astrocytes were detected using immunofluorescence (IF). Furthermore, binding detection was performed by bioinformatics database and double luciferase reporter assay. Lnc_AABR07044470.1 localization was performed using fluorescence in situ hybridization (FISH).Lnc_AABR07044470.1, miR-214-3pand PERM1mRNA expression was performed using RT-qPCR. NLRP3, ASC, Caspase-1 and PERM1 protein expression was performed using Western blotting. IL-1β was detected by ELISA assay.

RESULTS

Mouse four-vessel occlusion could easily establish the animal model, and AIS animal model had an obvious time-dependence. HE staining showed that, compared with the sham group, infarction volume and pathological changes of mouse hippocampal neurons were deteriorated in the model group. Furthermore, compared with the sham group, neurons were significantly reduced, while microglia and astrocytes were significantly activated. Moreover, the bioinformatics prediction and detection of double luciferase reporter confirmed the binding site of lnc_AABR07044470.1 to miR-214-3p and miR-214-3p to Perm1. lnc_AABR07044470.1 and PERM1 expression was significantly down-regulated and miR-214-3pexpression was significantly up-regulated in AIS animal model in vivo. At the same time, the expression of inflammasome NLRP3, ASC, Caspase-1 and pro-inflammatory factor IL-1β was significantly up-regulated in vivo and in vitro. The over-expression of lnc_AABR07044470.1 and miR-214-3p inhibitor could inhibit the neuron apoptosis and the expression of inflammasome NLRP3, ASC, Caspase-1 and pro-inflammatory factor IL-1β and up-regulate the expression of PERM1 in vitro. Finally, over-expression of lnc_AABR07044470.1 and miR-214-3p inhibitor transfected cell model was significant in relieving the AIS and neuronal injury.

CONCLUSION

Lnc_AABR07044470.1 promotes inflammatory response to neuronal injury via miR-214-3p/PERM1 axis in AIS.

Fractured columnar small-world functional network organization in volumes of L2/3 of mouse auditory cortex

The sensory cortices of the brain exhibit large-scale functional topographic organization, such as the tonotopic organization of the primary auditory cortex (A1) according to sound frequency. However, at the level of individual neurons, layer 2/3 (L2/3) A1 appears functionally heterogeneous. To identify if there exists a higher-order functional organization of meso-scale neuronal networks within L2/3 that bridges order and disorder, we used in vivo two-photon calcium imaging of pyramidal neurons to identify networks in three-dimensional volumes of L2/3 A1 in awake mice. Using tonal stimuli, we found diverse receptive fields with measurable colocalization of similarly tuned neurons across depth but less so across L2/3 sublayers. These results indicate a fractured microcolumnar organization with a column radius of ∼50 µm, with a more random organization of the receptive field over larger radii. We further characterized the functional networks formed within L2/3 by analyzing the spatial distribution of signal correlations (SCs). Networks show evidence of Rentian scaling in physical space, suggesting effective spatial embedding of subnetworks. Indeed, functional networks have characteristics of small-world topology, implying that there are clusters of functionally similar neurons with sparse connections between differently tuned neurons. These results indicate that underlying the regularity of the tonotopic map on large scales in L2/3 is significant tuning diversity arranged in a hybrid organization with microcolumnar structures and efficient network topologies.

Performance Evaluation of Matrix Factorization for fMRI Data

A hypothesis in the study of the brain is that sparse coding is realized in information representation of external stimuli, which has been experimentally confirmed for visual stimulus recently. However, unlike the specific functional region in the brain, sparse coding in information processing in the whole brain has not been clarified sufficiently. In this study, we investigate the validity of sparse coding in the whole human brain by applying various matrix factorization methods to functional magnetic resonance imaging data of neural activities in the brain. The result suggests the sparse coding hypothesis in information representation in the whole human brain, because extracted features from the sparse matrix factorization (MF) method, sparse principal component analysis (SparsePCA), or method of optimal directions (MOD) under a high sparsity setting or an approximate sparse MF method, fast independent component analysis (FastICA), can classify external visual stimuli more accurately than the nonsparse MF method or sparse MF method under a low sparsity setting.

Adversarial attacks and defenses using feature-space stochasticity

Recent studies in deep neural networks have shown that injecting random noise in the input layer of the networks contributes towards ℓp-norm-bounded adversarial perturbations. However, to defend against unrestricted adversarial examples, most of which are not ℓp-norm-bounded in the input layer, such input-layer random noise may not be sufficient. In the first part of this study, we generated a novel class of unrestricted adversarial examples termed feature-space adversarial examples. These examples are far from the original data in the input space but adjacent to the original data in a hidden-layer feature space and far again in the output layer. In the second part of this study, we empirically showed that while injecting random noise in the input layer was unable to defend these feature-space adversarial examples, they were defended by injecting random noise in the hidden layer. These results highlight the novel benefit of stochasticity in higher layers, in that it is useful for defending against these feature-space adversarial examples, a class of unrestricted adversarial examples.

Beyond ℓ1 sparse coding in V1

Growing evidence indicates that only a sparse subset from a pool of sensory neurons is active for the encoding of visual stimuli at any instant in time. Traditionally, to replicate such biological sparsity, generative models have been using the ℓ1 norm as a penalty due to its convexity, which makes it amenable to fast and simple algorithmic solvers. In this work, we use biological vision as a test-bed and show that the soft thresholding operation associated to the use of the ℓ1 norm is highly suboptimal compared to other functions suited to approximating ℓp with 0 ≤ p < 1 (including recently proposed continuous exact relaxations), in terms of performance. We show that ℓ1 sparsity employs a pool with more neurons, i.e. has a higher degree of overcompleteness, in order to maintain the same reconstruction error as the other methods considered. More specifically, at the same sparsity level, the thresholding algorithm using the ℓ1 norm as a penalty requires a dictionary of ten times more units compared to the proposed approach, where a non-convex continuous relaxation of the ℓ0 pseudo-norm is used, to reconstruct the external stimulus equally well. At a fixed sparsity level, both ℓ0- and ℓ1-based regularization develop units with receptive field (RF) shapes similar to biological neurons in V1 (and a subset of neurons in V2), but ℓ0-based regularization shows approximately five times better reconstruction of the stimulus. Our results in conjunction with recent metabolic findings indicate that for V1 to operate efficiently it should follow a coding regime which uses a regularization that is closer to the ℓ0 pseudo-norm rather than the ℓ1 one, and suggests a similar mode of operation for the sensory cortex in general.

Population coding of time-varying sounds in the non-lemniscal Inferior Colliculus

The inferior colliculus (IC) of the midbrain is important for complex sound processing, such as discriminating conspecific vocalizations and human speech. The IC's non-lemniscal, dorsal "shell" region is likely important for this process, as neurons in these layers project to higher-order thalamic nuclei that subsequently funnel acoustic signals to the amygdala and non-primary auditory cortices; forebrain circuits important for vocalization coding in a variety of mammals, including humans. However, the extent to which shell IC neurons transmit acoustic features necessary to discern vocalizations is less clear, owing to the technical difficulty of recording from neurons in the IC's superficial layers via traditional approaches. Here we use 2-photon Ca2+ imaging in mice of either sex to test how shell IC neuron populations encode the rate and depth of amplitude modulation, important sound cues for speech perception. Most shell IC neurons were broadly tuned, with a low neurometric discrimination of amplitude modulation rate; only a subset were highly selective to specific modulation rates. Nevertheless, neural network classifier trained on fluorescence data from shell IC neuron populations accurately classified amplitude modulation rate, and decoding accuracy was only marginally reduced when highly tuned neurons were omitted from training data. Rather, classifier accuracy increased monotonically with the modulation depth of the training data, such that classifiers trained on full-depth modulated sounds had median decoding errors of ~0.2 octaves. Thus, shell IC neurons may transmit time-varying signals via a population code, with perhaps limited reliance on the discriminative capacity of any individual neuron.

Intersecting distributed networks support convergent linguistic functioning across different languages in bilinguals

How bilingual brains accomplish the processing of more than one language has been widely investigated by neuroimaging studies. The assimilation-accommodation hypothesis holds that both the same brain neural networks supporting the native language and additional new neural networks are utilized to implement second language processing. However, whether and how this hypothesis applies at the finer-grained levels of both brain anatomical organization and linguistic functions remains unknown. To address this issue, we scanned Chinese-English bilinguals during an implicit reading task involving Chinese words, English words and Chinese pinyin. We observed broad brain cortical regions wherein interdigitated distributed neural populations supported the same cognitive components of different languages. Although spatially separate, regions including the opercular and triangular parts of the inferior frontal gyrus, temporal pole, superior and middle temporal gyrus, precentral gyrus and supplementary motor areas were found to perform the same linguistic functions across languages, indicating regional-level functional assimilation supported by voxel-wise anatomical accommodation. Taken together, the findings not only verify the functional independence of neural representations of different languages, but show co-representation organization of both languages in most language regions, revealing linguistic-feature specific accommodation and assimilation between first and second languages.

Do sparse brain activity patterns underlie human cognition?

Accumulating multivariate pattern analysis (MVPA) results from fMRI studies suggest that information is represented in fingerprint patterns of activations and deactivations during perception, emotions, and cognition. We postulate that these fingerprint patterns might reflect neuronal-population level sparse code documented in two-photon calcium imaging studies in animal models, i.e., information represented in specific and reproducible ensembles of a few percent of active neurons amidst widespread inhibition in neural populations. We suggest that such representations constitute a fundamental organizational principle via interacting across multiple levels of brain hierarchy, thus giving rise to perception, emotions, and cognition.

Neural Decoding of Visual Information Across Different Neural Recording Modalities and Approaches

Vision plays a peculiar role in intelligence. Visual information, forming a large part of the sensory information, is fed into the human brain to formulate various types of cognition and behaviours that make humans become intelligent agents. Recent advances have led to the development of brain-inspired algorithms and models for machine vision. One of the key components of these methods is the utilization of the computational principles underlying biological neurons. Additionally, advanced experimental neuroscience techniques have generated different types of neural signals that carry essential visual information. Thus, there is a high demand for mapping out functional models for reading out visual information from neural signals. Here, we briefly review recent progress on this issue with a focus on how machine learning techniques can help in the development of models for contending various types of neural signals, from fine-scale neural spikes and single-cell calcium imaging to coarse-scale electroencephalography (EEG) and functional magnetic resonance imaging recordings of brain signals.

Contribution of behavioural variability to representational drift

Neuronal responses to similar stimuli change dynamically over time, raising the question of how internal representations can provide a stable substrate for neural coding. Recent work has suggested a large degree of drift in neural representations even in sensory cortices, which are believed to store stable representations of the external world. While the drift of these representations is mostly characterized in relation to external stimuli, the behavioural state of the animal (for instance, the level of arousal) is also known to strongly modulate the neural activity. We therefore asked how the variability of such modulatory mechanisms can contribute to representational changes. We analysed large-scale recording of neural activity from the Allen Brain Observatory, which was used before to document representational drift in the mouse visual cortex. We found that, within these datasets, behavioural variability significantly contributes to representational changes. This effect was broadcasted across various cortical areas in the mouse, including the primary visual cortex, higher order visual areas, and even regions not primarily linked to vision like hippocampus. Our computational modelling suggests that these results are consistent with independent modulation of neural activity by behaviour over slower timescales. Importantly, our analysis suggests that reliable but variable modulation of neural representations by behaviour can be misinterpreted as representational drift if neuronal representations are only characterized in the stimulus space and marginalized over behavioural parameters.

Selective representations of texture and motion in mouse higher visual areas

The mouse visual cortex contains interconnected higher visual areas, but their functional specializations are unclear. Here, we used a data-driven approach to examine the representations of complex visual stimuli by L2/3 neurons across mouse higher visual areas, measured using large-field-of-view two-photon calcium imaging. Using specialized stimuli, we found higher fidelity representations of texture in area LM, compared to area AL. Complementarily, we found higher fidelity representations of motion in area AL, compared to area LM. We also observed this segregation of information in response to naturalistic videos. Finally, we explored how receptive field models of visual cortical neurons could produce the segregated representations of texture and motion we observed. These selective representations could aid in behaviors such as visually guided navigation.

Decoding Pixel-Level Image Features from Two-Photon Calcium Signals of Macaque Visual Cortex

Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.

Brain-wide projection reconstruction of single functionally defined neurons

Reconstructing axonal projections of single neurons at the whole-brain level is currently a converging goal of the neuroscience community that is fundamental for understanding the logic of information flow in the brain. Thousands of single neurons from different brain regions have recently been morphologically reconstructed, but the corresponding physiological functional features of these reconstructed neurons are unclear. By combining two-photon Ca²⁺ imaging with targeted single-cell plasmid electroporation, we reconstruct the brain-wide morphologies of single neurons that are defined by a sound-evoked response map in the auditory cortices (AUDs) of awake mice. Long-range interhemispheric projections can be reliably labelled via co-injection with an adeno-associated virus, which enables enhanced expression of indicator protein in the targeted neurons. Here we show that this method avoids the randomness and ambiguity of conventional methods of neuronal morphological reconstruction, offering an avenue for developing a precise one-to-one map of neuronal projection patterns and physiological functional features.

Brain-wide axonal projections of single neurons have been extensively reconstructed without any functional characterization. The authors present a method that allows for developing a precise one-to-one map of both projection patterns and functional features of single neurons in mice.

Luminance and Contrast of Images in the THINGS Database

The THINGS database is a freely available stimulus set that has the potential to facilitate the generation of theory that bridges multiple areas within cognitive neuroscience. The database consists of 26,107 high quality digital photos that are sorted into 1,854 concepts. While a valuable resource, relatively few technical details relevant to the design of studies in cognitive neuroscience have been described. We present an analysis of two key low-level properties of THINGS images, luminance and luminance contrast. These image statistics are known to influence common physiological and neural correlates of perceptual and cognitive processes. In general, we found that the distributions of luminance and contrast are in close agreement with the statistics of natural images reported previously. However, we found that image concepts are separable in their luminance and contrast: we show that luminance and contrast alone are sufficient to classify images into their concepts with above chance accuracy. We describe how these factors may confound studies using the THINGS images, and suggest simple controls that can be implemented a priori or post-hoc. We discuss the importance of using such natural images as stimuli in psychological research.

Inference via sparse coding in a hierarchical vision model

Sparse coding has been incorporated in models of the visual cortex for its computational advantages and connection to biology. But how the level of sparsity contributes to performance on visual tasks is not well understood. In this work, sparse coding has been integrated into an existing hierarchical V2 model (Hosoya & Hyvärinen, 2015), but replacing its independent component analysis (ICA) with an explicit sparse coding in which the degree of sparsity can be controlled. After training, the sparse coding basis functions with a higher degree of sparsity resembled qualitatively different structures, such as curves and corners. The contributions of the models were assessed with image classification tasks, specifically tasks associated with mid-level vision including figure–ground classification, texture classification, and angle prediction between two line stimuli. In addition, the models were assessed in comparison with a texture sensitivity measure that has been reported in V2 (Freeman et al., 2013) and a deleted-region inference task. The results from the experiments show that although sparse coding performed worse than ICA at classifying images, only sparse coding was able to better match the texture sensitivity level of V2 and infer deleted image regions, both by increasing the degree of sparsity in sparse coding. Greater degrees of sparsity allowed for inference over larger deleted image regions. The mechanism that allows for this inference capability in sparse coding is described in this article.

The contribution of low contrast-preferring neurons to information representation in the primary visual cortex after learning

Animals exhibit improved perception of lower-contrast visual objects after training. We explored this neuronal mechanism using multiple single-unit recordings from deep layers of the primary visual cortex (V1) of trained rats during orientation discrimination. We found that the firing rates of a subset of neurons increased by reducing luminance contrast, being at least above basal activities at low contrast. These low contrast–preferring neurons were rare during passive viewing without training or anesthesia after training. They fired more frequently in correct-choice than incorrect-choice trials. At single-neuron and population levels, they efficiently represented low-contrast orientations. Following training, in addition to generally enhanced excitation, the phase synchronization of spikes to beta oscillations at high contrast was stronger in putative inhibitory than excitatory neurons. The change in excitation-inhibition balance might contribute to low-contrast preference. Thus, low-contrast preference in V1 activity is strengthened in an experience-dependent manner, which may contribute to low-contrast visual discrimination.

Excitatory-inhibitory balance modulates the formation and dynamics of neuronal assemblies in cortical networks

Repetitive activation of subpopulations of neurons leads to the formation of neuronal assemblies, which can guide learning and behavior. Recent technological advances have made the artificial induction of these assemblies feasible, yet how various parameters of induction can be optimized is not clear. Here, we studied this question in large-scale cortical network models with excitatory-inhibitory balance. We found that the background network in which assemblies are embedded can strongly modulate their dynamics and formation. Networks with dominant excitatory interactions enabled a fast formation of assemblies, but this was accompanied by recruitment of other non-perturbed neurons, leading to some degree of nonspecific induction. On the other hand, networks with strong excitatory-inhibitory interactions ensured that the formation of assemblies remained constrained to the perturbed neurons, but slowed down the induction. Our results suggest that these two regimes can be suitable for computational and cognitive tasks with different trade-offs between speed and specificity.

Looking down: a model for visual route following in flying insects

Insect visual navigation is often assumed to depend on panoramic views of the horizon, and how these change as the animal moves. However, it is known that honey bees can visually navigate in flat, open meadows where visual information at the horizon is minimal, or would remain relatively constant across a wide range of positions. In this paper we hypothesise that these animals can navigate using view memories of the ground. We find that in natural scenes, low resolution views from an aerial perspective of ostensibly self-similar terrain (e.g. within a field of grass) provide surprisingly robust descriptors of precise spatial locations. We propose a new visual route following approach that makes use of transverse oscillations to centre a flight path along a sequence of learned views of the ground. We deploy this model on an autonomous quadcopter and demonstrate that it provides robust performance in the real world on journeys of up to 30 m. The success of our method is contingent on a robust view matching process which can evaluate the familiarity of a view with a degree of translational invariance. We show that a previously developed wavelet based bandpass orientated filter approach fits these requirements well, exhibiting double the catchment area of standard approaches. Using a realistic simulation package, we evaluate the robustness of our approach to variations in heading direction and aircraft height between inbound and outbound journeys. We also demonstrate that our approach can operate using a vision system with a biologically relevant visual acuity and viewing direction.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Synergistic population coding of natural communication stimuli by hindbrain electrosensory neurons

Understanding how neural populations encode natural stimuli with complex spatiotemporal structure to give rise to perception remains a central problem in neuroscience. Here we investigated population coding of natural communication stimuli by hindbrain neurons within the electrosensory system of weakly electric fish Apteronotus leptorhynchus. Overall, we found that simultaneously recorded neural activities were correlated: signal but not noise correlations were variable depending on the stimulus waveform as well as the distance between neurons. Combining the neural activities using an equal-weight sum gave rise to discrimination performance between different stimulus waveforms that was limited by redundancy introduced by noise correlations. However, using an evolutionary algorithm to assign different weights to individual neurons before combining their activities (i.e., a weighted sum) gave rise to increased discrimination performance by revealing synergistic interactions between neural activities. Our results thus demonstrate that correlations between the neural activities of hindbrain electrosensory neurons can enhance information about the structure of natural communication stimuli that allow for reliable discrimination between different waveforms by downstream brain areas.

Population Coding of Natural Electrosensory Stimuli by Midbrain Neurons

Natural stimuli display spatiotemporal characteristics that typically vary over orders of magnitude, and their encoding by sensory neurons remains poorly understood. We investigated population coding of highly heterogeneous natural electrocommunication stimuli in Apteronotus leptorhynchus of either sex. Neuronal activities were positively correlated with one another in the absence of stimulation, and correlation magnitude decayed with increasing distance between recording sites. Under stimulation, we found that correlations between trial-averaged neuronal responses (i.e., signal correlations) were positive and higher in magnitude for neurons located close to another, but that correlations between the trial-to-trial variability (i.e., noise correlations) were independent of physical distance. Overall, signal and noise correlations were independent of stimulus waveform as well as of one another. To investigate how neuronal populations encoded natural electrocommunication stimuli, we considered a nonlinear decoder for which the activities were combined. Decoding performance was best for a timescale of 6 ms, indicating that midbrain neurons transmit information via precise spike timing. A simple summation of neuronal activities (equally weighted sum) revealed that noise correlations limited decoding performance by introducing redundancy. Using an evolution algorithm to optimize performance when considering instead unequally weighted sums of neuronal activities revealed much greater performance values, indicating that midbrain neuron populations transmit information that reliably enable discrimination between different stimulus waveforms. Interestingly, we found that different weight combinations gave rise to similar discriminability, suggesting robustness. Our results have important implications for understanding how natural stimuli are integrated by downstream brain areas to give rise to behavioral responses.SIGNIFICANCE STATEMENT We show that midbrain electrosensory neurons display correlations between their activities and that these can significantly impact performance of decoders. While noise correlations limited discrimination performance by introducing redundancy, considering unequally weighted sums of neuronal activities gave rise to much improved performance and mitigated the deleterious effects of noise correlations. Further analysis revealed that increased discriminability was achieved by making trial-averaged responses more separable, as well as by reducing trial-to-trial variability by eliminating noise correlations. We further found that multiple combinations of weights could give rise to similar discrimination performances, which suggests that such combinatorial codes could be achieved in the brain. We conclude that the activities of midbrain neuronal populations can be used to reliably discriminate between highly heterogeneous stimulus waveforms.

Deep learning in electron microscopy

Deep learning is transforming most areas of science and technology, including electron microscopy. This review paper offers a practical perspective aimed at developers with limited familiarity. For context, we review popular applications of deep learning in electron microscopy. Following, we discuss hardware and software needed to get started with deep learning and interface with electron microscopes. We then review neural network components, popular architectures, and their optimization. Finally, we discuss future directions of deep learning in electron microscopy.

Scaling of sensory information in large neural populations shows signatures of information-limiting correlations

How is information distributed across large neuronal populations within a given brain area? Information may be distributed roughly evenly across neuronal populations, so that total information scales linearly with the number of recorded neurons. Alternatively, the neural code might be highly redundant, meaning that total information saturates. Here we investigate how sensory information about the direction of a moving visual stimulus is distributed across hundreds of simultaneously recorded neurons in mouse primary visual cortex. We show that information scales sublinearly due to correlated noise in these populations. We compartmentalized noise correlations into information-limiting and nonlimiting components, then extrapolate to predict how information grows with even larger neural populations. We predict that tens of thousands of neurons encode 95% of the information about visual stimulus direction, much less than the number of neurons in primary visual cortex. These findings suggest that the brain uses a widely distributed, but nonetheless redundant code that supports recovering most sensory information from smaller subpopulations.

Engrams of Fast Learning

Fast learning designates the behavioral and neuronal mechanisms underlying the acquisition of a long-term memory trace after a unique and brief experience. As such it is opposed to incremental, slower reinforcement or procedural learning requiring repetitive training. This learning process, found in most animal species, exists in a large spectrum of natural behaviors, such as one-shot associative, spatial, or perceptual learning, and is a core principle of human episodic memory. We review here the neuronal and synaptic long-term changes associated with fast learning in mammals and discuss some hypotheses related to their underlying mechanisms. We first describe the variety of behavioral paradigms used to test fast learning memories: those preferentially involve a single and brief (from few hundred milliseconds to few minutes) exposures to salient stimuli, sufficient to trigger a long-lasting memory trace and new adaptive responses. We then focus on neuronal activity patterns observed during fast learning and the emergence of long-term selective responses, before documenting the physiological correlates of fast learning. In the search for the engrams of fast learning, a growing body of evidence highlights long-term changes in gene expression, structural, intrinsic, and synaptic plasticities. Finally, we discuss the potential role of the sparse and bursting nature of neuronal activity observed during the fast learning, especially in the induction plasticity mechanisms leading to the rapid establishment of long-term synaptic modifications. We conclude with more theoretical perspectives on network dynamics that could enable fast learning, with an overview of some theoretical approaches in cognitive neuroscience and artificial intelligence.

Opposing Influence of Top-down and Bottom-up Input on Excitatory Layer 2/3 Neurons in Mouse Primary Visual Cortex

Processing in cortical circuits is driven by combinations of cortical and subcortical inputs. These inputs are often conceptually categorized as bottom-up, conveying sensory information, and top-down, conveying contextual information. Using intracellular recordings in mouse primary visual cortex, we measured neuronal responses to visual input, locomotion, and visuomotor mismatches. We show that layer 2/3 (L2/3) neurons compute a difference between top-down motor-related input and bottom-up visual flow input. Most L2/3 neurons responded to visuomotor mismatch with either hyperpolarization or depolarization, and the size of this response was correlated with distinct physiological properties. Consistent with a subtraction of bottom-up and top-down input, visual and motor-related inputs had opposing influence on L2/3 neurons. In infragranular neurons, we found no evidence of a difference computation and responses were consistent with positive integration of visuomotor inputs. Our results provide evidence that L2/3 functions as a bidirectional comparator of top-down and bottom-up input.

•

Layer 2/3 neurons show widespread subthreshold mismatch responses

•

Mismatch response sign is predicted by visual flow and locomotion-related responses

•

Layer 5/6 has a scarcity of depolarizing mismatch responses

•

Visual flow and locomotion speed have opposing signs of influence only in layer 2/3

Theory of neuronal perturbome in cortical networks

Brains are composed of networks of neurons that are highly interconnected. A central question in neuroscience is how such neuronal networks operate in tandem to make a functioning brain. To understand this, we need to study how neurons interact with each other in action, such as when viewing a visual scene or performing a motor task. One way to approach this question is by perturbing the activity of functioning neurons and measuring the resulting influence on other neurons. By using computational models of neuronal networks, we studied how this influence in visual networks depends on connectivity. Our results help to interpret contradictory results from previous experimental studies and explain how different connectivity patterns can enhance information processing during natural vision.

To unravel the functional properties of the brain, we need to untangle how neurons interact with each other and coordinate in large-scale recurrent networks. One way to address this question is to measure the functional influence of individual neurons on each other by perturbing them in vivo. Application of such single-neuron perturbations in mouse visual cortex has recently revealed feature-specific suppression between excitatory neurons, despite the presence of highly specific excitatory connectivity, which was deemed to underlie feature-specific amplification. Here, we studied which connectivity profiles are consistent with these seemingly contradictory observations, by modeling the effect of single-neuron perturbations in large-scale neuronal networks. Our numerical simulations and mathematical analysis revealed that, contrary to the prima facie assumption, neither inhibition dominance nor broad inhibition alone were sufficient to explain the experimental findings; instead, strong and functionally specific excitatory–inhibitory connectivity was necessary, consistent with recent findings in the primary visual cortex of rodents. Such networks had a higher capacity to encode and decode natural images, and this was accompanied by the emergence of response gain nonlinearities at the population level. Our study provides a general computational framework to investigate how single-neuron perturbations are linked to cortical connectivity and sensory coding and paves the road to map the perturbome of neuronal networks in future studies.