Population codes enable learning from few examples by shaping inductive bias

Random compressed coding with neurons

Classical models of efficient coding in neurons assume simple mean responses-"tuning curves"- such as bell-shaped or monotonic functions of a stimulus feature. Real neurons, however, can be more complex: grid cells, for example, exhibit periodic responses that impart the neural population code with high accuracy. But do highly accurate codes require fine-tuning of the response properties? We address this question with the use of a simple model: a population of neurons with random, spatially extended, and irregular tuning curves. Irregularity enhances the local resolution of the code but gives rise to catastrophic, global errors. For optimal smoothness of the tuning curves, when local and global errors balance out, the neural population compresses information about a continuous stimulus into a low-dimensional representation, and the resulting distributed code achieves exponential accuracy. An analysis of recordings from monkey motor cortex points to such "compressed efficient coding." Efficient codes do not require a finely tuned design-they emerge robustly from irregularity or randomness.

Computing With Residue Numbers in High-Dimensional Representation

We introduce residue hyperdimensional computing, a computing framework that unifies residue number systems with an algebra defined over random, high-dimensional vectors. We show how residue numbers can be represented as high-dimensional vectors in a manner that allows algebraic operations to be performed with component-wise, parallelizable operations on the vector elements. The resulting framework, when combined with an efficient method for factorizing high-dimensional vectors, can represent and operate on numerical values over a large dynamic range using resources that scale only logarithmically with the range, a vast improvement over previous methods. It also exhibits impressive robustness to noise. We demonstrate the potential for this framework to solve computationally difficult problems in visual perception and combinatorial optimization, showing improvement over baseline methods. More broadly, the framework provides a possible account for the computational operations of grid cells in the brain, and it suggests new machine learning architectures for representing and manipulating numerical data.

Compartmentalized pooling generates orientation selectivity in wide-field amacrine cells

Orientation is one of the most salient features in visual scenes. Neurons at multiple levels of the visual system detect orientation, but in many cases, the underlying biophysical mechanisms remain unresolved. Here, we studied mechanisms for orientation detection at the earliest stage in the visual system, in B/K wide-field amacrine cells (B/K WACs), a group of giant, nonspiking interneurons in the mouse retina that coexpress Bhlhe22 (B) and Kappa Opioid Receptor (K). B/K WACs exhibit orientation-tuned calcium signals along their long, straight, unbranching dendrites, which contain both synaptic inputs and outputs. Simultaneous dendritic calcium imaging and somatic voltage recordings reveal that individual B/K dendrites are electrotonically isolated, exhibiting a spatially confined yet extended receptive field along the dendrite, which we term "compartmentalized pooling." Further, the receptive field of a B/K WAC dendrite exhibits center-surround antagonism. Phenomenological receptive field models demonstrate that compartmentalized pooling generates orientation selectivity, and center-surround antagonism shapes band-pass spatial frequency tuning. At the microcircuit level, B/K WACs receive excitation driven by one contrast polarity (e.g., "ON") and glycinergic inhibition driven by the opposite polarity (e.g., "OFF"). However, this "crossover" inhibition is not essential for generating orientation selectivity. A minimal biophysical model reproduced compartmentalized pooling from feedforward excitatory inputs combined with a substantial increase in the specific membrane resistance between somatic and dendritic compartments. Collectively, our results reveal the biophysical mechanism for generating orientation selectivity in dendrites of B/K WACs, enriching our understanding of the diverse strategies employed throughout the visual system to detect orientation.

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles

Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features. Here, we develop a theory of feature-bagging in noisy least-squares ridge ensembles and simplify the resulting learning curves in the special case of equicorrelated data. Using analytical learning curves, we demonstrate that subsampling shifts the double-descent peak of a linear predictor. This leads us to introduce heterogeneous feature ensembling, with estimators built on varying numbers of feature dimensions, as a computationally efficient method to mitigate double-descent. Then, we compare the performance of a feature-subsampling ensemble to a single linear predictor, describing a trade-off between noise amplification due to subsampling and noise reduction due to ensembling. Our qualitative insights carry over to linear classifiers applied to image classification tasks with realistic datasets constructed using a state-of-the-art deep learning feature map.

Computing with Residue Numbers in High-Dimensional Representation

We introduce Residue Hyperdimensional Computing, a computing framework that unifies residue number systems with an algebra defined over random, high-dimensional vectors. We show how residue numbers can be represented as high-dimensional vectors in a manner that allows algebraic operations to be performed with component-wise, parallelizable operations on the vector elements. The resulting framework, when combined with an efficient method for factorizing high-dimensional vectors, can represent and operate on numerical values over a large dynamic range using vastly fewer resources than previous methods, and it exhibits impressive robustness to noise. We demonstrate the potential for this framework to solve computationally difficult problems in visual perception and combinatorial optimization, showing improvement over baseline methods. More broadly, the framework provides a possible account for the computational operations of grid cells in the brain, and it suggests new machine learning architectures for representing and manipulating numerical data.

Coordinated drift of receptive fields in Hebbian/anti-Hebbian network models during noisy representation learning

Recent experiments have revealed that neural population codes in many brain areas continuously change even when animals have fully learned and stably perform their tasks. This representational 'drift' naturally leads to questions about its causes, dynamics and functions. Here we explore the hypothesis that neural representations optimize a representational objective with a degenerate solution space, and noisy synaptic updates drive the network to explore this (near-)optimal space causing representational drift. We illustrate this idea and explore its consequences in simple, biologically plausible Hebbian/anti-Hebbian network models of representation learning. We find that the drifting receptive fields of individual neurons can be characterized by a coordinated random walk, with effective diffusion constants depending on various parameters such as learning rate, noise amplitude and input statistics. Despite such drift, the representational similarity of population codes is stable over time. Our model recapitulates experimental observations in the hippocampus and posterior parietal cortex and makes testable predictions that can be probed in future experiments.