Coordinated multiplexing of information about separate objects in visual cortex

Signal switching may enhance processing power of the brain

Our ability to perceive multiple objects is mysterious. Sensory neurons are broadly tuned, producing potential overlap in the populations of neurons activated by each object in a scene. This overlap raises questions about how distinct information is retained about each item. We present a novel signal switching theory of neural representation, which posits that neural signals may interleave representations of individual items across time. Evidence for this theory comes from new statistical tools that overcome the limitations inherent to standard time-and-trial-pooled assessments of neural signals. Our theory has implications for diverse domains of neuroscience, including attention, figure binding/scene segregation, oscillations, and divisive normalization. The general concept of switching between functions could also lend explanatory power to theories of grounded cognition.

Multiple objects evoke fluctuating responses in several regions of the visual pathway

How neural representations preserve information about multiple stimuli is mysterious. Because tuning of individual neurons is coarse (e.g., visual receptive field diameters can exceed perceptual resolution), the populations of neurons potentially responsive to each individual stimulus can overlap, raising the question of how information about each item might be segregated and preserved in the population. We recently reported evidence for a potential solution to this problem: when two stimuli were present, some neurons in the macaque visual cortical areas V1 and V4 exhibited fluctuating firing patterns, as if they responded to only one individual stimulus at a time (Jun et al., 2022). However, whether such an information encoding strategy is ubiquitous in the visual pathway and thus could constitute a general phenomenon remains unknown. Here, we provide new evidence that such fluctuating activity is also evoked by multiple stimuli in visual areas responsible for processing visual motion (middle temporal visual area, MT), and faces (middle fundus and anterolateral face patches in inferotemporal cortex - areas MF and AL), thus extending the scope of circumstances in which fluctuating activity is observed. Furthermore, consistent with our previous results in the early visual area V1, MT exhibits fluctuations between the representations of two stimuli when these form distinguishable objects but not when they fuse into one perceived object, suggesting that fluctuating activity patterns may underlie visual object formation. Taken together, these findings point toward an updated model of how the brain preserves sensory information about multiple stimuli for subsequent processing and behavioral action.

Compact deep neural network models of visual cortex

A powerful approach to understanding the computations carried out in visual cortex is to develop models that predict neural responses to arbitrary images. Deep neural network (DNN) models have worked remarkably well at predicting neural responses [1, 2, 3], yet their underlying computations remain buried in millions of parameters. Have we simply replaced one complicated system in vivo with another in silico ? Here, we train a data-driven deep ensemble model that predicts macaque V4 responses ∼50% more accurately than currently-used task-driven DNN models. We then compress this deep ensemble to identify compact models that have 5,000x fewer parameters yet equivalent accuracy as the deep ensemble. We verified that the stimulus preferences of the compact models matched those of the real V4 neurons by measuring V4 responses to both 'maximizing' and adversarial images generated using compact models. We then analyzed the inner workings of the compact models and discovered a common circuit motif: Compact models share a similar set of filters in early stages of processing but then specialize by heavily consolidating this shared representation with a precise readout. This suggests that a V4 neuron's stimulus preference is determined entirely by its consolidation step. To demonstrate this, we investigated the compression step of a dot-detecting compact model and found a set of simple computations that may be carried out by dot-selective V4 neurons. Overall, our work demonstrates that the DNN models currently used in computational neuroscience are needlessly large; our approach provides a new way forward for obtaining explainable, high-accuracy models of visual cortical neurons.

A recruitment through coherence theory of working memory

The interactions between prefrontal cortex and other areas during working memory have been studied for decades. Here we outline a conceptual framework describing interactions between these areas during working memory, and review evidence for key elements of this model. We specifically suggest that a top-down signal sent from prefrontal to sensory areas drives oscillations in these areas. Spike timing within sensory areas becomes locked to these working-memory-driven oscillations, and the phase of spiking conveys information about the representation available within these areas. Downstream areas receiving these phase-locked spikes from sensory areas can recover this information via a combination of coherent oscillations and gating of input efficacy based on the phase of their local oscillations. Although the conceptual framework is based on prefrontal interactions with sensory areas during working memory, we also discuss the broader implications of this framework for flexible communication between brain areas in general.

Redundant representations are required to disambiguate simultaneously presented complex stimuli

A pedestrian crossing a street during rush hour often looks and listens for potential danger. When they hear several different horns, they localize the cars that are honking and decide whether or not they need to modify their motor plan. How does the pedestrian use this auditory information to pick out the corresponding cars in visual space? The integration of distributed representations like these is called the assignment problem, and it must be solved to integrate distinct representations across but also within sensory modalities. Here, we identify and analyze a solution to the assignment problem: the representation of one or more common stimulus features in pairs of relevant brain regions-for example, estimates of the spatial position of cars are represented in both the visual and auditory systems. We characterize how the reliability of this solution depends on different features of the stimulus set (e.g., the size of the set and the complexity of the stimuli) and the details of the split representations (e.g., the precision of each stimulus representation and the amount of overlapping information). Next, we implement this solution in a biologically plausible receptive field code and show how constraints on the number of neurons and spikes used by the code force the brain to navigate a tradeoff between local and catastrophic errors. We show that, when many spikes and neurons are available, representing stimuli from a single sensory modality can be done more reliably across multiple brain regions, despite the risk of assignment errors. Finally, we show that a feedforward neural network can learn the optimal solution to the assignment problem, even when it receives inputs in two distinct representational formats. We also discuss relevant results on assignment errors from the human working memory literature and show that several key predictions of our theory already have support.

Multiple objects evoke fluctuating responses in several regions of the visual pathway

How neural representations preserve information about multiple stimuli is mysterious. Because tuning of individual neurons is coarse (for example, visual receptive field diameters can exceed perceptual resolution), the populations of neurons potentially responsive to each individual stimulus can overlap, raising the question of how information about each item might be segregated and preserved in the population. We recently reported evidence for a potential solution to this problem: when two stimuli were present, some neurons in the macaque visual cortical areas V1 and V4 exhibited fluctuating firing patterns, as if they responded to only one individual stimulus at a time. However, whether such an information encoding strategy is ubiquitous in the visual pathway and thus could constitute a general phenomenon remains unknown. Here we provide new evidence that such fluctuating activity is also evoked by multiple stimuli in visual areas responsible for processing visual motion (middle temporal visual area, MT), and faces (middle fundus and anterolateral face patches in inferotemporal cortex - areas MF and AL), thus extending the scope of circumstances in which fluctuating activity is observed. Furthermore, consistent with our previous results in the early visual area V1, MT exhibits fluctuations between the representations of two stimuli when these form distinguishable objects but not when they fuse into one perceived object, suggesting that fluctuating activity patterns may underlie visual object formation. Taken together, these findings point toward an updated model of how the brain preserves sensory information about multiple stimuli for subsequent processing and behavioral action.

Impact Statement

We find neural fluctuations in multiple areas along the visual cortical hierarchy that could allow the brain to represent distinct co-occurring visual stimuli.

Distinctive properties of biological neural networks and recent advances in bottom-up approaches toward a better biologically plausible neural network

Although it may appear infeasible and impractical, building artificial intelligence (AI) using a bottom-up approach based on the understanding of neuroscience is straightforward. The lack of a generalized governing principle for biological neural networks (BNNs) forces us to address this problem by converting piecemeal information on the diverse features of neurons, synapses, and neural circuits into AI. In this review, we described recent attempts to build a biologically plausible neural network by following neuroscientifically similar strategies of neural network optimization or by implanting the outcome of the optimization, such as the properties of single computational units and the characteristics of the network architecture. In addition, we proposed a formalism of the relationship between the set of objectives that neural networks attempt to achieve, and neural network classes categorized by how closely their architectural features resemble those of BNN. This formalism is expected to define the potential roles of top-down and bottom-up approaches for building a biologically plausible neural network and offer a map helping the navigation of the gap between neuroscience and AI engineering.

Rhythmic temporal coordination of neural activity prevents representational conflict during working memory

Selective attention¹ is characterized by alternating states associated with either attentional sampling or attentional shifting, helping to prevent functional conflicts by isolating function-specific neural activity in time.^2,3,4,5 We hypothesized that such rhythmic temporal coordination might also help to prevent representational conflicts during working memory.⁶ Multiple items can be simultaneously held in working memory, and these items can be represented by overlapping neural populations.^7,8,9 Traditional theories propose that the short-term storage of to-be-remembered items occurs through persistent neural activity,^10,11,12 but when neurons are simultaneously representing multiple items, persistent activity creates a potential for representational conflicts. In comparison, more recent, "activity-silent" theories of working memory propose that synaptic changes also contribute to short-term storage of to-be-remembered items.^13,14,15,16 Transient bursts in neural activity,¹⁷ rather than persistent activity, could serve to occasionally refresh these synaptic changes. Here, we used EEG and response times to test whether rhythmic temporal coordination helps to isolate neural activity associated with different to-be-remembered items, thereby helping to prevent representational conflicts. Consistent with this hypothesis, we report that the relative strength of different item representations alternates over time as a function of the frequency-specific phase. Although RTs were linked to theta (∼6 Hz) and beta (∼25 Hz) phases during a memory delay, the relative strength of item representations only alternated as a function of the beta phase. The present findings (1) are consistent with rhythmic temporal coordination being a general mechanism for preventing functional or representational conflicts during cognitive processes and (2) inform models describing the role of oscillatory dynamics in organizing working memory.^{13,18,19,20,21}.