Gaussian mixture models and semantic gating improve reconstructions from human brain activity

The expanding horizons of network neuroscience: From description to prediction and control

The field of network neuroscience has emerged as a natural framework for the study of the brain and has been increasingly applied across divergent problems in neuroscience. From a disciplinary perspective, network neuroscience originally emerged as a formal integration of graph theory (from mathematics) and neuroscience (from biology). This early integration afforded marked utility in describing the interconnected nature of neural units, both structurally and functionally, and underscored the relevance of that interconnection for cognition and behavior. But since its inception, the field has not remained static in its methodological composition. Instead, it has grown to use increasingly advanced graph-theoretic tools and to bring in several other disciplinary perspectives-including machine learning and systems engineering-that have proven complementary. In doing so, the problem space amenable to the discipline has expanded markedly. In this review, we discuss three distinct flavors of investigation in state-of-the-art network neuroscience: (i) descriptive network neuroscience, (ii) predictive network neuroscience, and (iii) a perturbative network neuroscience that draws on recent advances in network control theory. In considering each area, we provide a brief summary of the approaches, discuss the nature of the insights obtained, and highlight future directions.

End-to-end neural system identification with neural information flow

Neural information flow (NIF) provides a novel approach for system identification in neuroscience. It models the neural computations in multiple brain regions and can be trained end-to-end via stochastic gradient descent from noninvasive data. NIF models represent neural information processing via a network of coupled tensors, each encoding the representation of the sensory input contained in a brain region. The elements of these tensors can be interpreted as cortical columns whose activity encodes the presence of a specific feature in a spatiotemporal location. Each tensor is coupled to the measured data specific to a brain region via low-rank observation models that can be decomposed into the spatial, temporal and feature receptive fields of a localized neuronal population. Both these observation models and the convolutional weights defining the information processing within regions are learned end-to-end by predicting the neural signal during sensory stimulation. We trained a NIF model on the activity of early visual areas using a large-scale fMRI dataset recorded in a single participant. We show that we can recover plausible visual representations and population receptive fields that are consistent with empirical findings.

3D Contrast Image Reconstruction From Human Brain Activity

Several studies demonstrated that functional magnetic resonance imaging (fMRI) signals in early visual cortex can be used to reconstruct 2-dimensional (2D) visual contents. However, it remains unknown how to reconstruct 3-dimensional (3D) visual stimuli from fMRI signals in visual cortex. 3D visual stimuli contain 2D visual features and depth information. Moreover, binocular disparity is an important cue for depth perception. Thus, it is more challenging to reconstruct 3D visual stimuli than 2D visual stimuli from the fMRI signals of visual cortex. This study aimed to reconstruct 3D visual images by constructing three decoding models: contrast-decoding, disparity-decoding and contrast-disparity-decoding models, and testing these models with fMRI data from humans viewing 3D contrast images. The results revealed that the 3D contrast stimuli can be reconstructed from the visual cortex. And the early visual regions (V1, V2) showed predominant advantages in reconstructing the contrast in 3D images for the contrast-decoding model. The dorsal visual regions (V3A, V7 and MT) showed predominant advantages in decoding the disparity in 3D images for the disparity-decoding model. The combination of the early and dorsal visual regions showed predominant advantages in decoding both the contrast and disparity for the contrast-disparity-decoding model. The results suggested that the contrast and disparity in 3D images were mainly represented in the early and dorsal visual regions separately. The two visual systems may interact with each other to decode 3D-contrast images.

Reconstructing Perceived Images From Human Brain Activities With Bayesian Deep Multiview Learning

Neural decoding, which aims to predict external visual stimuli information from evoked brain activities, plays an important role in understanding human visual system. Many existing methods are based on linear models, and most of them only focus on either the brain activity pattern classification or visual stimuli identification. Accurate reconstruction of the perceived images from the measured human brain activities still remains challenging. In this paper, we propose a novel deep generative multiview model for the accurate visual image reconstruction from the human brain activities measured by functional magnetic resonance imaging (fMRI). Specifically, we model the statistical relationships between the two views (i.e., the visual stimuli and the evoked fMRI) by using two view-specific generators with a shared latent space. On the one hand, we adopt a deep neural network architecture for visual image generation, which mimics the stages of human visual processing. On the other hand, we design a sparse Bayesian linear model for fMRI activity generation, which can effectively capture voxel correlations, suppress data noise, and avoid overfitting. Furthermore, we devise an efficient mean-field variational inference method to train the proposed model. The proposed method can accurately reconstruct visual images via Bayesian inference. In particular, we exploit a posterior regularization technique in the Bayesian inference to regularize the model posterior. The quantitative and qualitative evaluations conducted on multiple fMRI data sets demonstrate the proposed method can reconstruct visual images more accurately than the state of the art.

Constraint-Free Natural Image Reconstruction From fMRI Signals Based on Convolutional Neural Network

In recent years, research on decoding brain activity based on functional magnetic resonance imaging (fMRI) has made eye-catching achievements. However, constraint-free natural image reconstruction from brain activity remains a challenge, as specifying brain activity for all possible images is impractical. The problem was often simplified by using semantic prior information or just reconstructing simple images, including digitals and letters. Without semantic prior information, we present a novel method to reconstruct natural images from the fMRI signals of human visual cortex based on the computation model of convolutional neural network (CNN). First, we extracted the unit output of viewed natural images in each layer of a pre-trained CNN as CNN features. Second, we transformed image reconstruction from fMRI signals into the problem of CNN feature visualization by training a sparse linear regression to map from the fMRI patterns to CNN features. By iteratively optimization to find the matched image, whose CNN unit features become most similar to those predicted from the brain activity, we finally achieved the promising results for the challenging constraint-free natural image reconstruction. The semantic prior information of the stimuli was not used when training decoding model, and any category of images (not constraint by the training set) could be reconstructed theoretically. We found that the reconstructed images resembled the natural stimuli, especially in position and shape. The experimental results suggest that hierarchical visual features may be an effective tool to express the human visual processing.

Serial Dependence in Perceptual Decisions Is Reflected in Activity Patterns in Primary Visual Cortex

Sensory signals are highly structured in both space and time. These regularities allow expectations about future stimulation to be formed, thereby facilitating decisions about upcoming visual features and objects. One such regularity is that the world is generally stable over short time scales. This feature of the world is exploited by the brain, leading to a bias in perception called serial dependence: previously seen stimuli bias the perception of subsequent stimuli, making them appear more similar to previous input than they really are. What are the neural processes that may underlie this bias in perceptual choice? Does serial dependence arise only in higher-level areas involved in perceptual decision-making, or does such a bias occur at the earliest levels of sensory processing? In this study, human subjects made decisions about the orientation of grating stimuli presented in the left or right visual field while activity patterns in their visual cortex were recorded using fMRI. In line with previous behavioral reports, reported orientation on the current trial was consistently biased toward the previously reported orientation. We found that the orientation signal in V1 was similarly biased toward the orientation presented on the previous trial. Both the perceptual decision and neural effects were spatially specific, such that the perceptual decision and neural representations on the current trial were only influenced by previous stimuli at the same location. These results suggest that biases in perceptual decisions induced by previous stimuli may result from neural biases in sensory cortex induced by recent perceptual history.

SIGNIFICANCE STATEMENT

We perceive a stable visual scene, although our visual input is constantly changing. This experience may in part be driven by a bias in visual perception that causes images to be perceived as similar to those previously seen. Here, we provide evidence for a sensory bias that may underlie this perceptual effect. We find that neural representations in early visual cortex are biased toward previous perceptual decisions. Our results suggest a direct neural correlate of serial dependencies in visual perception. These findings elucidate how our perceptual decisions are shaped by our perceptual history.