Task-Dependent Warping of Semantic Representations during Search for Visual Action Categories

From Multimodal Sensorimotor Integration to Semantic Networks: A Phylogenetic Perspective on Speech and Language Evolution

This integrative perspective article delves into the crucial role of the superior temporal sulcus (STS) and adjacent perisylvian regions in multimodal integration and semantic cognition. Drawing from a wide range of neuroscientific evidence, including studies on nonhuman primates and human brain evolution, the article highlights the significance of the STS in linking auditory and visual modalities, particularly in the establishment of associative links between auditory inputs and visual stimuli. Furthermore, it explores the expansion of the human temporal lobe and its implications for the amplification of multisensory regions, emphasizing the role of these regions in the development of word-related concepts and semantic networks. We propose a posteroanterior gradient organization in the human temporal lobe, from low-level sensorimotor integration in posterior regions to higher-order, transmodal semantic control in anterior portions, particularly in the anterior temporal lobe. Overall, this perspective provides a comprehensive overview of the functional and evolutionary aspects of the STS and adjacent regions in multimodal integration and semantic cognition, offering valuable insights for future research in this field.

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.

A Plug-In Graph Neural Network to Boost Temporal Sensitivity in fMRI Analysis

Learning-based methods offer performance leaps over traditional methods in classification analysis of high-dimensional functional MRI (fMRI) data. In this domain, deep-learning models that analyze functional connectivity (FC) features among brain regions have been particularly promising. However, many existing models receive as input temporally static FC features that summarize inter-regional interactions across an entire scan, reducing the temporal sensitivity of classifiers by limiting their ability to leverage information on dynamic FC features of brain activity. To improve the performance of baseline classification models without compromising efficiency, here we propose a novel plug-in based on a graph neural network, GraphCorr, to provide enhanced input features to baseline models. The proposed plug-in computes a set of latent FC features with enhanced temporal information while maintaining comparable dimensionality to static features. Taking brain regions as nodes and blood-oxygen-level-dependent (BOLD) signals as node inputs, GraphCorr leverages a node embedder module based on a transformer encoder to capture dynamic latent representations of BOLD signals. GraphCorr also leverages a lag filter module to account for delayed interactions across nodes by learning correlational features of windowed BOLD signals across time delays. These two feature groups are then fused via a message passing algorithm executed on the formulated graph. Comprehensive demonstrations on three public datasets indicate improved classification performance for several state-of-the-art graph and convolutional baseline models when they are augmented with GraphCorr.

A modified neural circuit framework for semantic memory retrieval with implications for circuit modulation to treat verbal retrieval deficits

Word finding difficulty is a frequent complaint in older age and disease states, but treatment options are lacking for such verbal retrieval deficits. Better understanding of the neurophysiological and neuroanatomical basis of verbal retrieval function may inform effective interventions. In this article, we review the current evidence of a neural retrieval circuit central to verbal production, including words and semantic memory, that involves the pre-supplementary motor area (pre-SMA), striatum (particularly caudate nucleus), and thalamus. We aim to offer a modified neural circuit framework expanded upon a memory retrieval model proposed in 2013 by Hart et al., as evidence from electrophysiological, functional brain imaging, and noninvasive electrical brain stimulation studies have provided additional pieces of information that converge on a shared neural circuit for retrieval of memory and words. We propose that both the left inferior frontal gyrus and fronto-polar regions should be included in the expanded circuit. All these regions have their respective functional roles during verbal retrieval, such as selection and inhibition during search, initiation and termination of search, maintenance of co-activation across cortical regions, as well as final activation of the retrieved information. We will also highlight the structural connectivity from and to the pre-SMA (e.g., frontal aslant tract and fronto-striatal tract) that facilitates communication between the regions within this circuit. Finally, we will discuss how this circuit and its correlated activity may be affected by disease states and how this circuit may serve as a novel target engagement for neuromodulatory treatment of verbal retrieval deficits.

BolT: Fused window transformers for fMRI time series analysis

Deep-learning models have enabled performance leaps in analysis of high-dimensional functional MRI (fMRI) data. Yet, many previous methods are suboptimally sensitive for contextual representations across diverse time scales. Here, we present BolT, a blood-oxygen-level-dependent transformer model, for analyzing multi-variate fMRI time series. BolT leverages a cascade of transformer encoders equipped with a novel fused window attention mechanism. Encoding is performed on temporally-overlapped windows within the time series to capture local representations. To integrate information temporally, cross-window attention is computed between base tokens in each window and fringe tokens from neighboring windows. To gradually transition from local to global representations, the extent of window overlap and thereby number of fringe tokens are progressively increased across the cascade. Finally, a novel cross-window regularization is employed to align high-level classification features across the time series. Comprehensive experiments on large-scale public datasets demonstrate the superior performance of BolT against state-of-the-art methods. Furthermore, explanatory analyses to identify landmark time points and regions that contribute most significantly to model decisions corroborate prominent neuroscientific findings in the literature.

Quantitative modelling demonstrates format-invariant representations of mathematical problems in the brain

Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modelling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modelling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.

A Large Video Set of Natural Human Actions for Visual and Cognitive Neuroscience Studies and Its Validation with fMRI

The investigation of the perception of others' actions and underlying neural mechanisms has been hampered by the lack of a comprehensive stimulus set covering the human behavioral repertoire. To fill this void, we present a video set showing 100 human actions recorded in natural settings, covering the human repertoire except for emotion-driven (e.g., sexual) actions and those involving implements (e.g., tools). We validated the set using fMRI and showed that observation of the 100 actions activated the well-established action observation network. We also quantified the videos' low-level visual features (luminance, optic flow, and edges). Thus, this comprehensive video set is a valuable resource for perceptual and neuronal studies.