Mode combinability: Exploring convex combinations of permutation aligned models

We explore element-wise convex combinations of two permutation-aligned neural network parameter vectors ΘA and ΘB of size d. We conduct extensive experiments by examining various distributions of such model combinations parametrized by elements of the hypercube [0,1]d and its vicinity. Our findings reveal that broad regions of the hypercube form surfaces of low loss values, indicating that the notion of linear mode connectivity extends to a more general phenomenon which we call mode combinability. We also make several novel observations regarding linear mode connectivity and model re-basin. We demonstrate a transitivity property: two models re-based to a common third model are also linear mode connected, and a robustness property: even with significant perturbations of the neuron matchings the resulting combinations continue to form a working model. Moreover, we analyze the functional and weight similarity of model combinations and show that such combinations are non-vacuous in the sense that there are significant functional differences between the resulting models.

Topographical similarity of cortical thickness represents generalized anxiety symptoms in adolescence

Generalized anxiety disorder (GAD) is a common condition characterized by excessive and uncontrollable worry, along with its high comorbidity rates. Despite increasing efforts to identify the neural underpinnings of GAD, neuroimaging research using cortical thickness have yielded largely inconsistent results. To address this, we adopted an inter-subject representational similarity analysis framework to explore a potential nonlinear relationship between vertex-wise cortical thickness and generalized anxiety symptom severity. We utilized a sample of 120 adolescents (13-18 years of age) from the Healthy Brain Network dataset. Here, we found greater topographical resemblance among participants with heightened generalized anxiety symptoms in the left caudal anterior cingulate and pericalcarine cortex. These results were not driven by the effects of age, sex, ADHD diagnosis, and GAD diagnosis. Such associations were not observed when including a group of younger participants (11-12 years of age) for analyses, highlighting the importance of age range selection when considering the link between cortical thickness and anxiety. Our findings reveal a novel cortical thickness topography that represents generalized anxiety in adolescents, which is embedded within the shared geometries between generalized anxiety symptoms and cortical thickness.

Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks

Forming transformation-tolerant object representations is critical to high-level primate vision. Despite its significance, many details of tolerance in the human brain remain unknown. Likewise, despite the ability of convolutional neural networks (CNNs) to exhibit human-like object categorization performance, whether CNNs form tolerance similar to that of the human brain is unknown. Here we provide the first comprehensive documentation and comparison of three tolerance measures in the human brain and CNNs. We measured fMRI responses from human ventral visual areas to real-world objects across both Euclidean and non-Euclidean feature changes. In single fMRI voxels in higher visual areas, we observed robust object response rank-order preservation across feature changes. This is indicative of functional smoothness in tolerance at the fMRI meso-scale level that has never been reported before. At the voxel population level, we found highly consistent object representational structure across feature changes towards the end of ventral processing. Rank-order preservation, consistency, and a third tolerance measure, cross-decoding success (i.e., a linear classifier's ability to generalize performance across feature changes) showed an overall tight coupling. These tolerance measures were in general lower for Euclidean than non-Euclidean feature changes in lower visual areas, but increased over the course of ventral processing for all feature changes. These characteristics of tolerance, however, were absent in eight CNNs pretrained with ImageNet images with varying network architecture, depth, the presence/absence of recurrent processing, or whether a network was pretrained with the original or stylized ImageNet images that encouraged shape processing. CNNs do not appear to develop the same kind of tolerance as the human brain over the course of visual processing.

Acute stress alters neural patterns of value representation for others

Acute stress is often evoked during social interactions, by feelings of threat or negative evaluation by other people. We also constantly interact with others while under stress - in the workplace or in private alike. However, it is not clear how stress affects social interactions. For one, individuals could become more selfish and focused on their own goals. On the other hand, individuals might also become more focused on affiliating with potential social partners, in order to secure their support. There is, indeed, accumulating behavioral evidence that prosocial behaviors increase rather than decrease under stress. Here, we tested the underlying brain processes of such findings, by assessing the effects of stress on the neural representations of (monetary) value for self and other. Participants (N ​= ​30; male, 18-40 years) played a gambling task for themselves and for another participant while undergoing functional magnetic resonance imaging (fMRI). Each participant played the gambling task twice: once immediately following acute stress induction, and once in a control session. We compared neural patterns of value representation in the dorsomedial prefrontal cortex (dmPFC), ventromedial prefrontal cortex (vmPFC) and striatum using representational similarity analysis (RSA). We found that under stress, dmPFC and striatum showed higher dissimilarity between neural patterns underlying high and low value for the other. Dissimilarity of neural patterns underlying high and low value for the self was unaffected by stress. These findings suggest that participants track the magnitude of possible rewards for others more under stress, suggesting increased prosocial orientation.

How are response properties in the middle temporal area related to inference on visual motion patterns?

Neurons in the primate middle temporal area (MT) respond to moving stimuli, with strong tuning for motion speed and direction. These responses have been characterized in detail, but the functional significance of these details (e.g. shapes and widths of speed tuning curves) is unclear, because they cannot be selectively manipulated. To estimate their functional significance, we used a detailed model of MT population responses as input to convolutional networks that performed sophisticated motion processing tasks (visual odometry and gesture recognition). We manipulated the distributions of speed and direction tuning widths, and studied the effects on task performance. We also studied performance with random linear mixtures of the responses, and with responses that had the same representational dissimilarity as the model populations, but were otherwise randomized. The width of speed and direction tuning both affected task performance, despite the networks having been optimized individually for each tuning variation, but the specific effects were different in each task. Random linear mixing improved performance of the odometry task, but not the gesture recognition task. Randomizing the responses while maintaining representational dissimilarity resulted in poor odometry performance. In summary, despite full optimization of the deep networks in each case, each manipulation of the representation affected performance of sophisticated visual tasks. Representation properties such as tuning width and representational similarity have been studied extensively from other perspectives, but this work provides new insight into their possible roles in sophisticated visual inference.

Neural representation of object orientation: A dissociation between MVPA and Repetition Suppression

How is object orientation represented in the brain? Behavioral error patterns reveal systematic tendencies to confuse certain orientations with one another. Using fMRI, we asked whether more confusable orientations are represented more similarly in object selective cortex (LOC). We compared two widely-used measures of neural similarity: multi-voxel pattern similarity (MVP-similarity) and Repetition Suppression. In LO, we found that multi-voxel pattern similarity was predicted by the confusability of two orientations. By contrast, Repetition Suppression effects in LO were unrelated to the confusability of orientations. To account for these differences between MVP-similarity and Repetition Suppression, we propose that MVP-similarity reflects the topographical distribution of neural populations, whereas Repetition Suppression depends on repeated activation of particular groups of neurons. This hypothesis leads to a unified interpretation of our results and may explain other dissociations between MVPA and Repetition Suppression observed in the literature.

The 'when' and 'where' of semantic coding in the anterior temporal lobe: Temporal representational similarity analysis of electrocorticogram data

Electrocorticograms (ECoG) provide a unique opportunity to monitor neural activity directly at the cortical surface. Ten patients with subdural electrodes covering ventral and lateral anterior temporal regions (ATL) performed a picture naming task. Temporal representational similarity analysis (RSA) was used, for the first time, to compare spatio-temporal neural patterns from the ATL surface with pre-defined theoretical models. The results indicate that the neural activity in the ventral subregion of the ATL codes semantic representations from 250 msec after picture onset. The observed activation similarity was not related to the visual similarity of the pictures or the phonological similarity of their names. In keeping with convergent evidence for the importance of the ATL in semantic processing, these results provide the first direct evidence of semantic coding from the surface of the ventral ATL and its time-course.

Reading visually embodied meaning from the brain: Visually grounded computational models decode visual-object mental imagery induced by written text

Embodiment theory predicts that mental imagery of object words recruits neural circuits involved in object perception. The degree of visual imagery present in routine thought and how it is encoded in the brain is largely unknown. We test whether fMRI activity patterns elicited by participants reading objects' names include embodied visual-object representations, and whether we can decode the representations using novel computational image-based semantic models. We first apply the image models in conjunction with text-based semantic models to test predictions of visual-specificity of semantic representations in different brain regions. Representational similarity analysis confirms that fMRI structure within ventral-temporal and lateral-occipital regions correlates most strongly with the image models and conversely text models correlate better with posterior-parietal/lateral-temporal/inferior-frontal regions. We use an unsupervised decoding algorithm that exploits commonalities in representational similarity structure found within both image model and brain data sets to classify embodied visual representations with high accuracy (8/10) and then extend it to exploit model combinations to robustly decode different brain regions in parallel. By capturing latent visual-semantic structure our models provide a route into analyzing neural representations derived from past perceptual experience rather than stimulus-driven brain activity. Our results also verify the benefit of combining multimodal data to model human-like semantic representations.

Visual representations are dominated by intrinsic fluctuations correlated between areas

Intrinsic cortical dynamics are thought to underlie trial-to-trial variability of visually evoked responses in animal models. Understanding their function in the context of sensory processing and representation is a major current challenge. Here we report that intrinsic cortical dynamics strongly affect the representational geometry of a brain region, as reflected in response-pattern dissimilarities, and exaggerate the similarity of representations between brain regions. We characterized the representations in several human visual areas by representational dissimilarity matrices (RDMs) constructed from fMRI response-patterns for natural image stimuli. The RDMs of different visual areas were highly similar when the response-patterns were estimated on the basis of the same trials (sharing intrinsic cortical dynamics), and quite distinct when patterns were estimated on the basis of separate trials (sharing only the stimulus-driven component). We show that the greater similarity of the representational geometries can be explained by coherent fluctuations of regional-mean activation within visual cortex, reflecting intrinsic dynamics. Using separate trials to study stimulus-driven representations revealed clearer distinctions between the representational geometries: a Gabor wavelet pyramid model explained representational geometry in visual areas V1–3 and a categorical animate–inanimate model in the object-responsive lateral occipital cortex.