On representation and reproducibility

Reproducibility and discriminability of brain patterns of semantic categories enhanced by congruent audiovisual stimuli

One of the central questions in cognitive neuroscience is the precise neural representation, or brain pattern, associated with a semantic category. In this study, we explored the influence of audiovisual stimuli on the brain patterns of concepts or semantic categories through a functional magnetic resonance imaging (fMRI) experiment. We used a pattern search method to extract brain patterns corresponding to two semantic categories: "old people" and "young people." These brain patterns were elicited by semantically congruent audiovisual, semantically incongruent audiovisual, unimodal visual, and unimodal auditory stimuli belonging to the two semantic categories. We calculated the reproducibility index, which measures the similarity of the patterns within the same category. We also decoded the semantic categories from these brain patterns. The decoding accuracy reflects the discriminability of the brain patterns between two categories. The results showed that both the reproducibility index of brain patterns and the decoding accuracy were significantly higher for semantically congruent audiovisual stimuli than for unimodal visual and unimodal auditory stimuli, while the semantically incongruent stimuli did not elicit brain patterns with significantly higher reproducibility index or decoding accuracy. Thus, the semantically congruent audiovisual stimuli enhanced the within-class reproducibility of brain patterns and the between-class discriminability of brain patterns, and facilitate neural representations of semantic categories or concepts. Furthermore, we analyzed the brain activity in superior temporal sulcus and middle temporal gyrus (STS/MTG). The strength of the fMRI signal and the reproducibility index were enhanced by the semantically congruent audiovisual stimuli. Our results support the use of the reproducibility index as a potential tool to supplement the fMRI signal amplitude for evaluating multimodal integration.

Neuromagnetic brain responses to words from semantic sub- and supercategories

Background

We explored spatio-temporal patterns of cortical activity evoked by written words from super-ordinate and sub-ordinate semantic categories and hoped to find a differential cortical and/or temporal distribution of the brain response depending on the level of the categories. Twenty-three subjects saw 360 words belonging to six sub-ordinate categories (mammals, birds, fish, fruit, flowers, trees) within two super-ordinate categories (fauna, flora). Visually evoked magnetic fields were determined from whole-head (148-sensor) magnetoencephalography and analyzed in the source space (Minimum Norm Estimate).

Results

Activity (MNE amplitudes) 100–150 ms after stimulus onset in the left occipito-temporal area distinguished super-ordinate categories, while later activity (300–550 ms) in the left temporal area distinguished the six sub-ordinate categories.

Conclusion

Our results document temporally and spatially distinct processing and representation of words according to their categorical information. If further studies can rule out possible confounds then our results may help constructing a theory about the internal structure of entries in the mental lexicon and its access.