Multiclass fMRI data decoding and visualization using supervised self-organizing maps

Modality-specific brain representations during automatic processing of face, voice and body expressions

A central question in affective science and one that is relevant for its clinical applications is how emotions provided by different stimuli are experienced and represented in the brain. Following the traditional view emotional signals are recognized with the help of emotion concepts that are typically used in descriptions of mental states and emotional experiences, irrespective of the sensory modality. This perspective motivated the search for abstract representations of emotions in the brain, shared across variations in stimulus type (face, body, voice) and sensory origin (visual, auditory). On the other hand, emotion signals like for example an aggressive gesture, trigger rapid automatic behavioral responses and this may take place before or independently of full abstract representation of the emotion. This pleads in favor specific emotion signals that may trigger rapid adaptative behavior only by mobilizing modality and stimulus specific brain representations without relying on higher order abstract emotion categories. To test this hypothesis, we presented participants with naturalistic dynamic emotion expressions of the face, the whole body, or the voice in a functional magnetic resonance (fMRI) study. To focus on automatic emotion processing and sidestep explicit concept-based emotion recognition, participants performed an unrelated target detection task presented in a different sensory modality than the stimulus. By using multivariate analyses to assess neural activity patterns in response to the different stimulus types, we reveal a stimulus category and modality specific brain organization of affective signals. Our findings are consistent with the notion that under ecological conditions emotion expressions of the face, body and voice may have different functional roles in triggering rapid adaptive behavior, even if when viewed from an abstract conceptual vantage point, they may all exemplify the same emotion. This has implications for a neuroethologically grounded emotion research program that should start from detailed behavioral observations of how face, body, and voice expressions function in naturalistic contexts.

Searchlight Classification Informative Region Mixture Model (SCIM): Identification of Cortical Regions Showing Discriminable BOLD Patterns in Event-Related Auditory fMRI Data

The investigation of abstract cognitive tasks, e.g., semantic processing of speech, requires the simultaneous use of a carefully selected stimulus design and sensitive tools for the analysis of corresponding neural activity that are comparable across different studies investigating similar research questions. Multi-voxel pattern analysis (MVPA) methods are commonly used in neuroimaging to investigate BOLD responses corresponding to neural activation associated with specific cognitive tasks. Regions of significant activation are identified by a thresholding operation during multivariate pattern analysis, the results of which are susceptible to the applied threshold value. Investigation of analysis approaches that are robust to a large extent with respect to thresholding, is thus an important goal pursued here. The present paper contributes a novel statistical analysis method for fMRI experiments, searchlight classification informative region mixture model (SCIM), that is based on the assumption that the whole brain volume can be subdivided into two groups of voxels: spatial voxel positions around which recorded BOLD activity does convey information about the present stimulus condition and those that do not. A generative statistical model is proposed that assigns a probability of being informative to each position in the brain, based on a combination of a support vector machine searchlight analysis and Gaussian mixture models. Results from an auditory fMRI study investigating cortical regions that are engaged in the semantic processing of speech indicate that the SCIM method identifies physiologically plausible brain regions as informative, similar to those from two standard methods as reference that we compare to, with two important differences. SCIM-identified regions are very robust to the choice of the threshold for significance, i.e., less “noisy,” in contrast to, e.g., the binomial test whose results in the present experiment are highly dependent on the chosen significance threshold or random permutation tests that are additionally bound to very high computational costs. In group analyses, the SCIM method identifies a physiologically plausible pre-frontal region, anterior cingulate sulcus, to be involved in semantic processing that other methods succeed to identify only in single subject analyses.

Spatial Information of Somatosensory Stimuli in the Brain: Multivariate Pattern Analysis of Functional Magnetic Resonance Imaging Data

Background

Multivoxel pattern analysis has provided new evidence on somatotopic representation in the human brain. However, the effects of stimulus modality (e.g., penetrating needle versus non-penetrating touch) and level of classification (e.g., multiclass versus binary classification) on patterns of brain activity encoding spatial information of body parts have not yet been studied. We hypothesized that performance of brain-based prediction models may vary across the types of stimuli, and neural patterns of voxels in the SI and parietal cortex would significantly contribute to the prediction of stimulated locations.

Objective

We aimed to (1) test whether brain responses to tactile stimuli could distinguish among stimulated locations on the body surface, (2) investigate whether the stimulus modality and number of classes affect classification performance, and (3) localize brain regions encoding the spatial information of somatosensory stimuli.

Methods

Fifteen healthy participants completed two functional magnetic resonance imaging (MRI) scans and were stimulated via the insertion of acupuncture needles or by non-invasive touch stimuli (5.46-sized von Frey filament). Participants received the stimuli at four different locations on the upper and lower limbs (two sites each) for 5 min while blood-oxygen-level-dependent activity (BOLD) was measured using 3-Tesla MRI. We performed multivariate pattern analysis (MVPA) using parameter estimate images of each trial for each participant and the support vector classifier (SVC) function, and the prediction accuracy and other MVPA outcomes were evaluated using stratified five-fold cross validation. We estimated the significance of the classification accuracy using a permutation test with randomly labeled training data (n = 10,000). Searchlight analysis was conducted to identify brain regions associated with significantly higher accuracy compared to predictions based on chance as obtained from a random classifier.

Results

For the four-class classification (classifying four stimulated points on the body), SVC analysis of whole-brain beta values in response to acupuncture stimulation was able to discriminate among stimulated locations (mean accuracy, 0.31; q < 0.01). The searchlight analysis found that values related to the right primary somatosensory cortex (SI) and intraparietal sulcus were significantly more accurate than those due to chance (p < 0.01). On the other hand, the same classifier did not predict stimulated locations accurately for touch stimulation (mean accuracy, 0.25; q = 0.66). For binary classification (discriminating between two stimulated body parts, i.e., the arm or leg), the SVC algorithm successfully predicted the stimulated body parts for both acupuncture (mean accuracy, 0.63; q < 0.001) and touch stimulation (mean accuracy, 0.60; q < 0.01). Searchlight analysis revealed that predictions based on the right SI, primary motor cortex (MI), paracentral gyrus, and superior frontal gyrus were significantly more accurate compared to predictions based on chance (p < 0.05).

Conclusion

Our findings suggest that the SI, as well as the MI, intraparietal sulcus, paracentral gyrus, and superior frontal gyrus, is responsible for the somatotopic representation of body parts stimulated by tactile stimuli. The MVPA approach for identifying neural patterns encoding spatial information of somatosensory stimuli may be affected by the stimulus type (penetrating needle versus non-invasive touch) and the number of classes (classification of four small points on the body versus two large body parts). Future studies with larger samples will identify stimulus-specific neural patterns representing stimulated locations, independent of subjective tactile perception and emotional responses. Identification of distinct neural patterns of body surfaces will help in improving neural biomarkers for pain and other sensory percepts in the future.

Hyperalignment of motor cortical areas based on motor imagery during action observation

Multivariate Pattern Analysis (MVPA) has grown in importance due to its capacity to use both coarse and fine scale patterns of brain activity. However, a major limitation of multivariate analysis is the difficulty of aligning features across brains, which makes MVPA a subject specific analysis. Recent work by Haxby et al. (2011) introduced a method called Hyperalignment that explored neural activity in ventral temporal cortex during object recognition and demonstrated the ability to align individual patterns of brain activity into a common high dimensional space to facilitate Between Subject Classification (BSC). Here we examined BSC based on Hyperalignment of motor cortex during a task of motor imagery of three natural actions (lift, knock and throw). To achieve this we collected brain activity during the combined tasks of action observation and motor imagery to a parametric action space containing 25 stick-figure blends of the three natural actions. From these responses we derived Hyperalignment transformation parameters that were used to map subjects’ representational spaces of the motor imagery task in the motor cortex into a common model representational space. Results showed that BSC of the neural response patterns based on Hyperalignment exceeded both BSC based on anatomical alignment as well as a standard Within Subject Classification (WSC) approach. We also found that results were sensitive to the order in which participants entered the Hyperalignment algorithm. These results demonstrate the effectiveness of Hyperalignment to align neural responses across subject in motor cortex to enable BSC of motor imagery.

Combining self-organizing mapping and supervised affinity propagation clustering approach to investigate functional brain networks involved in motor imagery and execution with fMRI measurements

Clustering analysis methods have been widely applied to identifying the functional brain networks of a multitask paradigm. However, the previously used clustering analysis techniques are computationally expensive and thus impractical for clinical applications. In this study a novel method, called SOM-SAPC that combines self-organizing mapping (SOM) and supervised affinity propagation clustering (SAPC), is proposed and implemented to identify the motor execution (ME) and motor imagery (MI) networks. In SOM-SAPC, SOM was first performed to process fMRI data and SAPC is further utilized for clustering the patterns of functional networks. As a result, SOM-SAPC is able to significantly reduce the computational cost for brain network analysis. Simulation and clinical tests involving ME and MI were conducted based on SOM-SAPC, and the analysis results indicated that functional brain networks were clearly identified with different response patterns and reduced computational cost. In particular, three activation clusters were clearly revealed, which include parts of the visual, ME and MI functional networks. These findings validated that SOM-SAPC is an effective and robust method to analyze the fMRI data with multitasks.