Decoding sound categories based on whole-brain functional connectivity patterns

Functional Connectivity Encodes Sound Locations by Lateralization Angles

The ability to localize sound sources rapidly allows human beings to efficiently understand the surrounding environment. Previous studies have suggested that there is an auditory "where" pathway in the cortex for processing sound locations. The neural activation in regions along this pathway encodes sound locations by opponent hemifield coding, in which each unilateral region is activated by sounds coming from the contralateral hemifield. However, it is still unclear how these regions interact with each other to form a unified representation of the auditory space. In the present study, we investigated whether functional connectivity in the auditory "where" pathway encoded sound locations during passive listening. Participants underwent functional magnetic resonance imaging while passively listening to sounds from five distinct horizontal locations (-90°, -45°, 0°, 45°, 90°). We were able to decode sound locations from the functional connectivity patterns of the "where" pathway. Furthermore, we found that such neural representation of sound locations was primarily based on the coding of sound lateralization angles to the frontal midline. In addition, whole-brain analysis indicated that functional connectivity between occipital regions and the primary auditory cortex also encoded sound locations by lateralization angles. Overall, our results reveal a lateralization-angle-based representation of sound locations encoded by functional connectivity patterns, which could add on the activation-based opponent hemifield coding to provide a more precise representation of the auditory space.

Preliminary Evidence for Global Properties in Human Listeners During Natural Auditory Scene Perception

Theories of auditory and visual scene analysis suggest the perception of scenes relies on the identification and segregation of objects within it, resembling a detail-oriented processing style. However, a more global process may occur while analyzing scenes, which has been evidenced in the visual domain. It is our understanding that a similar line of research has not been explored in the auditory domain; therefore, we evaluated the contributions of high-level global and low-level acoustic information to auditory scene perception. An additional aim was to increase the field's ecological validity by using and making available a new collection of high-quality auditory scenes. Participants rated scenes on 8 global properties (e.g., open vs. enclosed) and an acoustic analysis evaluated which low-level features predicted the ratings. We submitted the acoustic measures and average ratings of the global properties to separate exploratory factor analyses (EFAs). The EFA of the acoustic measures revealed a seven-factor structure explaining 57% of the variance in the data, while the EFA of the global property measures revealed a two-factor structure explaining 64% of the variance in the data. Regression analyses revealed each global property was predicted by at least one acoustic variable (R2 = 0.33-0.87). These findings were extended using deep neural network models where we examined correlations between human ratings of global properties and deep embeddings of two computational models: an object-based model and a scene-based model. The results support that participants' ratings are more strongly explained by a global analysis of the scene setting, though the relationship between scene perception and auditory perception is multifaceted, with differing correlation patterns evident between the two models. Taken together, our results provide evidence for the ability to perceive auditory scenes from a global perspective. Some of the acoustic measures predicted ratings of global scene perception, suggesting representations of auditory objects may be transformed through many stages of processing in the ventral auditory stream, similar to what has been proposed in the ventral visual stream. These findings and the open availability of our scene collection will make future studies on perception, attention, and memory for natural auditory scenes possible.

Toward Precise Localization of Abnormal Brain Activity: 1D CNN on Single Voxel fMRI Time-Series

Functional magnetic resonance imaging (fMRI) is one of the best techniques for precise localization of abnormal brain activity non-invasively. Machine-learning approaches have been widely used in neuroimaging studies; however, few studies have investigated the single-voxel modeling of fMRI data under cognitive tasks. We proposed a hybrid one-dimensional (1D) convolutional neural network (1D-CNN) based on the temporal dynamics of single-voxel fMRI time-series and successfully differentiated two continuous task states, namely, self-initiated (SI) and visually guided (VG) motor tasks. First, 25 activation peaks were identified from the contrast maps of SI and VG tasks in a blocked design. Then, the fMRI time-series of each peak voxel was transformed into a temporal-frequency domain by using continuous wavelet transform across a broader frequency range (0.003–0.313 Hz, with a step of 0.01 Hz). The transformed time-series was inputted into a 1D-CNN model for the binary classification of SI and VG continuous tasks. Compared with the univariate analysis, e.g., amplitude of low-frequency fluctuation (ALFF) at each frequency band, including, wavelet-ALFF, the 1D-CNN model highly outperformed wavelet-ALFF, with more efficient decoding models [46% of 800 models showing area under the curve (AUC) > 0.61] and higher decoding accuracies (94% of the efficient models), especially on the high-frequency bands (>0.1 Hz). Moreover, our results also demonstrated the advantages of wavelet decompositions over the original fMRI series by showing higher decoding performance on all peak voxels. Overall, this study suggests a great potential of single-voxel analysis using 1D-CNN and wavelet transformation of fMRI series with continuous, naturalistic, steady-state task design or resting-state design. It opens new avenues to precise localization of abnormal brain activity and fMRI-guided precision brain stimulation therapy.

Network Representations of Facial and Bodily Expressions: Evidence From Multivariate Connectivity Pattern Classification

Emotions can be perceived from both facial and bodily expressions. Our previous study has found the successful decoding of facial expressions based on the functional connectivity (FC) patterns. However, the role of the FC patterns in the recognition of bodily expressions remained unclear, and no neuroimaging studies have adequately addressed the question of whether emotions perceiving from facial and bodily expressions are processed rely upon common or different neural networks. To address this, the present study collected functional magnetic resonance imaging (fMRI) data from a block design experiment with facial and bodily expression videos as stimuli (three emotions: anger, fear, and joy), and conducted multivariate pattern classification analysis based on the estimated FC patterns. We found that in addition to the facial expressions, bodily expressions could also be successfully decoded based on the large-scale FC patterns. The emotion classification accuracies for the facial expressions were higher than that for the bodily expressions. Further contributive FC analysis showed that emotion-discriminative networks were widely distributed in both hemispheres, containing regions that ranged from primary visual areas to higher-level cognitive areas. Moreover, for a particular emotion, discriminative FCs for facial and bodily expressions were distinct. Together, our findings highlight the key role of the FC patterns in the emotion processing, indicating how large-scale FC patterns reconfigure in processing of facial and bodily expressions, and suggest the distributed neural representation for the emotion recognition. Furthermore, our results also suggest that the human brain employs separate network representations for facial and bodily expressions of the same emotions. This study provides new evidence for the network representations for emotion perception and may further our understanding of the potential mechanisms underlying body language emotion recognition.