Naturalistic Stimuli: A Paradigm for Multi-Scale Functional Characterization of the Human Brain

A large-scale MEG and EEG dataset for object recognition in naturalistic scenes

Neuroimaging with large-scale naturalistic stimuli is increasingly employed to elucidate neural mechanisms of object recognition in natural scenes. However, most existing large-scale neuroimaging datasets with naturalistic stimuli primarily rely on functional magnetic resonance imaging (fMRI), which provides high spatial resolution but is limited in capturing the temporal dynamics. To address this limitation, we extended our Natural Object Dataset-fMRI (NOD-fMRI) by collecting both magnetoencephalography (MEG) and electroencephalography (EEG) data from the same participants while viewing the same naturalistic stimuli. As a result, NOD contains fMRI, MEG, and EEG responses to 57,000 naturalistic images from 30 participants. This enables the examination of brain activity elicited by naturalistic stimuli with both high spatial resolution (via fMRI) and high temporal resolution (via MEG and EEG). Furthermore, the multimodal nature of NOD allows researchers to combine datasets from different modalities to achieve a more comprehensive view of object processing. We believe that the NOD dataset will serve as a valuable resource for advancing our understanding of the cognitive and neural mechanisms underlying object recognition.

Differential fMRI neural synchrony associated with migraine during naturalistic stimuli with negative emotional valence

Migraine is a common neurological disorder that impacts approximately 12% of the general population and is characterized by moderate to severe headaches, nausea, mood changes, and fatigue. It impacts lower-level visual and auditory processing, causing hypersensitivities that lead to heightened audiovisual multisensory integration. However, the impact of migraine on the processing of complex, audiovisual stimuli is still unclear. Additionally, migraine may induce hypersensitivities to emotional arousal and valence, though the relative significance of these factors remains unknown. The current study seeks to identify how migraine impacts synchronous neural processing of complex, audiovisual stimuli, and how this differs based on the emotional arousal and valence of the stimulus. To do so, we collected functional magnetic resonance imaging data (fMRI) from 22 migraineurs and 21 healthy controls during the passive viewing of three audiovisual films of differing emotional arousal and valence. We identified that, in response to a negative valence, high arousal emotional stimulus, the migraine group showed greater neural synchrony in regions associated with multisensory integration, including the bilateral posterior superior temporal gyrus (pSTG), superior parietal lobule (SPL), and left middle temporal gyrus (MTG). There were no significant differences in neural synchrony between the migraine and control groups in response to positive valence, high arousal and neutral valence, low arousal stimuli. These findings suggest that migraine involves hypersensitivity to audiovisual movies as a function of negative emotional valence, where negative/aversive emotional states may drive greater synchrony in multisensory integration. Overall, this research highlights distinct pathways through which emotion and arousal impact neural processing in migraine.

Dynamic Neural Mechanisms Underlying Sustained Motor Training Post-Stroke: An fNIRS Investigation Employing Time-Varying ALFF and Functional Connectivity Analysis

Motor dysfunction of the upper limbs following a stroke predominantly arises from abnormal motor patterning caused by the disrupted balance of inter-cortical communication within motor-associated cortical regions. Temporal analysis offers a more precise reflection of the cortical functional state in affected patients. This study employed fNIRS to capture hemodynamic responses among 20 stroke patients and 19 healthy controls in both resting and Baduanjin task state. Including computing the coefficient of variation of fractional amplitude of low-frequency fluctuations (fALFF) for each channel, alongside extracting dynamic state metrics. Findings indicate that, during sustained motor tasks, stroke patients exhibit a diminished fALFF variability in targeted cortical regions; these individuals display a higher prevalence of low-intensity states and a lower prevalence of high-intensity states during task execution. These dynamic state attributes are significantly associated with scores on the upper limb motor function scale (FMA-UE), thereby proposing a time-domain perspective for investigating the underlying mechanisms of stroke-induced motor dysfunction.

Neural processing of naturalistic audiovisual events in space and time

Our brain seamlessly integrates distinct sensory information to form a coherent percept. However, when real-world audiovisual events are perceived, the specific brain regions and timings for processing different levels of information remain less investigated. To address that, we curated naturalistic videos and recorded functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) data when participants viewed videos with accompanying sounds. Our findings reveal early asymmetrical cross-modal interaction, with acoustic information represented in both early visual and auditory regions, while visual information only identified in visual cortices. The visual and auditory features were processed with similar onset but different temporal dynamics. High-level categorical and semantic information emerged in multisensory association areas later in time, indicating late cross-modal integration and its distinct role in converging conceptual information. Comparing neural representations to a two-branch deep neural network model highlighted the necessity of early cross-modal connections to build a biologically plausible model of audiovisual perception. With EEG-fMRI fusion, we provided a spatiotemporally resolved account of neural activity during the processing of naturalistic audiovisual stimuli.

Psychological well-being modulates neural synchrony during naturalistic fMRI

Psychological well-being (PWB) is a combination of feeling good and functioning efficiently, and has a significant relationship with physical and mental health. Previous research has shown that PWB is associated with improvements in selective attention, mindfulness, semantic self-images, and adaptive decision making, however, it is unclear how these differences manifest in the brain. Naturalistic stimuli better encapsulate everyday experiences and can elicit more "true-to-life" neural responses. The current study seeks to identify how differing levels of PWB modulate neural synchrony in response to an audiovisual film. With consideration of the inherent variability of the literature, we aim to ascertain the validity of the regions previously associated with PWB. We identified that higher levels of PWB were associated with heightened stimulus driven neural synchrony in the bilateral superior parietal lobule, right planum temporale, and left superior temporal gyrus, and that lower levels of PWB were associated with heightened neural synchrony in the bilateral lateral occipital cortex and precuneus. Taken together, this research suggests that there is an association between differing levels of PWB and differential neural synchrony during movie-watching. PWB may therefore have an effect on complex, multimodal processing.

Cortical oscillations are modified by expertise in dance and music: Evidence from live dance audience

Over the past decades, the focus of brain research has expanded from using strictly controlled stimuli towards understanding brain functioning in complex naturalistic contexts. Interest has increased in measuring brain processes in natural interaction, including classrooms, theatres, concerts and museums to understand the brain functions in the real world. Here, we examined how watching a live dance performance with music in a real-world dance performance setting engages the brains of the spectators. Expertise in dance or music has been shown to modify brain functions, including when watching dance or listening to music. Therefore, we recorded electroencephalography (EEG) from an audience of dancers, musicians and novices as they watched the live dance performance and analysed their cortical oscillations. We compared intrabrain oscillations when participants watched the performance (with music) or listened to the music alone without the dance. We found that dancers have stronger fronto-central and parieto-occipital theta phase synchrony (4-8 Hz) than novices when watching dance, likely reflecting the effects of dance experience on motor imagery, multisensory and social interaction processes. Also, compared with novices, dancers had stronger delta phase synchrony (0.5-4 Hz) when listening to music, and musicians had stronger delta phase synchrony when watching dance, suggesting expertise in music and dance enhances sensitivity or attention to temporal regularities in movement and sound.

A comparison of EEG encoding models using audiovisual stimuli and their unimodal counterparts

Communication in the real world is inherently multimodal. When having a conversation, typically sighted and hearing people use both auditory and visual cues to understand one another. For example, objects may make sounds as they move in space, or we may use the movement of a person's mouth to better understand what they are saying in a noisy environment. Still, many neuroscience experiments rely on unimodal stimuli to understand encoding of sensory features in the brain. The extent to which visual information may influence encoding of auditory information and vice versa in natural environments is thus unclear. Here, we addressed this question by recording scalp electroencephalography (EEG) in 11 subjects as they listened to and watched movie trailers in audiovisual (AV), visual (V) only, and audio (A) only conditions. We then fit linear encoding models that described the relationship between the brain responses and the acoustic, phonetic, and visual information in the stimuli. We also compared whether auditory and visual feature tuning was the same when stimuli were presented in the original AV format versus when visual or auditory information was removed. In these stimuli, visual and auditory information was relatively uncorrelated, and included spoken narration over a scene as well as animated or live-action characters talking with and without their face visible. For this stimulus, we found that auditory feature tuning was similar in the AV and A-only conditions, and similarly, tuning for visual information was similar when stimuli were presented with the audio present (AV) and when the audio was removed (V only). In a cross prediction analysis, we investigated whether models trained on AV data predicted responses to A or V only test data similarly to models trained on unimodal data. Overall, prediction performance using AV training and V test sets was similar to using V training and V test sets, suggesting that the auditory information has a relatively smaller effect on EEG. In contrast, prediction performance using AV training and A only test set was slightly worse than using matching A only training and A only test sets. This suggests the visual information has a stronger influence on EEG, though this makes no qualitative difference in the derived feature tuning. In effect, our results show that researchers may benefit from the richness of multimodal datasets, which can then be used to answer more than one research question.

Frequency-specific functional difference between gyri and sulci in naturalistic paradigm fMRI

Disentangling functional difference between cortical folding patterns of gyri and sulci provides novel insights into the relationship between brain structure and function. Previous studies using resting-state functional magnetic resonance imaging (rsfMRI) have revealed that sulcal signals exhibit stronger high-frequency but weaker low-frequency components compared to gyral ones, suggesting that gyri may serve as functional integration centers while sulci are segregated local processing units. In this study, we utilize naturalistic paradigm fMRI (nfMRI) to explore the functional difference between gyri and sulci as it has proven to record stronger functional integrations compared to rsfMRI. We adopt a convolutional neural network (CNN) to classify gyral and sulcal fMRI signals in the whole brain (the global model) and within functional brain networks (the local models). The frequency-specific difference between gyri and sulci is then inferred from the power spectral density (PSD) profiles of the learned filters in the CNN model. Our experimental results show that nfMRI shows higher gyral-sulcal PSD contrast effect sizes in the global model compared to rsfMRI. In the local models, the effect sizes are either increased or decreased depending on frequency bands and functional complexity of the FBNs. This study highlights the advantages of nfMRI in depicting the functional difference between gyri and sulci, and provides novel insights into unraveling the relationship between brain structure and function.

Math on cortex-enhanced delta phase synchrony in math experts during long and complex math demonstrations

Neural oscillations are important for working memory and reasoning and they are modulated during cognitively challenging tasks, like mathematics. Previous work has examined local cortical synchrony on theta (4-8 Hz) and alpha (8-13 Hz) bands over frontal and parietal electrodes during short mathematical tasks when sitting. However, it is unknown whether processing of long and complex math stimuli evokes inter-regional functional connectivity. We recorded cortical activity with EEG while math experts and novices watched long (13-68 seconds) and complex (bachelor-level) math demonstrations when sitting and standing. Fronto-parietal connectivity over the left hemisphere was stronger in math experts than novices reflected by enhanced delta (0.5-4 Hz) phase synchrony in experts. Processing of complex math tasks when standing extended the difference to right hemisphere, suggesting that other cognitive processes, such as maintenance of body balance when standing, may interfere with novice's internal concentration required during complex math tasks more than in experts. There were no groups differences in phase synchrony over theta or alpha frequencies. These results suggest that low-frequency oscillations modulate inter-regional connectivity during long and complex mathematical cognition and demonstrate one way in which the brain functions of math experts differ from those of novices: through enhanced fronto-parietal functional connectivity.

Impact of hypomanic personality traits on brain functional connectivity during a dynamic theory-of-mind task

Hypomanic personality traits are present in the general population and represent a risk factor for developing bipolar disorder. This personality style, notably its social component, is linked to difficulties in theory of mind (i.e., ability to infer mental states). Exploring the neural correlates of mental states' inference in individuals with these personality traits can provide meaningful insights into the development of bipolar disorder. The aim of the present study was therefore to investigate the potential impact of hypomanic traits on brain activation and task-based connectivity strength during a dynamic theory of mind task in a nonclinical population. A total of 52 nonclinical participants were recruited, and hypomanic traits were assessed with the Hypomanic Personality Scale. The severity of hypomanic traits was positively associated with right middle and inferior frontal gyri activations (in high vs. low inference in nonemotional condition and emotion vs. no emotion in high inference, respectively). It was also associated with stronger connectivity between the salience network (i.e., bilateral putamen and pallidum) and bilateral superior temporal gyri (high inference in nonemotional condition), and between cerebellar and temporal areas (high inference in emotional condition). These changes may either reflect adaptations or differential processing, and further studies are therefore mandatory.

The abilities of movie-watching functional connectivity in individual identifications and individualized predictions

Quite a few studies have been performed based on movie-watching functional connectivity (FC). As compared to its resting-state counterpart, however, there is still much to know about its abilities in individual identifications and individualized predictions. To pave the way for appropriate usage of movie-watching FC, we systemically evaluated the minimum number of time points, as well as the exact functional networks, supporting individual identifications and individualized predictions of apparent traits based on it. We performed the study based on the 7T movie-watching fMRI data included in the HCP S1200 Release, and took IQ as the test case for the prediction analyses. The results indicate that movie-watching FC based on only 15 time points can support successful individual identifications (99.47%), and the connectivity contributed more to identifications were much associated with higher-order cognitive processes (the secondary visual network, the frontoparietal network and the posterior multimodal network). For individualized predictions of IQ, it was found that successful predictions necessitated 60 time points (predicted vs. actual IQ correlation significant at P < 0.05, based on 5,000 permutations), and the prediction accuracy increased logarithmically with the number of time points used for connectivity calculation. Furthermore, the connectivity that contributed more to individual identifications exhibited the strongest prediction ability. Collectively, our findings demonstrate that movie-watching FC can capture rich information about human brain function, and its ability in individualized predictions depends heavily on the length of fMRI scans.

A large-scale fMRI dataset for the visual processing of naturalistic scenes

One ultimate goal of visual neuroscience is to understand how the brain processes visual stimuli encountered in the natural environment. Achieving this goal requires records of brain responses under massive amounts of naturalistic stimuli. Although the scientific community has put a lot of effort into collecting large-scale functional magnetic resonance imaging (fMRI) data under naturalistic stimuli, more naturalistic fMRI datasets are still urgently needed. We present here the Natural Object Dataset (NOD), a large-scale fMRI dataset containing responses to 57,120 naturalistic images from 30 participants. NOD strives for a balance between sampling variation between individuals and sampling variation between stimuli. This enables NOD to be utilized not only for determining whether an observation is generalizable across many individuals, but also for testing whether a response pattern is generalized to a variety of naturalistic stimuli. We anticipate that the NOD together with existing naturalistic neuroimaging datasets will serve as a new impetus for our understanding of the visual processing of naturalistic stimuli.

Naturalistic auditory stimuli with fNIRS prefrontal cortex imaging: A potential paradigm for disorder of consciousness diagnostics (a study with healthy participants)

Disorder of consciousness (DOC) is a devastating condition due to brain damage. A patient in this condition is non-responsive, but nevertheless might be conscious at least at some level. Determining the conscious level of DOC patients is important for both medical and ethical reasons, but reliably achieving this has been a major challenge. Naturalistic stimuli in combination with neuroimaging have been proposed as a promising approach for DOC patient diagnosis. Capitalizing on and extending this proposal, the goal of the present study conducted with healthy participants was to develop a new paradigm with naturalistic auditory stimuli and functional near-infrared spectroscopy (fNIRS) - an approach that can be used at the bedside. Twenty-four healthy participants passively listened to 9 min of auditory story, scrambled auditory story, classical music, and scrambled classical music segments while their prefrontal cortex activity was recorded using fNIRS. We found much higher intersubject correlation (ISC) during story compared to scrambled story conditions both at the group level and in the majority of individual subjects, suggesting that fNIRS imaging of the prefrontal cortex might be a sensitive method to capture neural changes associated with narrative comprehension. In contrast, the ISC during the classical music segment did not differ reliably from scrambled classical music and was also much lower than the story condition. Our main result is that naturalistic auditory stories with fNIRS might be used in a clinical setup to identify high-level processing and potential consciousness in DOC patients.

Nonlinear and machine learning analyses on high-density EEG data of math experts and novices

Current trend in neurosciences is to use naturalistic stimuli, such as cinema, class-room biology or video gaming, aiming to understand the brain functions during ecologically valid conditions. Naturalistic stimuli recruit complex and overlapping cognitive, emotional and sensory brain processes. Brain oscillations form underlying mechanisms for such processes, and further, these processes can be modified by expertise. Human cortical functions are often analyzed with linear methods despite brain as a biological system is highly nonlinear. This study applies a relatively robust nonlinear method, Higuchi fractal dimension (HFD), to classify cortical functions of math experts and novices when they solve long and complex math demonstrations in an EEG laboratory. Brain imaging data, which is collected over a long time span during naturalistic stimuli, enables the application of data-driven analyses. Therefore, we also explore the neural signature of math expertise with machine learning algorithms. There is a need for novel methodologies in analyzing naturalistic data because formulation of theories of the brain functions in the real world based on reductionist and simplified study designs is both challenging and questionable. Data-driven intelligent approaches may be helpful in developing and testing new theories on complex brain functions. Our results clarify the different neural signature, analyzed by HFD, of math experts and novices during complex math and suggest machine learning as a promising data-driven approach to understand the brain processes in expertise and mathematical cognition.

Sentiments analysis of fMRI using automatically generated stimuli labels under naturalistic paradigm

Our emotions and sentiments are influenced by naturalistic stimuli such as the movies we watch and the songs we listen to, accompanied by changes in our brain activation. Comprehension of these brain-activation dynamics can assist in identification of any associated neurological condition such as stress and depression, leading towards making informed decision about suitable stimuli. A large number of open-access functional magnetic resonance imaging (fMRI) datasets collected under naturalistic conditions can be used for classification/prediction studies. However, these datasets do not provide emotion/sentiment labels, which limits their use in supervised learning studies. Manual labeling by subjects can generate these labels, however, this method is subjective and biased. In this study, we are proposing another approach of generating automatic labels from the naturalistic stimulus itself. We are using sentiment analyzers (VADER, TextBlob, and Flair) from natural language processing to generate labels using movie subtitles. Subtitles generated labels are used as the class labels for positive, negative, and neutral sentiments for classification of brain fMRI images. Support vector machine, random forest, decision tree, and deep neural network classifiers are used. We are getting reasonably good classification accuracy (42-84%) for imbalanced data, which is increased (55-99%) for balanced data.

Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience

Determining and decoding emotional brain processes under ecologically valid conditions remains a key challenge in affective neuroscience. The current functional Magnetic Resonance Imaging (fMRI) based emotion decoding studies are mainly based on brief and isolated episodes of emotion induction, while sustained emotional experience in naturalistic environments that mirror daily life experiences are scarce. Here we used 12 different 10-minute movie clips as ecologically valid emotion-evoking procedures in n = 52 individuals to explore emotion-specific fMRI functional connectivity (FC) profiles on the whole-brain level at high spatial resolution (432 parcellations including cortical and subcortical structures). Employing machine-learning based decoding and cross validation procedures allowed to investigate FC profiles contributing to classification that can accurately distinguish sustained happiness and sadness and that generalize across subjects, movie clips, and parcellations. Both functional brain network-based and subnetwork-based emotion classification results suggested that emotion manifests as distributed representation of multiple networks, rather than a single functional network or subnetwork. Further, the results showed that the Visual Network (VN) and Default Mode Network (DMN) associated functional networks, especially VN-DMN, exhibited a strong contribution to emotion classification. To further estimate the temporal accumulative effect of naturalistic long-term movie-based video-evoking emotions, we divided the 10-min episode into three stages: early stimulation (1∼200 s), middle stimulation (201∼400 s), and late stimulation (401∼600 s) and examined the emotion classification performance at different stimulation stages. We found that the late stimulation contributes most to the classification (accuracy=85.32%, F1-score=85.62%) compared to early and middle stimulation stages, implying that continuous exposure to emotional stimulation can lead to more intense emotions and further enhance emotion-specific distinguishable representations. The present work demonstrated that sustained happiness and sadness under naturalistic conditions are presented in emotion-specific network profiles and these expressions may play different roles in the generation and modulation of emotions. These findings elucidated the importance of network level adaptations for sustained emotional experiences during naturalistic contexts and open new venues for imaging network level contributions under naturalistic conditions.

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

Naturalistic stimulation (i.e., movies and auditory narratives of some minutes' length) has been a powerful approach to bringing more real-life experiences into laboratory experiments. Data-driven, intersubject correlation (ISC) analysis permits examining to what extent activity in a specific brain region correlates across participants during exposure to a naturalistic stimulus, as well as testing whether neural activity correlates with behavioral measures. Notably, most of the previous research with naturalistic stimuli was conducted using functional fMRI (fMRI). Here, we tested whether a naturalistic approach and the ISC are feasible using functional near-infrared spectroscopy (fNIRS) - the imaging method particularly suited for populations of patients and children. Fifty-three healthy adult participants watched twice a 3-min segment of a Charlie Chaplin movie while we recorded the brain activity on the surface of their prefrontal cortex using fNIRS. In addition, an independent group of 18 participants used a continuous scoring procedure to rate the extent to which they felt that different parts of the movie fragment were funny. Our two findings were as follows. First, we found higher-than-zero ISC in fNIRS signals in the prefrontal cortex lobes, a result that was particularly high in the oxygenated channels during the first repetition of the movie. Second, we found a significant negative correlation between oxygenated brain signals and ratings of the movie's humorousness. In a series of control analyses we demonstrated that this latter correlation could not be explained by various non-humor-related movie sensory properties (e.g., auditory volume and image brightness). The key overall outcome of the present study is that fNIRS in combination with the naturalistic paradigms and the ISC might be a sensitive and powerful research method to explore cognitive processing. Our results also suggest a potential role of the prefrontal cortex in humor appreciation.

Comparing the Effectiveness of Brain Structural Imaging, Resting-state fMRI, and Naturalistic fMRI in Recognizing Social Anxiety Disorder in Children and Adolescents

Social anxiety disorder (SAD) is a common anxiety disorder in childhood and adolescence. Studies on SAD in adults have reported both structural and functional aberrancies of the brain at the group level. However, evidence has shown differences in anxiety-related brain abnormalities between adolescents and adults. Since children and adolescents can afford limited scan time, optimizing the scan tasks is essential for SAD research in children and adolescents. Thus, we need to address whether brain structure, resting-state fMRI, and naturalistic imaging enable individualized identification of SAD in children and adolescents, which measurement is more effective, and whether pooling multi-modal features can improve the identification of SAD. We comprehensively addressed these questions by building machine learning models based on parcel-wise brain features. We found that naturalistic fMRI yielded higher classification accuracy (69.17%) than the other modalities and the classification performance showed dependence on the contents of the movie. The classification models also identified contributing brain regions, some of which exhibited correlations with the symptoms scores of SAD. However, pooling brain features from the three modalities did not help enhance the classification accuracy. These results support the application of carefully designed naturalistic imaging in recognizing children and adolescents at risk of SAD.

Movie Events Detecting Reveals Inter-Subject Synchrony Difference of Functional Brain Activity in Autism Spectrum Disorder

Recently, movie-watching fMRI has been recognized as a novel method to explore brain working patterns. Previous researchers correlated natural stimuli with brain responses to explore brain functional specialization by “reverse correlation” methods, which were based on within-group analysis. However, what external stimuli drove significantly different brain responses in two groups of different subjects were still unknown. To address this, sliding time windows technique combined with inter-Subject functional correlation (ISFC) was proposed to detect movie events with significant group differences between autism spectrum disorder (ASD) and typical development (TD) subjects. Then, using inter-Subject correlation (ISC) and ISFC analysis, we found that in three movie events involving character emotions, the ASD group showed significantly lower ISC in the middle temporal gyrus, temporal pole, cerebellum, caudate, precuneus, and showed decreased functional connectivity between large scale networks than that in TD. Under the movie event focusing on objects and scenes shot, the dorsal and ventral attentional networks of ASD had a strong synchronous response. Meanwhile, ASD also displayed increased functional connectivity between the frontoparietal network (FPN) and dorsal attention network (DAN), FPN, and sensorimotor network (SMN) than TD. ASD has its own unique synchronous response rather than being “unresponsive” in natural movie-watching. Our findings provide a new method and valuable insight for exploring the inconsistency of the brain “tick collectively” to same natural stimuli. This analytic approach has the potential to explore pathological mechanisms and promote training methods of ASD.