Visuospatial processing in early brain-based visual impairment is associated with differential recruitment of dorsal and ventral visual streams

Visuospatial processing impairments are prevalent in individuals with cerebral visual impairment (CVI) and are typically ascribed to "dorsal stream dysfunction" (DSD). However, the contribution of other cortical regions, including early visual cortex (EVC), frontal cortex, or the ventral visual stream, to such impairments remains unknown. Thus, here, we examined fMRI activity in these regions, while individuals with CVI (and neurotypicals) performed a visual search task within a dynamic naturalistic scene. First, behavioral performance was measured with eye tracking. Participants were instructed to search and follow a walking human target. CVI participants took significantly longer to find the target, and their eye gaze patterns were less accurate and less precise. Second, we used the same task in the MRI scanner. Along the dorsal stream, activation was reduced in CVI participants, consistent with the proposed DSD in CVI. Intriguingly, however, visual areas along the ventral stream showed the complete opposite pattern, with greater activation in CVI participants. In contrast, we found no differences in either EVC or frontal cortex between groups. These results suggest that the impaired visuospatial processing abilities in CVI are associated with differential recruitment of the dorsal and ventral visual streams, likely resulting from impaired selective attention.

Computational Language Modeling and the Promise of In Silico Experimentation

Language neuroscience currently relies on two major experimental paradigms: controlled experiments using carefully hand-designed stimuli, and natural stimulus experiments. These approaches have complementary advantages which allow them to address distinct aspects of the neurobiology of language, but each approach also comes with drawbacks. Here we discuss a third paradigm-in silico experimentation using deep learning-based encoding models-that has been enabled by recent advances in cognitive computational neuroscience. This paradigm promises to combine the interpretability of controlled experiments with the generalizability and broad scope of natural stimulus experiments. We show four examples of simulating language neuroscience experiments in silico and then discuss both the advantages and caveats of this approach.

Differential contributions of body form, motion, and temporal information to subjective action understanding in naturalistic stimuli

Introduction

We investigated the factors underlying naturalistic action recognition and understanding, as well as the errors occurring during recognition failures.

Methods

Participants saw full-light stimuli of ten different whole-body actions presented in three different conditions: as normal videos, as videos with the temporal order of the frames scrambled, and as single static representative frames. After each stimulus presentation participants completed one of two tasks-a forced choice task where they were given the ten potential action labels as options, or a free description task, where they could describe the action performed in each stimulus in their own words.

Results

While generally, a combination of form, motion, and temporal information led to the highest action understanding, for some actions form information was sufficient and adding motion and temporal information did not increase recognition accuracy. We also analyzed errors in action recognition and found primarily two different types.

Discussion

One type of error was on the semantic level, while the other consisted of reverting to the kinematic level of body part processing without any attribution of semantics. We elaborate on these results in the context of naturalistic action perception.

How a speaker herds the audience: Multi-brain neural convergence over time during naturalistic storytelling

Storytelling-an ancient way for humans to share individual experiences with others-has been found to induce neural synchronization among listeners. In our exploration of the dynamic fluctuations in listener-listener (LL) coupling throughout stories, we uncover a significant correlation between LL and lag-speaker-listener (lag-SL) couplings over time. Using the analogy of neural pattern (dis)similarity as distances between participants, we term this phenomenon the "herding effect": like a shepherd guiding a group of sheep, the more closely listeners follow the speaker's prior brain activity patterns (higher lag-SL similarity), the more tightly they cluster together (higher LL similarity). This herding effect is particularly pronounced in brain regions where neural synchronization among listeners tracks with behavioral ratings of narrative engagement, highlighting the mediating role of narrative content in the observed multi-brain neural coupling dynamics. By integrating LL and SL neural couplings, this study illustrates how unfolding stories shape a dynamic multi-brain functional network and how the configuration of this network may be associated with moment-by-moment efficacy of communication.

Sense of own body shapes neural processes of memory encoding and reinstatement

How is the fundamental sense of one's body, a basic aspect of selfhood, incorporated into memories for events? Disrupting bodily self-awareness during encoding impairs functioning of the left posterior hippocampus during retrieval, which implies weakened encoding. However, how changes in bodily self-awareness influence neural encoding is unknown. We investigated how the sense of body ownership, a core aspect of the bodily self, impacts encoding in the left posterior hippocampus and additional core memory regions including the angular gyrus. Furthermore, we assessed the degree to which memories are reinstated according to body ownership during encoding and vividness during retrieval as a measure of memory strength. We immersed participants in naturalistic scenes where events unfolded while we manipulated feelings of body ownership with a full-body-illusion during functional magnetic resonance imaging scanning. One week later, participants retrieved memories for the videos during functional magnetic resonance imaging scanning. A whole brain analysis revealed that patterns of activity in regions including the right hippocampus and angular gyrus distinguished between events encoded with strong versus weak body ownership. A planned region-of-interest analysis showed that patterns of activity in the left posterior hippocampus specifically could predict body ownership during memory encoding. Using the wider network of regions sensitive to body ownership during encoding and the left posterior hippocampus as separate regions-of-interest, we observed that patterns of activity present at encoding were reinstated more during the retrieval of events encoded with strong body ownership and high memory vividness. Our results demonstrate how the sense of physical self is bound within an event during encoding, which facilitates reactivation of a memory trace during retrieval.

Cross-movie prediction of individualized functional topography

Participant-specific, functionally defined brain areas are usually mapped with functional localizers and estimated by making contrasts between responses to single categories of input. Naturalistic stimuli engage multiple brain systems in parallel, provide more ecologically plausible estimates of real-world statistics, and are friendly to special populations. The current study shows that cortical functional topographies in individual participants can be estimated with high fidelity from naturalistic stimuli. Importantly, we demonstrate that robust, individualized estimates can be obtained even when participants watched different movies, were scanned with different parameters/scanners, and were sampled from different institutes across the world. Our results create a foundation for future studies that allow researchers to estimate a broad range of functional topographies based on naturalistic movies and a normative database, making it possible to integrate high-level cognitive functions across datasets from laboratories worldwide.

Probing neurodynamics of experienced emotions-a Hitchhiker's guide to film fMRI

Film functional magnetic resonance imaging (fMRI) has gained tremendous popularity in many areas of neuroscience. However, affective neuroscience remains somewhat behind in embracing this approach, even though films lend themselves to study how brain function gives rise to complex, dynamic and multivariate emotions. Here, we discuss the unique capabilities of film fMRI for emotion research, while providing a general guide of conducting such research. We first give a brief overview of emotion theories as these inform important design choices. Next, we discuss films as experimental paradigms for emotion elicitation and address the process of annotating them. We then situate film fMRI in the context of other fMRI approaches, and present an overview of results from extant studies so far with regard to advantages of film fMRI. We also give an overview of state-of-the-art analysis techniques including methods that probe neurodynamics. Finally, we convey limitations of using film fMRI to study emotion. In sum, this review offers a practitioners' guide to the emerging field of film fMRI and underscores how it can advance affective neuroscience.

Modeling naturalistic face processing in humans with deep convolutional neural networks

Deep convolutional neural networks (DCNNs) trained for face identification can rival and even exceed human-level performance. The ways in which the internal face representations in DCNNs relate to human cognitive representations and brain activity are not well understood. Nearly all previous studies focused on static face image processing with rapid display times and ignored the processing of naturalistic, dynamic information. To address this gap, we developed the largest naturalistic dynamic face stimulus set in human neuroimaging research (700+ naturalistic video clips of unfamiliar faces). We used this naturalistic dataset to compare representational geometries estimated from DCNNs, behavioral responses, and brain responses. We found that DCNN representational geometries were consistent across architectures, cognitive representational geometries were consistent across raters in a behavioral arrangement task, and neural representational geometries in face areas were consistent across brains. Representational geometries in late, fully connected DCNN layers, which are optimized for individuation, were much more weakly correlated with cognitive and neural geometries than were geometries in late-intermediate layers. The late-intermediate face-DCNN layers successfully matched cognitive representational geometries, as measured with a behavioral arrangement task that primarily reflected categorical attributes, and correlated with neural representational geometries in known face-selective topographies. Our study suggests that current DCNNs successfully capture neural cognitive processes for categorical attributes of faces but less accurately capture individuation and dynamic features.

Descending neurons of the hoverfly respond to pursuits of artificial targets

Many animals use motion vision information to control dynamic behaviors. Predatory animals, for example, show an exquisite ability to detect rapidly moving prey, followed by pursuit and capture. Such target detection is not only used by predators but is also important in conspecific interactions, such as for male hoverflies defending their territories against conspecific intruders. Visual target detection is believed to be subserved by specialized target-tuned neurons found in a range of species, including vertebrates and arthropods. However, how these target-tuned neurons respond to actual pursuit trajectories is currently not well understood. To redress this, we recorded extracellularly from target-selective descending neurons (TSDNs) in male Eristalis tenax hoverflies. We show that they have dorso-frontal receptive fields with a preferred direction up and away from the visual midline. We reconstructed visual flow fields as experienced during pursuits of artificial targets (black beads). We recorded TSDN responses to six reconstructed pursuits and found that each neuron responded consistently at remarkably specific time points but that these time points differed between neurons. We found that the observed spike probability was correlated with the spike probability predicted from each neuron's receptive field and size tuning. Interestingly, however, the overall response rate was low, with individual neurons responding to only a small part of each reconstructed pursuit. In contrast, the TSDN population responded to substantially larger proportions of the pursuits but with lower probability. This large variation between neurons could be useful if different neurons control different parts of the behavioral output.

The effect of repetition on intersubject synchrony assessed with fMRI

We investigated how repeated exposure to a stimulus affects intersubject synchrony in the brains of young and older adults. We used functional magnetic resonance imaging (fMRI) to measure brain responses to familiar and novel stimuli. Young adults participated in a familiarization paradigm designed to mimic 'natural' exposure while older adults were presented with stimuli they had known for more than 50 years. Intersubject synchrony was calculated to detect common stimulus-driven brain activity across young and older adults as they listened to the novel and familiar stimuli. Contrary to our hypotheses, synchrony was not related to the amount of stimulus exposure; both young and older adults showed more synchrony to novel than to familiar stimuli regardless of whether the stimuli had been heard once, known for a few weeks, or known for more than 50 years. In young adults these synchrony differences were found across the brain in the bilateral temporal lobes, and in the frontal orbital cortex. In older adults the synchrony differences were found only in the bilateral temporal lobes. This reduction may be related to an increase in idiosyncratic responses after exposure to a stimulus but does not seem to be related to how well the stimuli are learned or to differences in attention. Until the effects of repeated exposure on synchrony are fully understood, future studies using intersubject synchrony, where the novelty of the stimuli cannot be guaranteed, may consider exposing all of their participants to the stimuli once before data are collected to mitigate the effects of any systematic differences in stimulus exposure.

Mapping the time-varying functional brain networks in response to naturalistic movie stimuli

One of human brain's remarkable traits lies in its capacity to dynamically coordinate the activities of multiple brain regions or networks, adapting to an externally changing environment. Studying the dynamic functional brain networks (DFNs) and their role in perception, assessment, and action can significantly advance our comprehension of how the brain responds to patterns of sensory input. Movies provide a valuable tool for studying DFNs, as they offer a naturalistic paradigm that can evoke complex cognitive and emotional experiences through rich multimodal and dynamic stimuli. However, most previous research on DFNs have predominantly concentrated on the resting-state paradigm, investigating the topological structure of temporal dynamic brain networks generated via chosen templates. The dynamic spatial configurations of the functional networks elicited by naturalistic stimuli demand further exploration. In this study, we employed an unsupervised dictionary learning and sparse coding method combing with a sliding window strategy to map and quantify the dynamic spatial patterns of functional brain networks (FBNs) present in naturalistic functional magnetic resonance imaging (NfMRI) data, and further evaluated whether the temporal dynamics of distinct FBNs are aligned to the sensory, cognitive, and affective processes involved in the subjective perception of the movie. The results revealed that movie viewing can evoke complex FBNs, and these FBNs were time-varying with the movie storylines and were correlated with the movie annotations and the subjective ratings of viewing experience. The reliability of DFNs was also validated by assessing the Intra-class coefficient (ICC) among two scanning sessions under the same naturalistic paradigm with a three-month interval. Our findings offer novel insight into comprehending the dynamic properties of FBNs in response to naturalistic stimuli, which could potentially deepen our understanding of the neural mechanisms underlying the brain's dynamic changes during the processing of visual and auditory stimuli.

A Novel Exploratory Graph-Based Analytical Tool for Functional Near-Infrared Spectroscopy in Naturalistic Experiments: An Illustrative Application in Typically Developing Children

Naturalistic paradigms are being increasingly applied to investigate human brain function. Compared with resting-state and task-based paradigms in neuroimaging, naturalistic stimuli and situations can be potentially more readily translated to daily-life applications. Among neuroimaging modalities, functional near-infrared spectroscopy (fNIRS) is particularly suitable for naturalistic investigations and applications. However, specific and tailored statistical analysis to interrogate brain function using naturalistic fNIRS is warranted. Here, we describe an exploratory graph-centrality-based approach to investigating participants' spatiotemporal similarities from the fNIRS signal. We illustrate the usefulness of our approach in a sample of typically developing children (10 males and 9 females; mean age of 5.2 years old; sd = 0.78) while they watch the Inscapes movie designed for neuroimaging acquisition. A node in the left dorsal prefrontal cortex presented similar responses across children, and those fNIRS responses were in line with scene transitions in the movie stimulus. Our results suggest the feasibility of applying centrality graph-based measures to investigate brain function in naturalistic fNIRS during development.

Higher levels of narrativity lead to similar patterns of posterior EEG activity across individuals

Introduction

The focus of cognitive and psychological approaches to narrative has not so much been on the elucidation of important aspects of narrative, but rather on using narratives as tools for the investigation of higher order cognitive processes elicited by narratives (e.g., understanding, empathy, etc.). In this study, we work toward a scalar model of narrativity, which can provide testable criteria for selecting and classifying communication forms in their level of narrativity. We investigated whether being exposed to videos with different levels of narrativity modulates shared neural responses, measured by inter-subject correlation, and engagement levels.

Methods

Thirty-two participants watched video advertisements with high-level and low-level of narrativity while their neural responses were measured through electroencephalogram. Additionally, participants' engagement levels were calculated based on the composite of their self-reported attention and immersion scores.

Results

Results demonstrated that both calculated inter-subject correlation and engagement scores for high-level video ads were significantly higher than those for low-level, suggesting that narrativity levels modulate inter-subject correlation and engagement.

Discussion

We believe that these findings are a step toward the elucidation of the viewers' way of processing and understanding a given communication artifact as a function of the narrative qualities expressed by the level of narrativity.

Individual differences in neural event segmentation of continuous experiences

Event segmentation is a spontaneous part of perception, important for processing continuous information and organizing it into memory. Although neural and behavioral event segmentation show a degree of inter-subject consistency, meaningful individual variability exists atop these shared patterns. Here we characterized individual differences in the location of neural event boundaries across four short movies that evoked variable interpretations. Event boundary alignment across subjects followed a posterior-to-anterior gradient that was tightly correlated with the rate of segmentation: slower-segmenting regions that integrate information over longer time periods showed more individual variability in boundary locations. This relationship held irrespective of the stimulus, but the degree to which boundaries in particular regions were shared versus idiosyncratic depended on certain aspects of movie content. Furthermore, this variability was behaviorally significant in that similarity of neural boundary locations during movie-watching predicted similarity in how the movie was ultimately remembered and appraised. In particular, we identified a subset of regions in which neural boundary locations are both aligned with behavioral boundaries during encoding and predictive of stimulus interpretation, suggesting that event segmentation may be a mechanism by which narratives generate variable memories and appraisals of stimuli.

Deeper Than You Think: Partisanship-Dependent Brain Responses in Early Sensory and Motor Brain Regions

Recent political polarization has illustrated how individuals with opposing political views often experience ongoing events in markedly different ways. In this study, we explored the neural mechanisms underpinning this phenomenon. We conducted fMRI scanning of 34 right- and left-wing participants (45% females) watching political videos (e.g., campaign ads and political speeches) just before the elections in Israel. As expected, we observed significant differences between left- and right-wing participants in their interpretation of the videos' content. Furthermore, neuroimaging results revealed partisanship-dependent differences in activation and synchronization in higher-order regions. Surprisingly, such differences were also revealed in early sensory, motor, and somatosensory regions. We found that the political content synchronized the responses of primary visual and auditory cortices in a partisanship-dependent manner. Moreover, right-wing (and not left-wing) individuals' sensorimotor cortex was involved in processing right-wing (and not left-wing) political content. These differences were pronounced to the extent that we could predict political orientation from the early brain-response alone. Importantly, no such differences were found with respect to neutral content. Therefore, these results uncover more fundamental neural mechanisms underlying processes of political polarization.SIGNIFICANCE STATEMENT The political sphere has become highly polarized in recent years. Would it be possible to identify the neural mechanisms underpinning such processes? In our study, left- and right-wing participants were scanned in fMRI while watching political video clips just before the elections in Israel. We found that political content was potent in synchronizing the brain responses of individuals holding similar views. This was far more pronounced in individuals holding right-wing views. Moreover, partisan-dependent differences in neural responses were identified already in early sensory, somatosensory, and motor regions, and only for political content. These results suggest that individuals' political views shape their neural responses at a very basic level.

Neural unscrambling of temporal information during a nonlinear narrative

Although we must experience our lives chronologically, storytellers often manipulate the order in which they relay events. How the brain processes temporal information while encoding a nonlinear narrative remains unclear. Here, we use functional magnetic resonance imaging during movie watching to investigate which brain regions are sensitive to information about time in a narrative and test whether the representation of temporal context across a narrative is more influenced by the order in which events are presented or their underlying chronological sequence. Results indicate that medial parietal regions are sensitive to cued jumps through time over and above other changes in context (i.e., location). Moreover, when processing non-chronological narrative information, the precuneus and posterior cingulate engage in on-the-fly temporal unscrambling to represent information chronologically. Specifically, days that are closer together in chronological time are represented more similarly regardless of when they are presented in the movie, and this representation is consistent across participants. Additional analyses reveal a strong spatial signature associated with higher magnitude jumps through time. These findings are consistent with prior theorizing on medial parietal regions as central to maintaining and updating narrative situation models, and suggest the priority of chronological information when encoding narrative events.

Developmental differences in the perception of naturalistic human movements

Introduction

It is widely believed that we are more attentive towards moving versus static stimuli. However, the neural correlates underlying the perception of human movements have not been extensively investigated in ecologically valid settings, nor has the developmental aspect of this phenomenon. Here, we set forth to investigate how human limb movements displayed in naturalistic videos influence the attentional engagement of children and young adults.

Methods

Thirty-nine healthy participants (4-26 years old) were presented with naturalistic videos featuring human goal-directed movements, while neural activity was recorded using electroencephalography (EEG). Video scenes were automatically annotated as containing arm, leg or no movement, using a machine learning algorithm. The viewers' attentional engagement was quantified by the intersubject correlation of EEG responses evoked by the videos.

Results

Our results demonstrate that scenes featuring limb movements, especially simultaneous arm and leg movements, elicit higher attentional engagement than scenes with no limb movement. Interestingly, this effect was found to diminish with age.

Discussion

Overall, our findings extend previous work on the perception of human motion by implementing naturalistic stimuli in the experimental design and extend the list of factors influencing the viewer's engagement exerted by naturalistic videos.

Computational meaningfulness as the source of beneficial cognitive biases

The human brain has evolved to solve the problems it encounters in multiple environments. In solving these challenges, it forms mental simulations about multidimensional information about the world. These processes produce context-dependent behaviors. The brain as overparameterized modeling organ is an evolutionary solution for producing behavior in a complex world. One of the most essential characteristics of living creatures is that they compute the values of information they receive from external and internal contexts. As a result of this computation, the creature can behave in optimal ways in each environment. Whereas most other living creatures compute almost exclusively biological values (e.g., how to get food), the human as a cultural creature computes meaningfulness from the perspective of one's activity. The computational meaningfulness means the process of the human brain, with the help of which an individual tries to make the respective situation comprehensible to herself to know how to behave optimally. This paper challenges the bias-centric approach of behavioral economics by exploring different possibilities opened up by computational meaningfulness with insight into wider perspectives. We concentrate on confirmation bias and framing effect as behavioral economics examples of cognitive biases. We conclude that from the computational meaningfulness perspective of the brain, the use of these biases are indispensable property of an optimally designed computational system of what the human brain is like. From this perspective, cognitive biases can be rational under some conditions. Whereas the bias-centric approach relies on small-scale interpretable models which include only a few explanatory variables, the computational meaningfulness perspective emphasizes the behavioral models, which allow multiple variables in these models. People are used to working in multidimensional and varying environments. The human brain is at its best in such an environment and scientific study should increasingly take place in such situations simulating the real environment. By using naturalistic stimuli (e.g., videos and VR) we can create more realistic, life-like contexts for research purposes and analyze resulting data using machine learning algorithms. In this manner, we can better explain, understand and predict human behavior and choice in different contexts.

A Large Video Set of Natural Human Actions for Visual and Cognitive Neuroscience Studies and Its Validation with fMRI

The investigation of the perception of others' actions and underlying neural mechanisms has been hampered by the lack of a comprehensive stimulus set covering the human behavioral repertoire. To fill this void, we present a video set showing 100 human actions recorded in natural settings, covering the human repertoire except for emotion-driven (e.g., sexual) actions and those involving implements (e.g., tools). We validated the set using fMRI and showed that observation of the 100 actions activated the well-established action observation network. We also quantified the videos' low-level visual features (luminance, optic flow, and edges). Thus, this comprehensive video set is a valuable resource for perceptual and neuronal studies.

Information flow across the cortical timescale hierarchy during narrative construction

When listening to spoken narratives, we must integrate information over multiple, concurrent timescales, building up from words to sentences to paragraphs to a coherent narrative. Recent evidence suggests that the brain relies on a chain of hierarchically organized areas with increasing temporal receptive windows to process naturalistic narratives. We hypothesized that the structure of this cortical processing hierarchy should result in an observable sequence of response lags between networks comprising the hierarchy during narrative comprehension. This study uses functional MRI to estimate the response lags between functional networks during narrative comprehension. We use intersubject cross-correlation analysis to capture network connectivity driven by the shared stimulus. We found a fixed temporal sequence of response lags-on the scale of several seconds-starting in early auditory areas, followed by language areas, the attention network, and lastly the default mode network. This gradient is consistent across eight distinct stories but absent in data acquired during rest or using a scrambled story stimulus, supporting our hypothesis that narrative construction gives rise to internetwork lags. Finally, we build a simple computational model for the neural dynamics underlying the construction of nested narrative features. Our simulations illustrate how the gradual accumulation of information within the boundaries of nested linguistic events, accompanied by increased activity at each level of the processing hierarchy, can give rise to the observed lag gradient.

Social engagement revealed by gaze following in third-party observers of simulated social conflict

Humans and non-human primates can allocate visual attention to areas of high interest in their visual field based on the behaviors of their social partners. Allocation of attention is particularly important for third-party observers of social interactions. By following the gaze of interacting individuals, the observer can obtain information about the mental states, emotions, and intentions of others. We presented three adult monkeys (Macaca mulatta) with videos of simulated social interactions and quantified their eye movements to determine which observed behaviors were most conducive to gaze following. Social interactions were simulated by juxtaposing two videos depicting a threatening and an appeasing individual facing each other, with the timing of the facial and bodily displays adjusted to mimic an exchange of social signals. Socially meaningful facial displays combined with full body movements significantly enhanced the probability of gaze following and joint attention. Despite the synthetic nature of these interactions, the facial and bodily displays of the submissive individual elicited significantly more joint-attention than gaze-following saccades, suggesting a preferential allocation of attention to the recipients of threatening displays. Temporal alignment of gaze following and joint attention to the frames of each video showed numerous clusters of significant increases in the frequency of these saccades. These clusters suggest that some videos contained signals that can induce a quasi-automatic redirection of the observer's attention. However, these saccades occurred only on a fraction of the viewings, and we have documented large inter-individual variations. All viewers produced sequences of joint attention saccades (check-backs) shifting their attention between the two monkeys as though monitoring the simulated emitting-receiving cycle of social signals. These sequences reflect the viewer's interest in monitoring the ongoing exchange of agonistic and affiliative displays. It appears that in macaque monkeys, the scanpaths of third-party observers of simulated social interactions are informed by social-cognitive processes suggestive of mentalizing.

Vascular feature as a modulator of the aging brain

The cerebral functional reorganization and declined cognitive function of aging might associate with altered vascular features. Here, we explored the altered cerebral hierarchical functional network of 2 conditions (task-free and naturalistic stimuli) in older adults and its relationship with vascular features (systemic microvascular and perfusion features, measured by magnetic resonance imaging) and behavior. Using cerebral gradient analysis, we found that compressive gradient of resting-state mainly located on the primary sensory-motor system and transmodal regions in aging, and further compress in these regions under the continuous naturalistic stimuli. Combining cerebral functional gradient, vascular features, and cognitive performance, the more compressive gradient in the resting-state, the worse vascular state, the lower cognitive function in older adults. Further modulation analysis demonstrated that both vascular features can regulate the relationship between gradient scores in the insula and behavior. Interestingly, systemic microvascular oxygenation also can modulate the relationship between cerebral gradient and cerebral perfusion. Furthermore, the less alteration of the compressive gradient with naturalistic stimuli came with lower cognitive function. Our findings demonstrated that the altered cerebral hierarchical functional structure in aging was linked with changed vascular features and behavior, offering a new framework for studying the physiological mechanism of functional connectivity in aging.

The promises and pitfalls of functional magnetic resonance imaging hyperscanning for social interaction research

Social neuroscience combines tools and perspectives from social psychology and neuroscience to understand how people interact with their social world. Here we discuss a relatively new method-hyperscanning-to study real-time, interactive social interactions using functional magnetic resonance imaging (fMRI). We highlight three contributions that fMRI hyperscanning makes to the study of the social mind: (1) Naturalism: it shifts the focus from tightly-controlled stimuli to more naturalistic social interactions; (2) Multi-person Dynamics: it shifts the focus from individuals as the unit of analysis to dyads and groups; and (3) Neural Resolution: fMRI hyperscanning captures high-resolution neural patterns and dynamics across the whole brain, unlike other neuroimaging hyperscanning methods (e.g., electroencephalogram, functional near-infrared spectroscopy). Finally, we describe the practical considerations and challenges that fMRI hyperscanning researchers must navigate. We hope researchers will harness this powerful new paradigm to address pressing questions in today's society.

Visual continuous recognition reveals behavioral and neural differences for short- and long-term scene memory

Humans have a remarkably high capacity and long duration memory for complex scenes. Previous research documents the neural substrates that allow for efficient categorization of scenes from other complex stimuli like objects and faces, but the spatiotemporal neural dynamics underlying scene memory at timescales relevant to working and longer-term memory are less well understood. In the present study, we used high density EEG during a visual continuous recognition task in which new, old, and scrambled scenes consisting of color outdoor photographs were presented at an average rate 0.26 Hz. Old scenes were single repeated presentations occurring within either a short-term (< 20 s) or longer-term intervals of between 30 s and 3 min or 4 and 10 min. Overall recognition was far above chance, with better performance at shorter- than longer-term intervals. Sensor-level ANOVA and post hoc pairwise comparisons of event related potentials (ERPs) revealed three main findings: (1) occipital and parietal amplitudes distinguishing new and old from scrambled scenes; (2) frontal amplitudes distinguishing old from new scenes with a central positivity highest for hits compared to misses, false alarms and correct rejections; and (3) frontal and parietal changes from ∼300 to ∼600 ms distinguishing among old scenes previously encountered at short- and longer-term retention intervals. These findings reveal how distributed spatiotemporal neural changes evolve to support short- and longer-term recognition of complex scenes.

Functional near-infrared spectroscopy imaging of the prefrontal cortex during a naturalistic comedy movie

Naturalistic stimulation (i.e., movies and auditory narratives of some minutes' length) has been a powerful approach to bringing more real-life experiences into laboratory experiments. Data-driven, intersubject correlation (ISC) analysis permits examining to what extent activity in a specific brain region correlates across participants during exposure to a naturalistic stimulus, as well as testing whether neural activity correlates with behavioral measures. Notably, most of the previous research with naturalistic stimuli was conducted using functional fMRI (fMRI). Here, we tested whether a naturalistic approach and the ISC are feasible using functional near-infrared spectroscopy (fNIRS) - the imaging method particularly suited for populations of patients and children. Fifty-three healthy adult participants watched twice a 3-min segment of a Charlie Chaplin movie while we recorded the brain activity on the surface of their prefrontal cortex using fNIRS. In addition, an independent group of 18 participants used a continuous scoring procedure to rate the extent to which they felt that different parts of the movie fragment were funny. Our two findings were as follows. First, we found higher-than-zero ISC in fNIRS signals in the prefrontal cortex lobes, a result that was particularly high in the oxygenated channels during the first repetition of the movie. Second, we found a significant negative correlation between oxygenated brain signals and ratings of the movie's humorousness. In a series of control analyses we demonstrated that this latter correlation could not be explained by various non-humor-related movie sensory properties (e.g., auditory volume and image brightness). The key overall outcome of the present study is that fNIRS in combination with the naturalistic paradigms and the ISC might be a sensitive and powerful research method to explore cognitive processing. Our results also suggest a potential role of the prefrontal cortex in humor appreciation.

Neural Processing of Naturalistic Echolocation Signals in Bats

Echolocation behavior, a navigation strategy based on acoustic signals, allows scientists to explore neural processing of behaviorally relevant stimuli. For the purpose of orientation, bats broadcast echolocation calls and extract spatial information from the echoes. Because bats control call emission and thus the availability of spatial information, the behavioral relevance of these signals is undiscussable. While most neurophysiological studies, conducted in the past, used synthesized acoustic stimuli that mimic portions of the echolocation signals, recent progress has been made to understand how naturalistic echolocation signals are encoded in the bat brain. Here, we review how does stimulus history affect neural processing, how spatial information from multiple objects and how echolocation signals embedded in a naturalistic, noisy environment are processed in the bat brain. We end our review by discussing the huge potential that state-of-the-art recording techniques provide to gain a more complete picture on the neuroethology of echolocation behavior.

Asymmetrical cross-modal influence on neural encoding of auditory and visual features in natural scenes

Natural scenes contain multi-modal information, which is integrated to form a coherent perception. Previous studies have demonstrated that cross-modal information can modulate neural encoding of low-level sensory features. These studies, however, mostly focus on the processing of single sensory events or rhythmic sensory sequences. Here, we investigate how the neural encoding of basic auditory and visual features is modulated by cross-modal information when the participants watch movie clips primarily composed of non-rhythmic events. We presented audiovisual congruent and audiovisual incongruent movie clips, and since attention can modulate cross-modal interactions, we separately analyzed high- and low-arousal movie clips. We recorded neural responses using electroencephalography (EEG), and employed the temporal response function (TRF) to quantify the neural encoding of auditory and visual features. The neural encoding of sound envelope is enhanced in the audiovisual congruent condition than the incongruent condition, but this effect is only significant for high-arousal movie clips. In contrast, audiovisual congruency does not significantly modulate the neural encoding of visual features, e.g., luminance or visual motion. In summary, our findings demonstrate asymmetrical cross-modal interactions during the processing of natural scenes that lack rhythmicity: Congruent visual information enhances low-level auditory processing, while congruent auditory information does not significantly modulate low-level visual processing.

Naturalistic Audio-Movies reveal common spatial organization across "visual" cortices of different blind individuals

Occipital cortices of different sighted people contain analogous maps of visual information (e.g. foveal vs. peripheral). In congenital blindness, "visual" cortices respond to nonvisual stimuli. Do visual cortices of different blind people represent common informational maps? We leverage naturalistic stimuli and inter-subject pattern similarity analysis to address this question. Blindfolded sighted (n = 22) and congenitally blind (n = 22) participants listened to 6 sound clips (5-7 min each): 3 auditory excerpts from movies; a naturalistic spoken narrative; and matched degraded auditory stimuli (Backwards Speech, scrambled sentences), during functional magnetic resonance imaging scanning. We compared the spatial activity patterns evoked by each unique 10-s segment of the different auditory excerpts across blind and sighted people. Segments of meaningful naturalistic stimuli produced distinctive activity patterns in frontotemporal networks that were shared across blind and across sighted individuals. In the blind group only, segment-specific, cross-subject patterns emerged in visual cortex, but only for meaningful naturalistic stimuli and not Backwards Speech. Spatial patterns of activity within visual cortices are sensitive to time-varying information in meaningful naturalistic auditory stimuli in a broadly similar manner across blind individuals.

The hippocampus constructs narrative memories across distant events

Life's events are scattered throughout time, yet we often recall different events in the context of an integrated narrative. Prior research suggests that the hippocampus, which supports memory for past events, can support the integration of overlapping associations or separate events in memory. However, the conditions that lead to hippocampus-dependent memory integration are unclear. We used functional brain imaging to test whether the opportunity to form a larger narrative (narrative coherence) drives hippocampal memory integration. During encoding of fictional stories, patterns of hippocampal activity, including activity at boundaries between events, were more similar between distant events that formed one coherent narrative, compared with overlapping events taken from unrelated narratives. One day later, the hippocampus preferentially supported detailed recall of coherent narrative events, through reinstatement of hippocampal activity patterns from encoding. These findings demonstrate a key function of the hippocampus: the integration of events into a narrative structure for memory.

Decoding subjective emotional arousal from EEG during an immersive virtual reality experience

Immersive virtual reality (VR) enables naturalistic neuroscientific studies while maintaining experimental control, but dynamic and interactive stimuli pose methodological challenges. We here probed the link between emotional arousal, a fundamental property of affective experience, and parieto-occipital alpha power under naturalistic stimulation: 37 young healthy adults completed an immersive VR experience, which included rollercoaster rides, while their EEG was recorded. They then continuously rated their subjective emotional arousal while viewing a replay of their experience. The association between emotional arousal and parieto-occipital alpha power was tested and confirmed by (1) decomposing the continuous EEG signal while maximizing the comodulation between alpha power and arousal ratings and by (2) decoding periods of high and low arousal with discriminative common spatial patterns and a long short-term memory recurrent neural network. We successfully combine EEG and a naturalistic immersive VR experience to extend previous findings on the neurophysiology of emotional arousal towards real-world neuroscience.

Investigation of functional brain network reconfiguration during exposure to naturalistic stimuli using graph-theoretical analysis

Objective.One of the most significant features of the human brain is that it can dynamically reconfigure itself to adapt to a changing environment. However, dynamic interaction characteristics of the brain networks in naturalistic scenes remain unclear.Approach.We used open-source functional magnetic resonance imaging (fMRI) data from 15 participants who underwent fMRI scans while watching an audio-visual movie 'Forrest Gump'. The community detection algorithm based on inter-subject functional correlation was used to study the time-varying functional networks only induced by the movie stimuli. The whole brain reconfiguration patterns were quantified by the temporal co-occurrence matrix that describes the probability of two brain regions engage in the same community (or putative functional module) across time and the time-varying brain modularity. Four graph metrics of integration, recruitment, spatio-temporal diversity and within-community normalised centrality were further calculated to summarise the brain network dynamic roles and hub features in their spatio-temporal topology.Main results.Our results suggest that the networks that were involved in attention and audio-visual information processing, such as the visual network, auditory network, and dorsal attention network, were considered to play a role of 'stable loners'. By contrast, 'unstable loner' networks such as the default mode network (DMN) and fronto-parietal network tended to interact more flexibly with the other networks. In addition, global brain network showed significant fluctuations in modularity. The 'stable loner' networks always maintained high functional connectivity (FC) strength while 'unstable loner' networks, especially the DMN, exhibited high intra- and inter-network FC only during a low modularity period. Finally, changes in brain modularity were significantly associated with variations in emotions induced by the movie.Significance.Our findings provide new insight for understanding the dynamic interaction characteristics of functional brain networks during naturalistic stimuli.

Referential Chains Reveal Predictive Processes and Form-to-Function Mapping: An Electroencephalographic Study Using Naturalistic Story Stimuli

In discourse pragmatics, different referential forms are claimed to be indicative of the cognitive status of a referent in the current discourse. Referential expressions thereby possess a double function: They point back to an (existing) referent (form-to-function mapping), and they are used to derive predictions about a referent’s subsequent recurrence in discourse. Existing event-related potential (ERP) research has mainly focused on the form-to-function mapping of referential expression. In the present ERP study, we explore the relationship of form-to-function mapping and prediction derived from the antecedent of referential expressions in naturalistic auditory language comprehension. Specifically, the study investigates the relationship between the form of a referential expression (pronoun vs. noun) and the form of its antecedent (pronoun vs. noun); i.e., it examines the influence of the interplay of predictions derived from an antecedent (forward-looking function) and the form-to-function mapping of an anaphor (backward-looking function) on the ERPs time-locked to anaphoric expressions. The results in the time range of the P300 and N400 allow for a dissociation of these two functions during online language comprehension.

From Visual Perception to Aesthetic Appeal: Brain Responses to Aesthetically Appealing Natural Landscape Movies

During aesthetically appealing visual experiences, visual content provides a basis for computation of affectively tinged representations of aesthetic value. How this happens in the brain is largely unexplored. Using engaging video clips of natural landscapes, we tested whether cortical regions that respond to perceptual aspects of an environment (e.g., spatial layout, object content and motion) were directly modulated by rated aesthetic appeal. Twenty-four participants watched a series of videos of natural landscapes while being scanned using functional magnetic resonance imaging (fMRI) and reported both continuous ratings of enjoyment (during the videos) and overall aesthetic judgments (after each video). Although landscape videos engaged a greater expanse of high-level visual cortex compared to that observed for images of landscapes, independently localized category-selective visual regions (e.g., scene-selective parahippocampal place area and motion-selective hMT+) were not significantly modulated by aesthetic appeal. Rather, a whole-brain analysis revealed modulations by aesthetic appeal in ventral (collateral sulcus) and lateral (middle occipital sulcus, posterior middle temporal gyrus) clusters that were adjacent to scene and motion selective regions. These findings suggest that aesthetic appeal per se is not represented in well-characterized feature- and category-selective regions of visual cortex. Rather, we propose that the observed activations reflect a local transformation from a feature-based visual representation to a representation of "elemental affect," computed through information-processing mechanisms that detect deviations from an observer's expectations. Furthermore, we found modulation by aesthetic appeal in subcortical reward structures but not in regions of the default-mode network (DMN) nor orbitofrontal cortex, and only weak evidence for associated changes in functional connectivity. In contrast to other visual aesthetic domains, aesthetically appealing interactions with natural landscapes may rely more heavily on comparisons between ongoing stimulation and well-formed representations of the natural world, and less on top-down processes for resolving ambiguities or assessing self-relevance.

Functional Brain Connectivity During Narrative Processing Relates to Transportation and Story Influence

Engaging with narratives involves a complex array of cognitive and affective processes. These processes make stories persuasive in ways that standard arguments are not, though the underlying reasons for this remain unclear. Transportation theory proposes a potential explanation for this: narratives are processed in a way which makes individuals feel immersed in the world of a story, which in turn leads people to resonate emotionally with the events of the story. Recent fMRI studies have shown that the posterior medial cortex (PMC) and anterior insula (AI) play important roles in understanding the meaning of stories and experiencing the feelings they produce. In this study, we aimed to explore the AI’s and PMC’s role in narrative processing by measuring their functional connectivity with the rest of the brain during story listening, and how connectivity changes as a function of narrative transportation and the persuasiveness of the story. We analyzed data from 36 right-handed subjects who listened to two stories, obtained from podcasts, inside the fMRI scanner. After the scan, subjects were asked a series of questions, including a measure of how transported into the story they felt, how likely they would be to donate to causes related to the messages of the stories. We used searchlight multivariate pattern analysis (MVPA) to classify functional connectivity maps using seeds in both the AI and PMC and to compare these maps between participants who differed in transportation and prosocial intention. We found that connectivity to various regions successfully distinguished between high and low ratings on each of these behavioral measures with accuracies over 75%. However, only one pattern of connectivity was consistent across both stories: PMC-inferior frontal gyrus connectivity successfully distinguished high and low ratings of narrative transportation in both stories. All other findings were not consistent across stories. Instead, we found that patterns of connectivity may relate more to the specific content of the story rather than to a universal way in which narratives are processed.

Naturalistic Stimuli in Affective Neuroimaging: A Review

Naturalistic stimuli such as movies, music, and spoken and written stories elicit strong emotions and allow brain imaging of emotions in close-to-real-life conditions. Emotions are multi-component phenomena: relevant stimuli lead to automatic changes in multiple functional components including perception, physiology, behavior, and conscious experiences. Brain activity during naturalistic stimuli reflects all these changes, suggesting that parsing emotion-related processing during such complex stimulation is not a straightforward task. Here, I review affective neuroimaging studies that have employed naturalistic stimuli to study emotional processing, focusing especially on experienced emotions. I argue that to investigate emotions with naturalistic stimuli, we need to define and extract emotion features from both the stimulus and the observer.

EEG-Based Intersubject Correlations Reflect Selective Attention in a Competing Speaker Scenario

Several solutions have been proposed to study the relationship between ongoing brain activity and natural sensory stimuli, such as running speech. Computing the intersubject correlation (ISC) has been proposed as one possible approach. Previous evidence suggests that ISCs between the participants' electroencephalogram (EEG) may be modulated by attention. The current study addressed this question in a competing-speaker paradigm, where participants (N = 41) had to attend to one of two concurrently presented speech streams. ISCs between participants' EEG were higher for participants attending to the same story compared to participants attending to different stories. Furthermore, we found that ISCs between individual and group data predicted whether an individual attended to the left or right speech stream. Interestingly, the magnitude of the shared neural response with others attending to the same story was related to the individual neural representation of the attended and ignored speech envelope. Overall, our findings indicate that ISC differences reflect the magnitude of selective attentional engagement to speech.

Voxel-Based State Space Modeling Recovers Task-Related Cognitive States in Naturalistic fMRI Experiments

Complex natural tasks likely recruit many different functional brain networks, but it is difficult to predict how such tasks will be represented across cortical areas and networks. Previous electrophysiology studies suggest that task variables are represented in a low-dimensional subspace within the activity space of neural populations. Here we develop a voxel-based state space modeling method for recovering task-related state spaces from human fMRI data. We apply this method to data acquired in a controlled visual attention task and a video game task. We find that each task induces distinct brain states that can be embedded in a low-dimensional state space that reflects task parameters, and that attention increases state separation in the task-related subspace. Our results demonstrate that the state space framework offers a powerful approach for modeling human brain activity elicited by complex natural tasks.

Towards a multi-brain framework for hypnosis: a review of quantitative methods

Most real-world applications of hypnosis involve a pair of actors: a hypnotist and a subject. Accordingly, most current models of hypnosis acknowledge the relevance of social factors in the development of the hypnotic response. Yet, psychophysiological research on hypnosis has been mostly restricted to techniques that are studying one individual, neglecting the complexity of hypnosis as a social phenomenon. In this paper, we review evidence suggesting that a multi-brain approach to studying the psychophysiology of hypnosis could lead to a breakthrough in our understanding of the neural correlates of hypnosis. In particular, we aim to highlight how this approach which relies on the information conveyed by complex verbal stimuli can be utilized to deal with the multifaceted nature of hypnosis. Furthermore, we present analytical approaches to assessing brain-to-brain coupling developed in the field of social cognitive neuroscience in the past decade, to aid the design of similar multi-brain studies in hypnosis research.

Behavioral, Physiological, and Neural Signatures of Surprise during Naturalistic Sports Viewing

Surprise signals a discrepancy between past and current beliefs. It is theorized to be linked to affective experiences, the creation of particularly resilient memories, and segmentation of the flow of experience into discrete perceived events. However, the ability to precisely measure naturalistic surprise has remained elusive. We used advanced basketball analytics to derive a quantitative measure of surprise and characterized its behavioral, physiological, and neural correlates in human subjects observing basketball games. We found that surprise was associated with segmentation of ongoing experiences, as reflected by subjectively perceived event boundaries and shifts in neocortical patterns underlying belief states. Interestingly, these effects differed by whether surprising moments contradicted or bolstered current predominant beliefs. Surprise also positively correlated with pupil dilation, activation in subcortical regions associated with dopamine, game enjoyment, and long-term memory. These investigations support key predictions from event segmentation theory and extend theoretical conceptualizations of surprise to real-world contexts.

EEG Fingerprints under Naturalistic Viewing Using a Portable Device

The electroencephalogram (EEG) has been proven to be a promising technique for personal identification and verification. Recently, the aperiodic component of the power spectrum was shown to outperform other commonly used EEG features. Beyond that, EEG characteristics may capture relevant features related to emotional states. In this work, we aim to understand if the aperiodic component of the power spectrum, as shown for resting-state experimental paradigms, is able to capture EEG-based subject-specific features in a naturalistic stimuli scenario. In order to answer this question, we performed an analysis using two freely available datasets containing EEG recordings from participants during viewing of film clips that aim to trigger different emotional states. Our study confirms that the aperiodic components of the power spectrum, as evaluated in terms of offset and exponent parameters, are able to detect subject-specific features extracted from the scalp EEG. In particular, our results show that the performance of the system was significantly higher for the film clip scenario if compared with resting-state, thus suggesting that under naturalistic stimuli it is even easier to identify a subject. As a consequence, we suggest a paradigm shift, from task-based or resting-state to naturalistic stimuli, when assessing the performance of EEG-based biometric systems.

High-amplitude cofluctuations in cortical activity drive functional connectivity

Resting-state functional connectivity is used throughout neuroscience to study brain organization and to generate biomarkers of development, disease, and cognition. The processes that give rise to correlated activity are, however, poorly understood. Here we decompose resting-state functional connectivity using a temporal unwrapping procedure to assess the contributions of moment-to-moment activity cofluctuations to the overall connectivity pattern. This approach temporally resolves functional connectivity at a timescale of single frames, which enables us to make direct comparisons of cofluctuations of network organization with fluctuations in the blood oxygen level-dependent (BOLD) time series. We show that surprisingly, only a small fraction of frames exhibiting the strongest cofluctuation amplitude are required to explain a significant fraction of variance in the overall pattern of connection weights as well as the network's modular structure. These frames coincide with frames of high BOLD activity amplitude, corresponding to activity patterns that are remarkably consistent across individuals and identify fluctuations in default mode and control network activity as the primary driver of resting-state functional connectivity. Finally, we demonstrate that cofluctuation amplitude synchronizes across subjects during movie watching and that high-amplitude frames carry detailed information about individual subjects (whereas low-amplitude frames carry little). Our approach reveals fine-scale temporal structure of resting-state functional connectivity and discloses that frame-wise contributions vary across time. These observations illuminate the relation of brain activity to functional connectivity and open a number of directions for future research.

Visually evoked responses are enhanced when engaging in a video game

While it is well known that vision guides movement, less appreciated is that the motor cortex also provides input to the visual system. Here, we asked whether neural processing of visual stimuli is acutely modulated during motor activity, hypothesizing that visual evoked responses are enhanced when engaged in a motor task that depends on the visual stimulus. To test this, we told participants that their brain activity was controlling a video game that was in fact the playback of a prerecorded game. The deception, which was effective in half of participants, aimed to engage the motor system while avoiding evoked responses related to actual movement or somatosensation. In other trials, subjects actively played the game with keyboard control or passively watched a playback. The strength of visually evoked responses was measured as the temporal correlation between the continuous stimulus and the evoked potentials on the scalp. We found reduced correlation during passive viewing, but no difference between active and sham play. Alpha‐band (8–12 Hz) activity was reduced over central electrodes during sham play, indicating recruitment of motor cortex despite the absence of overt movement. To account for the potential increase of attention during gameplay, we conducted a second study with subjects counting screen items during viewing. We again found increased correlation during sham play, but no difference between counting and passive viewing. While we cannot fully rule out the involvement of attention, our findings do demonstrate an enhancement of visual evoked responses during active vision.

Narratives for Neuroscience

People organize and convey their thoughts according to narratives. However, neuroscientists are often reluctant to incorporate narrative stimuli into their experiments. We argue that narratives deserve wider adoption in human neuroscience because they tap into the brain's native machinery for representing the world and provide rich variability for testing hypotheses.

Learning Desire Is Predicted by Similar Neural Processing of Naturalistic Educational Materials

Naturalistic stimuli can elicit highly similar brain activity across viewers. How do naturalistic educational materials engage human brains and evoke learning desire? Here, we presented 15 audiovisual course clips (each lasting ∼120 s) to university students and recorded their neural activity through electroencephalography. Upon finishing all the video viewings, subjects ranked 15 courses in order of learning desire and reported the reasons for high learning desire (i.e., “value” and “interest”). The brain activity during the video viewing was measured as the neural similarity via intersubject correlation (ISC), that is, correlation between each subject’s neural responses and those of others. Based on averaged learning desire rankings across subjects, course clips were classified with high versus medium versus low motivational effectiveness. We found that the ISC of high effective course clips was larger than that of low effective ones. The ISC difference (high vs low) was positively associated with subjects’ learning desire difference (high vs low). Such an association occurred when viewing time accumulated to ∼80 s. Moreover, ISC was correlated with “interest-based” rather than “value-based” learning desire. These findings advance our understanding of learning motivation via the neural similarity in the context of on-line education and provide potential neurophysiological suggestions for pedagogical practices.

Measuring shared responses across subjects using intersubject correlation

Our capacity to jointly represent information about the world underpins our social experience. By leveraging one individual's brain activity to model another's, we can measure shared information across brains-even in dynamic, naturalistic scenarios where an explicit response model may be unobtainable. Introducing experimental manipulations allows us to measure, for example, shared responses between speakers and listeners or between perception and recall. In this tutorial, we develop the logic of intersubject correlation (ISC) analysis and discuss the family of neuroscientific questions that stem from this approach. We also extend this logic to spatially distributed response patterns and functional network estimation. We provide a thorough and accessible treatment of methodological considerations specific to ISC analysis and outline best practices.

Naturalistic Stimuli in Neuroscience: Critically Acclaimed

Cognitive neuroscience has traditionally focused on simple tasks, presented sparsely and using abstract stimuli. While this approach has yielded fundamental insights into functional specialisation in the brain, its ecological validity remains uncertain. Do these tasks capture how brains function 'in the wild', where stimuli are dynamic, multimodal, and crowded? Ecologically valid paradigms that approximate real life scenarios, using stimuli such as films, spoken narratives, music, and multiperson games emerged in response to these concerns over a decade ago. We critically appraise whether this approach has delivered on its promise to deliver new insights into brain function. We highlight the challenges, technological innovations, and clinical opportunities that are required should this field meet its full potential.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

During natural speech perception, humans must parse temporally continuous auditory and visual speech signals into sequences of words. However, most studies of speech perception present only single words or syllables. We used electrocorticography (subdural electrodes implanted on the brains of epileptic patients) to investigate the neural mechanisms for processing continuous audiovisual speech signals consisting of individual sentences. Using partial correlation analysis, we found that posterior superior temporal gyrus (pSTG) and medial occipital cortex tracked both the auditory and the visual speech envelopes. These same regions, as well as inferior temporal cortex, responded more strongly to a dynamic video of a talking face compared to auditory speech paired with a static face. Occipital cortex and pSTG carry temporal information about both auditory and visual speech dynamics. Visual speech tracking in pSTG may be a mechanism for enhancing perception of degraded auditory speech.

Neural Integration of Stimulus History Underlies Prediction for Naturalistically Evolving Sequences

Forming valid predictions about the environment is crucial to survival. However, whether humans are able to form valid predictions about natural stimuli based on their temporal statistical regularities remains unknown. Here, we presented subjects with tone sequences with pitch fluctuations that, over time, capture long-range temporal dependence structures prevalent in natural stimuli. We found that subjects were able to exploit such naturalistic statistical regularities to make valid predictions about upcoming items in a sequence. Magnetoencephalography (MEG) recordings revealed that slow, arrhythmic cortical dynamics tracked the evolving pitch sequence over time such that neural activity at a given moment was influenced by the pitch of up to seven previous tones. Importantly, such history integration contained in neural activity predicted the expected pitch of the upcoming tone, providing a concrete computational mechanism for prediction. These results establish humans' ability to make valid predictions based on temporal regularities inherent in naturalistic stimuli and further reveal the neural mechanisms underlying such predictive computation.SIGNIFICANCE STATEMENT A fundamental question in neuroscience is how the brain predicts upcoming events in the environment. To date, this question has primarily been addressed in experiments using relatively simple stimulus sequences. Here, we studied predictive processing in the human brain using auditory tone sequences that exhibit temporal statistical regularities similar to those found in natural stimuli. We observed that humans are able to form valid predictions based on such complex temporal statistical regularities. We further show that neural response to a given tone in the sequence reflects integration over the preceding tone sequence and that this history dependence forms the foundation for prediction. These findings deepen our understanding of how humans form predictions in an ecologically valid environment.

The Variability of Neural Responses to Naturalistic Videos Change with Age and Sex

Neural development is generally marked by an increase in the efficiency and diversity of neural processes. In a large sample (n = 114) of human children and adults with ages ranging from 5 to 44 yr, we investigated the neural responses to naturalistic video stimuli. Videos from both real-life classroom settings and Hollywood feature films were used to probe different aspects of attention and engagement. For all stimuli, older ages were marked by more variable neural responses. Variability was assessed by the intersubject correlation of evoked electroencephalographic responses. Young males also had less-variable responses than young females. These results were replicated in an independent cohort (n = 303). When interpreted in the context of neural maturation, we conclude that neural function becomes more variable with maturity, at least during the passive viewing of real-world stimuli.

Audiovisual Integration Varies With Target and Environment Richness in Immersive Virtual Reality

We are continually bombarded by information arriving to each of our senses; however, the brain seems to effortlessly integrate this separate information into a unified percept. Although multisensory integration has been researched extensively using simple computer tasks and stimuli, much less is known about how multisensory integration functions in real-world contexts. Additionally, several recent studies have demonstrated that multisensory integration varies tremendously across naturalistic stimuli. Virtual reality can be used to study multisensory integration in realistic settings because it combines realism with precise control over the environment and stimulus presentation. In the current study, we investigated whether multisensory integration as measured by the redundant signals effects (RSE) is observable in naturalistic environments using virtual reality and whether it differs as a function of target and/or environment cue-richness. Participants detected auditory, visual, and audiovisual targets which varied in cue-richness within three distinct virtual worlds that also varied in cue-richness. We demonstrated integrative effects in each environment-by-target pairing and further showed a modest effect on multisensory integration as a function of target cue-richness but only in the cue-rich environment. Our study is the first to definitively show that minimal and more naturalistic tasks elicit comparable redundant signals effects. Our results also suggest that multisensory integration may function differently depending on the features of the environment. The results of this study have important implications in the design of virtual multisensory environments that are currently being used for training, educational, and entertainment purposes.