Mobile cognition: imaging the human brain in the 'real world'

Closing the loop between environment, brain and mental health: how far we might go in real-life assessments?

PURPOSE OF REVIEW

Environmental factors such as climate, urbanicity, and exposure to nature are becoming increasingly important influencers of mental health. Incorporating data gathered from real-life contexts holds promise to substantially enhance laboratory experiments by providing a more comprehensive understanding of everyday behaviors in natural environments. We provide an up-to-date review of current technological and methodological developments in mental health assessments, neuroimaging and environmental sensing.

RECENT FINDINGS

Mental health research progressed in recent years towards integrating tools, such as smartphone based mental health assessments or mobile neuroimaging, allowing just-in-time daily assessments. Moreover, they are increasingly enriched by dynamic measurements of the environment, which are already being integrated with mental health assessments. To ensure ecological validity and accuracy it is crucial to capture environmental data with a high spatio-temporal granularity. Simultaneously, as a supplement to experimentally controlled conditions, there is a need for a better understanding of cognition in daily life, particularly regarding our brain's responses in natural settings.

SUMMARY

The presented overview on the developments and feasibility of "real-life" approaches for mental health and brain research and their potential to identify relationships along the mental health-environment-brain axis informs strategies for real-life individual and dynamic assessments.

Getting closer to social interactions using electroencephalography in developmental cognitive neuroscience

The field of developmental cognitive neuroscience is advancing rapidly, with large-scale, population-wide, longitudinal studies emerging as a key means of unraveling the complexity of the developing brain and cognitive processes in children. While numerous neuroscientific techniques like functional magnetic resonance imaging (fMRI), functional near-infrared spectroscopy (fNIRS), magnetoencephalography (MEG), and transcranial magnetic stimulation (TMS) have proved advantageous in such investigations, this perspective proposes a renewed focus on electroencephalography (EEG), leveraging underexplored possibilities of EEG. In addition to its temporal precision, low costs, and ease of application, EEG distinguishes itself with its ability to capture neural activity linked to social interactions in increasingly ecologically valid settings. Specifically, EEG can be measured during social interactions in the lab, hyperscanning can be used to study brain activity in two (or more) people simultaneously, and mobile EEG can be used to measure brain activity in real-life settings. This perspective paper summarizes research in these three areas, making a persuasive argument for the renewed inclusion of EEG into the toolkit of developmental cognitive and social neuroscientists.

Mobile neuroimaging: What we have learned about the neural control of human walking, with an emphasis on EEG-based research

Our understanding of the neural control of human walking has changed significantly over the last twenty years and mobile brain imaging methods have contributed substantially to current knowledge. High-density electroencephalography (EEG) has the advantages of being lightweight and mobile while providing temporal resolution of brain changes within a gait cycle. Advances in EEG hardware and processing methods have led to a proliferation of research on the neural control of locomotion in neurologically intact adults. We provide a narrative review of the advantages and disadvantages of different mobile brain imaging methods, then summarize findings from mobile EEG studies quantifying electrocortical activity during human walking. Contrary to historical views on the neural control of locomotion, recent studies highlight the widespread involvement of many areas, such as the anterior cingulate, posterior parietal, prefrontal, premotor, sensorimotor, supplementary motor, and occipital cortices, that show active fluctuations in electrical power during walking. The electrocortical activity changes with speed, stability, perturbations, and gait adaptation. We end with a discussion on the next steps in mobile EEG research.

Electrophysiological signatures of veridical head direction in humans

Information about heading direction is critical for navigation as it provides the means to orient ourselves in space. However, given that veridical head-direction signals require physical rotation of the head and most human neuroimaging experiments depend upon fixing the head in position, little is known about how the human brain is tuned to such heading signals. Here we adress this by asking 52 healthy participants undergoing simultaneous electroencephalography and motion tracking recordings (split into two experiments) and 10 patients undergoing simultaneous intracranial electroencephalography and motion tracking recordings to complete a series of orientation tasks in which they made physical head rotations to target positions. We then used a series of forward encoding models and linear mixed-effects models to isolate electrophysiological activity that was specifically tuned to heading direction. We identified a robust posterior central signature that predicts changes in veridical head orientation after regressing out confounds including sensory input and muscular activity. Both source localization and intracranial analysis implicated the medial temporal lobe as the origin of this effect. Subsequent analyses disentangled head-direction signatures from signals relating to head rotation and those reflecting location-specific effects. Lastly, when directly comparing head direction and eye-gaze-related tuning, we found that the brain maintains both codes while actively navigating, with stronger tuning to head direction in the medial temporal lobe. Together, these results reveal a taxonomy of population-level head-direction signals within the human brain that is reminiscent of those reported in the single units of rodents.

Biological motion perception in the theoretical framework of perceptual decision-making: An event-related potential study

Biological motion perception plays a critical role in various decisions in daily life. Failure to decide accordingly in such a perceptual task could have life-threatening consequences. Neurophysiology and computational modeling studies suggest two processes mediating perceptual decision-making. One of these signals is associated with the accumulation of sensory evidence and the other with response selection. Recent EEG studies with humans have introduced an event-related potential called Centroparietal Positive Potential (CPP) as a neural marker aligned with the sensory evidence accumulation while effectively distinguishing it from motor-related lateralized readiness potential (LRP). The present study aims to investigate the neural mechanisms of biological motion perception in the framework of perceptual decision-making, which has been overlooked before. More specifically, we examine whether CPP would track the coherence of the biological motion stimuli and could be distinguished from the LRP signal. We recorded EEG from human participants while they performed a direction discrimination task of a point-light walker stimulus embedded in various levels of noise. Our behavioral findings revealed shorter reaction times and reduced miss rates as the coherence of the stimuli increased. In addition, CPP tracked the coherence of the biological motion stimuli with a tendency to reach a common level during the response, albeit with a later onset than the previously reported results in random-dot motion paradigms. Furthermore, CPP was distinguished from the LRP signal based on its temporal profile. Overall, our results suggest that the mechanisms underlying perceptual decision-making generalize to more complex and socially significant stimuli like biological motion.

A synergetic turn in cognitive neuroscience of brain diseases

Despite significant improvements in our understanding of brain diseases, many barriers remain. Cognitive neuroscience faces four major challenges: complex structure-function associations; disease phenotype heterogeneity; the lack of transdiagnostic models; and oversimplified cognitive approaches restricted to the laboratory. Here, we propose a synergetics framework that can help to perform the necessary dimensionality reduction of complex interactions between the brain, body, and environment. The key solutions include low-dimensional spatiotemporal hierarchies for brain-structure associations, whole-brain modeling to handle phenotype diversity, model integration of shared transdiagnostic pathophysiological pathways, and naturalistic frameworks balancing experimental control and ecological validity. Creating whole-brain models with reduced manifolds combined with ecological measures can improve our understanding of brain disease and help identify novel interventions. Synergetics provides an integrated framework for future progress in clinical and cognitive neuroscience, pushing the boundaries of brain health and disease toward more mature, naturalistic approaches.

The promise and pitfalls of a strength-based approach to child poverty and neurocognitive development: Implications for policy

There has been significant progress in understanding the effects of childhood poverty on neurocognitive development. This progress has captured the attention of policymakers and promoted progressive policy reform. However, the prevailing emphasis on the harms associated with childhood poverty may have inadvertently perpetuated a deficit-based narrative, focused on the presumed shortcomings of children and families in poverty. This focus can have unintended consequences for policy (e.g., overlooking strengths) as well as public discourse (e.g., focusing on individual rather than systemic factors). Here, we join scientists across disciplines in arguing for a more well-rounded, "strength-based" approach, which incorporates the positive and/or adaptive developmental responses to experiences of social disadvantage. Specifically, we first show the value of this approach in understanding normative brain development across diverse human environments. We then highlight its application to educational and social policy, explore pitfalls and ethical considerations, and offer practical solutions to conducting strength-based research responsibly. Our paper re-ignites old and recent calls for a strength-based paradigm shift, with a focus on its application to developmental cognitive neuroscience. We also offer a unique perspective from a new generation of early-career researchers engaged in this work, several of whom themselves have grown up in conditions of poverty. Ultimately, we argue that a balanced strength-based scientific approach will be essential to building more effective policies.

Comparing episodic memory outcomes from walking augmented reality and stationary virtual reality encoding experiences

Episodic Memory (EM) is the neurocognitive capacity to consciously recollect personally experienced events in specific spatio-temporal contexts. Although the relevance of spatial and temporal information is widely acknowledged in the EM literature, it remains unclear whether and how EM performance and organisation is modulated by self-motion, and by motor- and visually- salient environmental features (EFs) of the encoding environment. This study examines whether and how EM is modulated by locomotion and the EFs encountered in a controlled lifelike learning route within a large-scale building. Twenty-eight healthy participants took part in a museum-tour encoding task implemented in walking Augmented Reality (AR) and stationary Virtual Reality (VR) conditions. EM performance and organisation were assessed immediately and 48-hours after trials using a Remember/Familiar recognition paradigm. Results showed a significant positive modulation effect of locomotion on distinctive EM aspects. Findings highlighted a significant performance enhancement effect of stairway-adjacent locations compared to dead-end and mid-route stimuli-presentation locations. The results of this study may serve as design criteria to facilitate neurocognitive rehabilitative interventions of EM. The underlying technological framework developed for this study represents a novel and ecologically sound method for evaluating EM processes in lifelike situations, allowing researchers a naturalistic perspective into the complex nature of EM.

Fixation-related potentials during mobile map assisted navigation in the real world: The effect of landmark visualization style

An often-proposed enhancement for mobile maps to aid assisted navigation is the presentation of landmark information, yet understanding of the manner in which they should be displayed is limited. In this study, we investigated whether the visualization of landmarks as 3D map symbols with either an abstract or realistic style influenced the subsequent processing of those landmarks during route navigation. We utilized a real-world mobile electroencephalography approach to this question by combining several tools developed to overcome the challenges typically encountered in real-world neuroscience research. We coregistered eye-movement and EEG recordings from 45 participants as they navigated through a real-world environment using a mobile map. Analyses of fixation event-related potentials revealed that the amplitude of the parietal P200 component was enhanced when participants fixated landmarks in the real world that were visualized on the mobile map in a realistic style, and that frontal P200 latencies were prolonged for landmarks depicted in either a realistic or abstract style compared with features of the environment that were not presented on the map, but only for the male participants. In contrast, we did not observe any significant effects of landmark visualization style on visual P1-N1 peaks or the parietal late positive component. Overall, the findings indicate that the cognitive matching process between landmarks seen in the environment and those previously seen on a map is facilitated by more realistic map display, while low-level perceptual processing of landmarks and recall of associated information are unaffected by map visualization style.

The Effect of Sensory Reweighting on Postural Control and Cortical Activity in Parkinson's Disease

Aims

Balance requires the cortical control of visual, somatosensory, and vestibular inputs. The aim of this cross-sectional study was to compare the contributions of each of these systems on postural control and cortical activity using a sensory reweighting approach between participants with Parkinson's disease (PD) and controls.

Methods

Ten participants with PD (age: 72 ± 9; 3 women; Hoehn & Yahr: 2 [1.5 - 2.50]) and 11 controls (age: 70 ± 3; 4 women) completed a sensory organization test in virtual reality (VR-SOT) while cortical activity was being recorded using electroencephalography (EEG). Conditions 1 to 3 were completed on a stable platform; conditions 4 to 6 on a foam. Conditions 1 and 4 were done with eyes open; conditions 2 and 5 in a darkened VR environment; and conditions 3 and 6 in a moving VR environment. Linear mixed models were used to evaluate changes in center of pressure (COP) displacement and EEG alpha and theta/beta ratio power between the two groups across the postural control conditions. Condition 1 was used as reference in all analyses.

Results

Participants with PD showed greater COP displacement than controls in the anteroposterior (AP) direction when relying on vestibular input (condition 5; p<0.0001). The mediolateral (ML) COP sway was greater in PD than in controls when relying on the somatosensory (condition 2; p = 0.03), visual (condition 4; p = 0.002), and vestibular (condition 5; p < 0.0001) systems. Participants with PD exhibited greater alpha power compared to controls when relying on visual input (condition 2; p = 0.003) and greater theta/beta ratio power when relying on somatosensory input (condition 4; p = 0.001).

Conclusions

PD affects reweighting of postural control, exemplified by greater COP displacement and increased cortical activity. Further research is needed to establish the temporal dynamics between cortical activity and COP displacement.

Learning of object-in-context sequences in freely-moving macaques

Flexible learning is a hallmark of primate cognition, which arises through interactions with changing environments. Studies of the neural basis for this flexibility are typically limited by laboratory settings that use minimal environmental cues and restrict interactions with the environment, including active sensing and exploration. To address this, we constructed a 3-D enclosure containing touchscreens on its walls, for studying cognition in freely moving macaques. To test flexible learning, two monkeys completed trials consisting of a regular sequence of object selections across four touchscreens. On each screen, the monkeys had to select by touching the sole correct object item ('target') among a set of four items, irrespective of their positions on the screen. Each item was the target on exactly one screen of the sequence, making correct performance conditioned on the spatiotemporal sequence rule across screens. Both monkeys successfully learned multiple 4-item sets (N=14 and 22 sets), totaling over 50 and 80 unique, conditional item-context memoranda, with no indication of capacity limits. The enclosure allowed freedom of movements leading up to and following the touchscreen interactions. To determine whether movement economy changed with learning, we reconstructed 3D position and movement dynamics using markerless tracking software and gyroscopic inertial measurements. Whereas general body positions remained consistent across repeated sequences, fine head movements varied as monkeys learned, within and across sequence sets, demonstrating learning set or "learning to learn". These results demonstrate monkeys' rapid, capacious, and flexible learning within an integrated, multisensory 3-D space. Furthermore, this approach enables the measurement of continuous behavior while ensuring precise experimental control and behavioral repetition of sequences over time. Overall, this approach harmonizes the design features that are needed for electrophysiological studies with tasks that showcase fully situated, flexible cognition.

Frontal-midline theta and posterior alpha oscillations index early processing of spatial representations during active navigation

Previous research has demonstrated that humans combine multiple sources of spatial information such as self-motion and landmark cues while navigating through an environment. However, it is unclear whether this involves comparing multiple representations obtained from different sources during navigation (parallel hypothesis) or building a representation first based on self-motion cues and then combining with landmarks later (serial hypothesis). We tested these two hypotheses (parallel vs serial) in an active navigation task using wireless mobile scalp EEG recordings. Participants walked through an immersive virtual hallway with or without conflicts between self-motion and landmarks (i.e., intersections) and pointed toward the starting position of the hallway. We employed the oscillatory signals recorded during mobile wireless scalp EEG as a means of identifying when participant representations based on self-motion versus landmark cues might have first emerged. We found that path segments, including intersections present early during navigation, were more strongly associated with later pointing error, regardless of when they appeared during encoding. We also found that there was sufficient information contained within the frontal-midline theta and posterior alpha oscillatory signals in the earliest segments of navigation involving intersections to decode condition (i.e., conflicting vs not conflicting). Together, these findings suggest that intersections play a pivotal role in the early development of spatial representations, suggesting that memory representations for the geometry of walked paths likely develop early during navigation, in support of the parallel hypothesis.

A neural code for time and space in the human brain

Time and space are primary dimensions of human experience. Separate lines of investigation have identified neural correlates of time and space, yet little is known about how these representations converge during self-guided experience. Here, 10 subjects with intracranially implanted microelectrodes play a timed, virtual navigation game featuring object search and retrieval tasks separated by fixed delays. Time cells and place cells activate in parallel during timed navigation intervals, whereas a separate time cell sequence spans inter-task delays. The prevalence, firing rates, and behavioral coding strengths of time cells and place cells are indistinguishable-yet time cells selectively remap between search and retrieval tasks, while place cell responses remain stable. Thus, the brain can represent time and space as overlapping but dissociable dimensions. Time cells and place cells may constitute a biological basis for the cognitive map of spatiotemporal context onto which memories are written.

Perceptual-Cognitive Integration for Goal-Directed Action in Naturalistic Environments

Real-world actions require one to simultaneously perceive, think, and act on the surrounding world, requiring the integration of (bottom-up) sensory information and (top-down) cognitive and motor signals. Studying these processes involves the intellectual challenge of cutting across traditional neuroscience silos, and the technical challenge of recording data in uncontrolled natural environments. However, recent advances in techniques, such as neuroimaging, virtual reality, and motion tracking, allow one to address these issues in naturalistic environments for both healthy participants and clinical populations. In this review, we survey six topics in which naturalistic approaches have advanced both our fundamental understanding of brain function and how neurologic deficits influence goal-directed, coordinated action in naturalistic environments. The first part conveys fundamental neuroscience mechanisms related to visuospatial coding for action, adaptive eye-hand coordination, and visuomotor integration for manual interception. The second part discusses applications of such knowledge to neurologic deficits, specifically, steering in the presence of cortical blindness, impact of stroke on visual-proprioceptive integration, and impact of visual search and working memory deficits. This translational approach-extending knowledge from lab to rehab-provides new insights into the complex interplay between perceptual, motor, and cognitive control in naturalistic tasks that are relevant for both basic and clinical research.

Dynamic neural representations of memory and space during human ambulatory navigation

Our ability to recall memories of personal experiences is an essential part of daily life. These episodic memories often involve movement through space and thus require continuous encoding of one's position relative to the surrounding environment. The medial temporal lobe (MTL) is thought to be critically involved, based on studies in freely moving rodents and stationary humans. However, it remains unclear if and how the MTL represents both space and memory especially during physical navigation, given challenges associated with deep brain recordings in humans during movement. We recorded intracranial electroencephalographic (iEEG) activity while participants completed an ambulatory spatial memory task within an immersive virtual reality environment. MTL theta activity was modulated by successful memory retrieval or spatial positions within the environment, depending on dynamically changing behavioral goals. Altogether, these results demonstrate how human MTL oscillations can represent both memory and space in a temporally flexible manner during freely moving navigation.

Consumer-Grade Electroencephalogram and Functional Near-Infrared Spectroscopy Neurofeedback Technologies for Mental Health and Wellbeing

Neurofeedback, utilizing an electroencephalogram (EEG) and/or a functional near-infrared spectroscopy (fNIRS) device, is a real-time measurement of brain activity directed toward controlling and optimizing brain function. This treatment has often been attributed to improvements in disorders such as ADHD, anxiety, depression, and epilepsy, among others. While there is evidence suggesting the efficacy of neurofeedback devices, the research is still inconclusive. The applicability of the measurements and parameters of consumer neurofeedback wearable devices has improved, but the literature on measurement techniques lacks rigorously controlled trials. This paper presents a survey and literary review of consumer neurofeedback devices and the direction toward clinical applications and diagnoses. Relevant devices are highlighted and compared for treatment parameters, structural composition, available software, and clinical appeal. Finally, a conclusion on future applications of these systems is discussed through the comparison of their advantages and drawbacks.

Consciousness: a neurosurgical perspective

Neurosurgeons are in a unique position to shed light on the neural basis for consciousness, not only by their clinical care of patients with compromised states of consciousness, but also by employing neurostimulation and neuronal recordings through intracranial electrodes in awake surgical patients, as well as during stages of sleep and anethesia. In this review, we discuss several aspects of consciousness, i.e., perception, memory, and willed actions, studied by electrical stimulation and single neuron recordings in the human brain. We demonstrate how specific neuronal activity underlie the emergence of concepts, memories, and intentions in human consciousness. We discuss the representation of specific conscious content by temporal lobe neurons and present the discovery of "concept cells" and the encoding and retrieval of memories by neurons in the medial temporal lobe. We review prefrontal and parietal neuronal activation that precedes conscious intentions to act. Taken together with other studies in the field, these findings suggest that specific conscious experience may arise from stochastic fluctuations of neuronal activity, reaching a dynamic threshold. Advances in brain recording and stimulation technology coupled with the rapid rise in artificial intelligence are likely to increase the amount and analysis capabilities of data obtained from the human brain, thereby improving the decoding of conscious and preconscious states and open new horizons for modulation of human cognitive functions such as memory and volition.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Moving beyond the lab: investigating empathy through the Empirical 5E approach

Empathy is a complex and multifaceted phenomenon that plays a crucial role in human social interactions. Recent developments in social neuroscience have provided valuable insights into the neural underpinnings and bodily mechanisms underlying empathy. This methodology often prioritizes precision, replicability, internal validity, and confound control. However, fully understanding the complexity of empathy seems unattainable by solely relying on artificial and controlled laboratory settings, while overlooking a comprehensive view of empathy through an ecological experimental approach. In this article, we propose articulating an integrative theoretical and methodological framework based on the 5E approach (the "E"s stand for embodied, embedded, enacted, emotional, and extended perspectives of empathy), highlighting the relevance of studying empathy as an active interaction between embodied agents, embedded in a shared real-world environment. In addition, we illustrate how a novel multimodal approach including mobile brain and body imaging (MoBi) combined with phenomenological methods, and the implementation of interactive paradigms in a natural context, are adequate procedures to study empathy from the 5E approach. In doing so, we present the Empirical 5E approach (E5E) as an integrative scientific framework to bridge brain/body and phenomenological attributes in an interbody interactive setting. Progressing toward an E5E approach can be crucial to understanding empathy in accordance with the complexity of how it is experienced in the real world.