The Neural Representation of Sequences: From Transition Probabilities to Algebraic Patterns and Linguistic Trees

Modeling rapid language learning by distilling Bayesian priors into artificial neural networks

Humans can learn languages from remarkably little experience. Developing computational models that explain this ability has been a major challenge in cognitive science. Existing approaches have been successful at explaining how humans generalize rapidly in controlled settings but are usually too restrictive to tractably handle naturalistic data. We show that learning from limited naturalistic data is possible with an approach that bridges the divide between two popular modeling traditions: Bayesian models and neural networks. This approach distills a Bayesian model's inductive biases-the factors that guide generalization-into a neural network that has flexible representations. Like a Bayesian model, the resulting system can learn formal linguistic patterns from limited data. Like a neural network, it can also learn aspects of English syntax from naturally-occurring sentences. Thus, this model provides a single system that can learn rapidly and can handle naturalistic data.

Hierarchical deviant processing in auditory cortex of awake mice

Detecting patterns, and noticing unexpected pattern changes, in the environment is a vital aspect of sensory processing. Adaptation and prediction error responses are two components of neural processing related to these tasks, and previous studies in the auditory system in rodents show that these two components are partially dissociable in terms of the topography and latency of neural responses to sensory deviants. However, many previous studies have focused on repetitions of single stimuli, such as pure tones, which have limited ecological validity. In this study, we tested whether the auditory cortical activity shows adaptation to repetition of more complex sound patterns (disyllabic pairs). Specifically, we compared neural responses to violations of sequences based on single stimulus probability only, against responses to more complex violations based on stimulus order. We employed an auditory oddball paradigm and monitored the auditory cortex (AC) activity of awake mice (N = 8) using wide-field calcium imaging. We found that cortical responses were sensitive both to single stimulus probabilities and to more global stimulus patterns, as mismatch signals were elicited following both substitution deviants and transposition deviants. Notably, higher order AC area elicited larger mismatch signaling to those deviants than primary AC, which suggests a hierarchical gradient of prediction error signaling in the auditory cortex. Such a hierarchical gradient was observed for late but not early peaks of calcium transients to deviants, suggesting that the late part of the deviant response may reflect prediction error signaling in response to more complex sensory pattern violations. SIGNIFICANCE STATEMENT: Detecting the unexpected change of patterns from the dynamic environment is vital for sensory processing, as it is essential to survival for humans and animals. Using wide-field calcium imaging, we investigated whether the auditory cortex of awake mice exhibits a hierarchical gradient of prediction error signaling and its sensitivity to violations of sequences based on stimulus features and stimulus order. We discovered the high-order auditory cortex elicited more significant mismatch signaling to those deviants than primary auditory cortex in substitution and transposition deviants. Calcium transients to deviants showed a hierarchical gradient for late but not for early peaks, indicating that the late part of the deviant response may reflect prediction error signaling in response to more complex sensory pattern violations.

A Computational Framework to Study Hierarchical Processing in Visual Narratives

Theories of visual narrative comprehension have advocated for a hierarchical grammar-based comprehension mechanism, but only limited work has investigated this hierarchy. Here, we provide a computational framework inspired by computational psycholinguistics to address hierarchy in visual narratives. The predictions generated by this framework were compared against behavior data to draw inferences about the hierarchical properties of visual narratives. A segmentation task-where participants ranked all possible segmental boundaries-demonstrated that participants' preferences were predicted by visual narrative grammar. Three kinds of models using surprisal theory-an Earley parser, a hidden Markov model (HMM), and an n-gram model-were then used to generate segmentation preferences for the same task. Earley parser's preferences were based on a hierarchical grammar with recursion properties, while the HMM and the n-grams used a flattened grammar for visual narrative comprehension. Given the differences in the mechanics of these models, contrasting their predictions against behavior data could provide crucial insights into understanding the underlying mechanisms of visual narrative comprehension. By investigating grammatical systems outside of language, this research provides new directions to explore the generic makeup of the cognitive structure of mental representations.

Auditory facilitation in deterministic versus stochastic worlds

The brain learns statistical regularities in sensory sequences, enhancing behavioral performance for predictable stimuli while impairing behavioral performance for unpredictable stimuli. While previous research has shown that violations of non-informative regularities hinder task performance, it remains unclear whether predictable but task-irrelevant structures can facilitate performance. In a tone duration discrimination task, we manipulated the task-irrelevant pitch dimension by varying transition probabilities (TP) between successive tone frequencies. Participants judged duration, while pitch sequences were either deterministic (a rule-governed pitch pattern, TP = 1) or stochastic (no discernible pitch pattern, TP = 1/number of pitch levels). The tone pitch was task-irrelevant and it did not predict duration. Results showed that reaction times (RTs) were significantly faster for deterministic sequences, suggesting that predictability in a task-irrelevant dimension still facilitates task performance. RTs were also faster in two-tone sequences compared to eight-tone sequences, likely due to reduced memory load. These findings suggest that statistical learning benefits extend beyond task-relevant dimensions, supporting a predictive coding framework in which the brain integrates predictable sensory input to optimize cognitive processing.

Direct brain recordings reveal implicit encoding of structure in random auditory streams

The brain excels at processing sensory input, even in rich or chaotic environments. Mounting evidence attributes this to sophisticated internal models of the environment that draw on statistical structures in the unfolding sensory input. Understanding how and where such modeling proceeds is a core question in statistical learning and predictive processing. In this context, we address the role of transitional probabilities as an implicit structure supporting the encoding of the temporal structure of a random auditory stream. Leveraging information-theoretical principles and the high spatiotemporal resolution of intracranial electroencephalography, we analyzed the trial-by-trial high-frequency activity representation of transitional probabilities. This unique approach enabled us to demonstrate how the brain automatically and continuously encodes structure in random stimuli and revealed the involvement of a network outside of the auditory system, including hippocampal, frontal, and temporal regions. Our work provides a comprehensive picture of the neural correlates of automatic encoding of implicit structure that can be the crucial substrate for the swift detection of patterns and unexpected events in the environment.

Relationships and representations of brain structures, connectivity, dynamics and functions

The review explores the complex interplay between brain structures and their associated functions, presenting a diversity of hierarchical models that enhances our understanding of these relationships. Central to this approach are structure-function flow diagrams, which offer a visual representation of how specific neuroanatomical structures are linked to their functional roles. These diagrams are instrumental in mapping the intricate connections between different brain regions, providing a clearer understanding of how functions emerge from the underlying neural architecture. The study details innovative attempts to develop new functional hierarchies that integrate structural and functional data. These efforts leverage recent advancements in neuroimaging techniques such as fMRI, EEG, MEG, and PET, as well as computational models that simulate neural dynamics. By combining these approaches, the study seeks to create a more refined and dynamic hierarchy that can accommodate the brain's complexity, including its capacity for plasticity and adaptation. A significant focus is placed on the overlap of structures and functions within the brain. The manuscript acknowledges that many brain regions are multifunctional, contributing to different cognitive and behavioral processes depending on the context. This overlap highlights the need for a flexible, non-linear hierarchy that can capture the brain's intricate functional landscape. Moreover, the study examines the interdependence of these functions, emphasizing how the loss or impairment of one function can impact others. Another crucial aspect discussed is the brain's ability to compensate for functional deficits following neurological diseases or injuries. The investigation explores how the brain reorganizes itself, often through the recruitment of alternative neural pathways or the enhancement of existing ones, to maintain functionality despite structural damage. This compensatory mechanism underscores the brain's remarkable plasticity, demonstrating its ability to adapt and reconfigure itself in response to injury, thereby ensuring the continuation of essential functions. In conclusion, the study presents a system of brain functions that integrates structural, functional, and dynamic perspectives. It offers a robust framework for understanding how the brain's complex network of structures supports a wide range of cognitive and behavioral functions, with significant implications for both basic neuroscience and clinical applications.

Chaotic recurrent neural networks for brain modelling: A review

Even in the absence of external stimuli, the brain is spontaneously active. Indeed, most cortical activity is internally generated by recurrence. Both theoretical and experimental studies suggest that chaotic dynamics characterize this spontaneous activity. While the precise function of brain chaotic activity is still puzzling, we know that chaos confers many advantages. From a computational perspective, chaos enhances the complexity of network dynamics. From a behavioural point of view, chaotic activity could generate the variability required for exploration. Furthermore, information storage and transfer are maximized at the critical border between order and chaos. Despite these benefits, many computational brain models avoid incorporating spontaneous chaotic activity due to the challenges it poses for learning algorithms. In recent years, however, multiple approaches have been proposed to overcome this limitation. As a result, many different algorithms have been developed, initially within the reservoir computing paradigm. Over time, the field has evolved to increase the biological plausibility and performance of the algorithms, sometimes going beyond the reservoir computing framework. In this review article, we examine the computational benefits of chaos and the unique properties of chaotic recurrent neural networks, with a particular focus on those typically utilized in reservoir computing. We also provide a detailed analysis of the algorithms designed to train chaotic RNNs, tracing their historical evolution and highlighting key milestones in their development. Finally, we explore the applications and limitations of chaotic RNNs for brain modelling, consider their potential broader impacts beyond neuroscience, and outline promising directions for future research.

Honey Bees Can Use Sequence Learning to Predict Rewards from a Prior Unrewarded Visual Stimulus

Learning to anticipate upcoming events can increase fitness by allowing animals to choose the best course of action, and many species can learn sequences of events and anticipate rewards. To date, most studies have focused on sequences over short time scales such as a few seconds. Whereas events separated by a few seconds are easily learned, events separated by longer delays are typically more difficult to learn. Here, we show that honey bees (Apis mellifera) can learn a sequence of two visually distinct food sources alternating in profitability every few minutes. Bees were challenged to learn that the rewarded pattern was the one that was non-rewarded on the prior visit. We show that bees can predict and choose the feeder that will be rewarding upon their next approach more frequently than predicted by chance, and they improve with experience, with 64% correct choices made in the second half of their visit sequence (N = 320 visits by 20 different bees). These results increase our understanding of honey bee visual sequential learning and further demonstrate the flexibility of foragers' learning strategies.

Unpacking the complexities of consciousness: Theories and reflections

As the field of consciousness science matures, the research agenda has expanded from an initial focus on the neural correlates of consciousness, to developing and testing theories of consciousness. Several theories have been put forward, each aiming to elucidate the relationship between consciousness and brain function. However, there is an ongoing, intense debate regarding whether these theories examine the same phenomenon. And, despite ongoing research efforts, it seems like the field has so far failed to converge around any single theory, and instead exhibits significant polarization. To advance this discussion, proponents of five prominent theories of consciousness-Global Neuronal Workspace Theory (GNWT), Higher-Order Theories (HOT), Integrated Information Theory (IIT), Recurrent Processing Theory (RPT), and Predictive Processing (PP)-engaged in a public debate in 2022, as part of the annual meeting of the Association for the Scientific Study of Consciousness (ASSC). They were invited to clarify the explananda of their theories, articulate the core mechanisms underpinning the corresponding explanations, and outline their foundational premises. This was followed by an open discussion that delved into the testability of these theories, potential evidence that could refute them, and areas of consensus and disagreement. Most importantly, the debate demonstrated that at this stage, there is more controversy than agreement between the theories, pertaining to the most basic questions of what consciousness is, how to identify conscious states, and what is required from any theory of consciousness. Addressing these core questions is crucial for advancing the field towards a deeper understanding and comparison of competing theories.

Sequence chunking through neural encoding of ordinal positions

Grouping sensory events into chunks is an efficient strategy to integrate information across long sequences such as speech, music, and complex movements. Although chunks can be constructed based on diverse cues (e.g., sensory features, statistical patterns, internal knowledge) recent studies have consistently demonstrated that the chunks constructed by different cues are all tracked by low-frequency neural dynamics. Here, I review evidence that chunking cues drive low-frequency activity in modality-dependent networks, which interact to generate chunk-tracking activity in broad brain areas. Functionally, this work suggests that a core computation underlying sequence chunking may assign each event its ordinal position within a chunk and that this computation is causally implemented by chunk-tracking neural activity during predictive sequence chunking.

Brain areas for reversible symbolic reference, a potential singularity of the human brain

The emergence of symbolic thinking has been proposed as a dominant cognitive criterion to distinguish humans from other primates during hominisation. Although the proper definition of a symbol has been the subject of much debate, one of its simplest features is bidirectional attachment: the content is accessible from the symbol, and vice versa. Behavioural observations scattered over the past four decades suggest that this criterion might not be met in non-human primates, as they fail to generalise an association learned in one temporal order (A to B) to the reverse order (B to A). Here, we designed an implicit fMRI test to investigate the neural mechanisms of arbitrary audio-visual and visual-visual pairing in monkeys and humans and probe their spontaneous reversibility. After learning a unidirectional association, humans showed surprise signals when this learned association was violated. Crucially, this effect occurred spontaneously in both learned and reversed directions, within an extended network of high-level brain areas, including, but also going beyond, the language network. In monkeys, by contrast, violations of association effects occurred solely in the learned direction and were largely confined to sensory areas. We propose that a human-specific brain network may have evolved the capacity for reversible symbolic reference.

The role of conscious attention in auditory statistical learning: Evidence from patients with impaired consciousness

The need for attention to enable statistical learning is debated. Testing individuals with impaired consciousness offers valuable insight, but very few studies have been conducted due to the difficulties inherent in such studies. Here, we examined the ability of patients with varying levels of disorders of consciousness (DOC) to extract statistical regularities from an artificial language composed of randomly concatenated pseudowords by measuring frequency tagging in EEG. The objectives were firstly, to assess the automaticity of the segmentation process and the correlations between the level of covert consciousness and statistical learning capacities; secondly, to identify potential new diagnostic indicators. We observed that segmentation abilities were preserved in some minimally conscious patients, suggesting that auditory statistical learning is an inherently automatic low-level process. Due to significant inter-individual variability, word segmentation might not be robust enough for clinical use. In contrast, temporal accuracy of auditory syllable responses correlates strongly with coma severity.

Extracting the fingerprints of sequences of random rhythmic auditory stimuli from electrophysiological data

It has been classically conjectured that the brain assigns probabilistic models to sequences of stimuli. An important issue associated with this conjecture is the identification of the classes of models used by the brain to perform this task. We address this issue by using a new clustering procedure for sets of electroencephalographic (EEG) data recorded from participants exposed to a sequence of auditory stimuli generated by a stochastic chain. This clustering procedure indicates that the brain uses the recurrent occurrences of a regular auditory stimulus in order to build a model.

Parallel mechanisms signal a hierarchy of sequence structure violations in the auditory cortex

The brain predicts regularities in sensory inputs at multiple complexity levels, with neuronal mechanisms that remain elusive. Here, we monitored auditory cortex activity during the local-global paradigm, a protocol nesting different regularity levels in sound sequences. We observed that mice encode local predictions based on stimulus occurrence and stimulus transition probabilities, because auditory responses are boosted upon prediction violation. This boosting was due to both short-term adaptation and an adaptation-independent surprise mechanism resisting anesthesia. In parallel, and only in wakefulness, VIP interneurons responded to the omission of the locally expected sound repeat at the sequence ending, thus providing a chunking signal potentially useful for establishing global sequence structure. When this global structure was violated, by either shortening the sequence or ending it with a locally expected but globally unexpected sound transition, activity slightly increased in VIP and PV neurons, respectively. Hence, distinct cellular mechanisms predict different regularity levels in sound sequences.

Decoding cortical chronotopy-Comparing the influence of different cortical organizational schemes

The brain's diverse intrinsic timescales enable us to perceive stimuli with varying temporal persistency. This study aimed to uncover the cortical organizational schemes underlying these variations, revealing the neural architecture for processing a wide range of sensory experiences. We collected resting-state fMRI, task-fMRI, and diffusion-weighted imaging data from 47 individuals. Based on this data, we extracted six organizational schemes: (1) the structural Rich Club (RC) architecture, shown to synchronize the connectome; (2) the structural Diverse Club architecture, as an alternative to the RC based on the network's module structure; (3) the functional uni-to-multimodal gradient, reflected in a wide range of structural and functional features; and (4) the spatial posterior/lateral-to-anterior/medial gradient, established for hierarchical levels of cognitive control. Also, we explored the effects of (5) structural graph theoretical measures of centrality and (6) cytoarchitectural differences. Using Bayesian model comparison, we contrasted the impact of these organizational schemes on (1) intrinsic resting-state timescales and (2) inter-subject correlation (ISC) from a task involving hierarchically nested digit sequences. As expected, resting-state timescales were slower in structural network hubs, hierarchically higher areas defined by the functional and spatial gradients, and thicker cortical regions. ISC analysis demonstrated hints for the engagement of higher cortical areas with more temporally persistent stimuli. Finally, the model comparison identified the uni-to-multimodal gradient as the best organizational scheme for explaining the chronotopy in both task and rest. Future research should explore the microarchitectural features that shape this gradient, elucidating how our brain adapts and evolves across different modes of processing.

Generative models for sequential dynamics in active inference

A central theme of theoretical neurobiology is that most of our cognitive operations require processing of discrete sequences of items. This processing in turn emerges from continuous neuronal dynamics. Notable examples are sequences of words during linguistic communication or sequences of locations during navigation. In this perspective, we address the problem of sequential brain processing from the perspective of active inference, which inherits from a Helmholtzian view of the predictive (Bayesian) brain. Underneath the active inference lies a generative model; namely, a probabilistic description of how (observable) consequences are generated by (unobservable) causes. We show that one can account for many aspects of sequential brain processing by assuming the brain entails a generative model of the sensed world that comprises central pattern generators, narratives, or well-defined sequences. We provide examples in the domains of motor control (e.g., handwriting), perception (e.g., birdsong recognition) through to planning and understanding (e.g., language). The solutions to these problems include the use of sequences of attracting points to direct complex movements-and the move from continuous representations of auditory speech signals to the discrete words that generate those signals.

Flexible Learning and Re-ordering of Context-dependent Object Sequences in Nonhuman Primates

Intelligent behavior involves mentally arranging learned information in novel ways and is particularly well developed in humans. While nonhuman primates (NHP) will learn to arrange new items in complex serial order and re-arrange neighboring items within that order, it has remained contentious whether they are capable to re-assign items more flexibly to non-adjacent positions. Such mental re-indexing is facilitated by inferring the latent temporal structure of experiences as opposed to learning serial chains of item-item associations. Here, we tested the ability for flexible mental re-indexing in rhesus macaques. Subjects learned to serially order five objects. A change of the background context indicated when the object order changed, probing the subjects to mentally re-arrange objects to non-adjacent positions of the learned serial structure. Subjects successfully used the context cue to pro-actively re-index items to new, non-adjacent positions. Mental re-indexing was more likely when the initial order had been learned at a higher level, improved with more experience of the re-indexing rule and correlated with working memory performance in a delayed match-to-sample task. These findings suggest that NHPs inferred the latent serial structure of experiences beyond a chaining of item-item associations and mentally rearrange items within that structure. The pattern of results indicates that NHPs form non-spatial cognitive maps of their experiences, which is a hallmark for flexible mental operations in many serially ordered behaviors including communication, counting or foraging.

Predictive learning shapes the representational geometry of the human brain

Predictive coding theories propose that the brain constantly updates internal models to minimize prediction errors and optimize sensory processing. However, the neural mechanisms that link prediction error encoding and optimization of sensory representations remain unclear. Here, we provide evidence how predictive learning shapes the representational geometry of the human brain. We recorded magnetoencephalography (MEG) in humans listening to acoustic sequences with different levels of regularity. We found that the brain aligns its representational geometry to match the statistical structure of the sensory inputs, by clustering temporally contiguous and predictable stimuli. Crucially, the magnitude of this representational shift correlates with the synergistic encoding of prediction errors in a network of high-level and sensory areas. Our findings suggest that, in response to the statistical regularities of the environment, large-scale neural interactions engaged in predictive processing modulate the representational content of sensory areas to enhance sensory processing.

Flexible control of sequence working memory in the macaque frontal cortex

To memorize a sequence, one must serially bind each item to its rank order. How the brain controls a given input to bind its associated order in sequence working memory (SWM) remains unexplored. Here, we investigated the neural representations underlying SWM control using electrophysiological recordings in the frontal cortex of macaque monkeys performing forward and backward SWM tasks. Separate and generalizable low-dimensional subspaces for sensory and memory information were found within the same frontal circuitry, and SWM control was reflected in these neural subspaces' organized dynamics. Each item at each rank was sequentially entered into a common sensory subspace and, depending on forward or backward task requirement, flexibly and timely sent into rank-selective SWM subspaces. Neural activity in these SWM subspaces faithfully predicted the recalled item and order information in single error trials. Thus, compositional neural population codes with well-orchestrated dynamics in frontal cortex support the flexible control of SWM.

Mental programming of spatial sequences in working memory in the macaque frontal cortex

How the brain mentally sorts a series of items in a specific order within working memory (WM) remains largely unknown. We investigated mental sorting using high-throughput electrophysiological recordings in the frontal cortex of macaque monkeys, who memorized and sorted spatial sequences in forward or backward orders according to visual cues. We discovered that items at each ordinal rank in WM were encoded in separate rank-WM subspaces and then, depending on cues, were maintained or reordered between the subspaces, accompanied by two extra temporary subspaces in two operation steps. Furthermore, the cue activity served as an indexical signal to trigger sorting processes. Thus, we propose a complete conceptual framework, where the neural landscape transitions in frontal neural states underlie the symbolic system for mental programming of sequence WM.

A geometrical solution underlies general neural principle for serial ordering

A general mathematical description of how the brain sequentially encodes knowledge remains elusive. We propose a linear solution for serial learning tasks, based on the concept of mixed selectivity in high-dimensional neural state spaces. In our framework, neural representations of items in a sequence are projected along a "geometric" mental line learned through classical conditioning. The model successfully solves serial position tasks and explains behaviors observed in humans and animals during transitive inference tasks amidst noisy sensory input and stochastic neural activity. This approach extends to recurrent neural networks performing motor decision tasks, where the same geometric mental line correlates with motor plans and modulates network activity according to the symbolic distance between items. Serial ordering is thus predicted to emerge as a monotonic mapping between sensory input and behavioral output, highlighting a possible pivotal role for motor-related associative cortices in transitive inference tasks.

Transcranial photobiomodulation on the left inferior frontal gyrus enhances Mandarin Chinese L1 and L2 complex sentence processing performances

This study investigated the causal enhancing effect of transcranial photobiomodulation (tPBM) over the left inferior frontal gyrus (LIFG) on syntactically complex Mandarin Chinese first language (L1) and second language (L2) sentence processing performances. Two (L1 and L2) groups of participants (thirty per group) were recruited to receive the double-blind, sham-controlled tPBM intervention via LIFG, followed by the sentence processing, the verbal working memory (WM), and the visual WM tasks. Results revealed a consistent pattern for both groups: (a) tPBM enhanced sentence processing performance but not verbal WM for linear processing of unstructured sequences and visual WM performances; (b) Participants with lower sentence processing performances under sham tPBM benefited more from active tPBM. Taken together, the current study substantiated that tPBM enhanced L1 and L2 sentence processing, and would serve as a promising and cost-effective noninvasive brain stimulation (NIBS) tool for future applications on upregulating the human language faculty.

Crossmodal Correspondence Mediates Crossmodal Transfer from Visual to Auditory Stimuli in Category Learning

This article investigated whether crossmodal correspondence, as a sensory translation phenomenon, can mediate crossmodal transfer from visual to auditory stimuli in category learning and whether multimodal category learning can influence the crossmodal correspondence between auditory and visual stimuli. Experiment 1 showed that the category knowledge acquired from elevation stimuli affected the categorization of pitch stimuli when there were robust crossmodal correspondence effects between elevation and size, indicating that crossmodal transfer occurred between elevation and pitch stimuli. Experiments 2 and 3 revealed that the size category knowledge could not be transferred to the categorization of pitches, but interestingly, size and pitch category learning determined the direction of the pitch-size correspondence, suggesting that the pitch-size correspondence was not stable and could be determined using multimodal category learning. Experiment 4 provided further evidence that there was no crossmodal transfer between size and pitch, due to the absence of a robust pitch-size correspondence. These results demonstrated that crossmodal transfer can occur between audio-visual stimuli with crossmodal correspondence, and multisensory category learning can change the corresponding relationship between audio-visual stimuli. These findings suggest that crossmodal transfer and crossmodal correspondence share similar abstract representations, which can be mediated by semantic content such as category labels.

Strong and weak alignment of large language models with human values

Minimizing negative impacts of Artificial Intelligent (AI) systems on human societies without human supervision requires them to be able to align with human values. However, most current work only addresses this issue from a technical point of view, e.g., improving current methods relying on reinforcement learning from human feedback, neglecting what it means and is required for alignment to occur. Here, we propose to distinguish strong and weak value alignment. Strong alignment requires cognitive abilities (either human-like or different from humans) such as understanding and reasoning about agents' intentions and their ability to causally produce desired effects. We argue that this is required for AI systems like large language models (LLMs) to be able to recognize situations presenting a risk that human values may be flouted. To illustrate this distinction, we present a series of prompts showing ChatGPT's, Gemini's and Copilot's failures to recognize some of these situations. We moreover analyze word embeddings to show that the nearest neighbors of some human values in LLMs differ from humans' semantic representations. We then propose a new thought experiment that we call "the Chinese room with a word transition dictionary", in extension of John Searle's famous proposal. We finally mention current promising research directions towards a weak alignment, which could produce statistically satisfying answers in a number of common situations, however so far without ensuring any truth value.

Crossmodal hierarchical predictive coding for audiovisual sequences in the human brain

Predictive coding theory suggests the brain anticipates sensory information using prior knowledge. While this theory has been extensively researched within individual sensory modalities, evidence for predictive processing across sensory modalities is limited. Here, we examine how crossmodal knowledge is represented and learned in the brain, by identifying the hierarchical networks underlying crossmodal predictions when information of one sensory modality leads to a prediction in another modality. We record electroencephalogram (EEG) during a crossmodal audiovisual local-global oddball paradigm, in which the predictability of transitions between tones and images are manipulated at both the stimulus and sequence levels. To dissect the complex predictive signals in our EEG data, we employed a model-fitting approach to untangle neural interactions across modalities and hierarchies. The model-fitting result demonstrates that audiovisual integration occurs at both the levels of individual stimulus interactions and multi-stimulus sequences. Furthermore, we identify the spatio-spectro-temporal signatures of prediction-error signals across hierarchies and modalities, and reveal that auditory and visual prediction errors are rapidly redirected to the central-parietal electrodes during learning through alpha-band interactions. Our study suggests a crossmodal predictive coding mechanism where unimodal predictions are processed by distributed brain networks to form crossmodal knowledge.

The developmental cognitive mechanism of learning algebraic rules from the dual-process theory perspective

Rule learning is an important ability that enables human beings to adapt to nature and develop civilizations. There have been many discussions on the mechanism and characteristics of algebraic rule learning, but there are still controversies due to the lack of theoretical guidance. Based on the dual-process theory, this study discussed the following arguments for algebraic rule learning across human and animal studies: whether algebraic rule learning is simply Type 1 processing, whether algebraic rule learning is a domain-general ability, whether algebraic rule learning is shared by humans and animals, and whether an algebraic rule is learned consciously. Moreover, we propose that algebraic rule learning is possibly a cognitive process that combines both Type 1 and Type 2 processing. Further exploration is required to establish the essence and neural basis of algebraic rule learning.

Shared structure facilitates working memory of multiple sequences

Daily experiences often involve the processing of multiple sequences, yet storing them challenges the limited capacity of working memory (WM). To achieve efficient memory storage, relational structures shared by sequences would be leveraged to reorganize and compress information. Here, participants memorized a sequence of items with different colors and spatial locations and later reproduced the full color and location sequences one after another. Crucially, we manipulated the consistency between location and color sequence trajectories. First, sequences with consistent trajectories demonstrate improved memory performance and a trajectory correlation between reproduced color and location sequences. Second, sequences with consistent trajectories show neural reactivation of common trajectories, and display spontaneous replay of color sequences when recalling locations. Finally, neural reactivation correlates with WM behavior. Our findings suggest that a shared common structure is leveraged for the storage of multiple sequences through compressed encoding and neural replay, together facilitating efficient information organization in WM.

Statistical learning shapes pain perception and prediction independently of external cues

The placebo and nocebo effects highlight the importance of expectations in modulating pain perception, but in everyday life we don't need an external source of information to form expectations about pain. The brain can learn to predict pain in a more fundamental way, simply by experiencing fluctuating, non-random streams of noxious inputs, and extracting their temporal regularities. This process is called statistical learning. Here, we address a key open question: does statistical learning modulate pain perception? We asked 27 participants to both rate and predict pain intensity levels in sequences of fluctuating heat pain. Using a computational approach, we show that probabilistic expectations and confidence were used to weigh pain perception and prediction. As such, this study goes beyond well-established conditioning paradigms associating non-pain cues with pain outcomes, and shows that statistical learning itself shapes pain experience. This finding opens a new path of research into the brain mechanisms of pain regulation, with relevance to chronic pain where it may be dysfunctional.

Is core knowledge a natural subdivision of infant cognition?

We examine Spelke's core knowledge taxonomy and test its boundaries. We ask whether Spelke's core knowledge is a distinct type of cognition in the sense that the cognitive processes it includes and excludes are biologically and mechanically coherent.

Phase-dependent word perception emerges from region-specific sensitivity to the statistics of language

Neural oscillations reflect fluctuations in excitability, which biases the percept of ambiguous sensory input. Why this bias occurs is still not fully understood. We hypothesized that neural populations representing likely events are more sensitive, and thereby become active on earlier oscillatory phases, when the ensemble itself is less excitable. Perception of ambiguous input presented during less-excitable phases should therefore be biased toward frequent or predictable stimuli that have lower activation thresholds. Here, we show such a frequency bias in spoken word recognition using psychophysics, magnetoencephalography (MEG), and computational modelling. With MEG, we found a double dissociation, where the phase of oscillations in the superior temporal gyrus and medial temporal gyrus biased word-identification behavior based on phoneme and lexical frequencies, respectively. This finding was reproduced in a computational model. These results demonstrate that oscillations provide a temporal ordering of neural activity based on the sensitivity of separable neural populations.

Neural representations of statistical and rule-based predictions in Gilles de la Tourette syndrome

Gilles de la Tourette syndrome (GTS) is a disorder characterised by motor and vocal tics, which may represent habitual actions as a result of enhanced learning of associations between stimuli and responses (S-R). In this study, we investigated how adults with GTS and healthy controls (HC) learn two types of regularities in a sequence: statistics (non-adjacent probabilities) and rules (predefined order). Participants completed a visuomotor sequence learning task while EEG was recorded. To understand the neurophysiological underpinnings of these regularities in GTS, multivariate pattern analyses on the temporally decomposed EEG signal as well as sLORETA source localisation method were conducted. We found that people with GTS showed superior statistical learning but comparable rule-based learning compared to HC participants. Adults with GTS had different neural representations for both statistics and rules than HC adults; specifically, adults with GTS maintained the regularity representations longer and had more overlap between them than HCs. Moreover, over different time scales, distinct fronto-parietal structures contribute to statistical learning in the GTS and HC groups. We propose that hyper-learning in GTS is a consequence of the altered sensitivity to encode complex statistics, which might lead to habitual actions.

Distributed Sensitivity to Syntax and Semantics throughout the Language Network

Human language is expressive because it is compositional: The meaning of a sentence (semantics) can be inferred from its structure (syntax). It is commonly believed that language syntax and semantics are processed by distinct brain regions. Here, we revisit this claim using precision fMRI methods to capture separation or overlap of function in the brains of individual participants. Contrary to prior claims, we find distributed sensitivity to both syntax and semantics throughout a broad frontotemporal brain network. Our results join a growing body of evidence for an integrated network for language in the human brain within which internal specialization is primarily a matter of degree rather than kind, in contrast with influential proposals that advocate distinct specialization of different brain areas for different types of linguistic functions.

Visual artificial grammar learning across 1 year in 7-year-olds and adults

Acquiring sequential information is of utmost importance, for example, for language acquisition in children. Yet, the long-term storage of statistical learning in children is poorly understood. To address this question, 27 7-year-olds and 28 young adults completed four sessions of visual sequence learning (Year 1). From this sample, 16 7-year-olds and 20 young adults participated in another four equivalent sessions after a 12-month-delay (Year 2). The first three sessions of each year used Stimulus Set 1, and the last session used Stimulus Set 2 to investigate transfer effects. Each session consisted of alternating learning and test phases in a modified artificial grammar learning task. In Year 1, 7-year-olds and adults learned the regularities and showed transfer to Stimulus Set 2. Both groups retained their final performance level over the 1-year period. In Year 2, children and adults continued to improve with Stimulus Set 1 but did not show additional transfer gains. Adults overall outperformed children, but transfer effects were indistinguishable between both groups. The current results suggest that long-term memory traces are formed from repeated sequence learning that can be used to generalize sequence rules to new visual input. However, the current study did not provide evidence for a childhood advantage in learning and remembering sequence rules.

Response times are affected by mispredictions in a stochastic game

Acting as a goalkeeper in a video-game, a participant is asked to predict the successive choices of the penalty taker. The sequence of choices of the penalty taker is generated by a stochastic chain with memory of variable length. It has been conjectured that the probability distribution of the response times is a function of the specific sequence of past choices governing the algorithm used by the penalty taker to make his choice at each step. We found empirical evidence that besides this dependence, the distribution of the response times depends also on the success or failure of the previous prediction made by the participant. Moreover, we found statistical evidence that this dependence propagates up to two steps forward after the prediction failure.

Long-Horizon Associative Learning Explains Human Sensitivity to Statistical and Network Structures in Auditory Sequences

Networks are a useful mathematical tool for capturing the complexity of the world. In a previous behavioral study, we showed that human adults were sensitive to the high-level network structure underlying auditory sequences, even when presented with incomplete information. Their performance was best explained by a mathematical model compatible with associative learning principles, based on the integration of the transition probabilities between adjacent and nonadjacent elements with a memory decay. In the present study, we explored the neural correlates of this hypothesis via magnetoencephalography (MEG). Participants (N = 23, 16 females) passively listened to sequences of tones organized in a sparse community network structure comprising two communities. An early difference (∼150 ms) was observed in the brain responses to tone transitions with similar transition probability but occurring either within or between communities. This result implies a rapid and automatic encoding of the sequence structure. Using time-resolved decoding, we estimated the duration and overlap of the representation of each tone. The decoding performance exhibited exponential decay, resulting in a significant overlap between the representations of successive tones. Based on this extended decay profile, we estimated a long-horizon associative learning novelty index for each transition and found a correlation of this measure with the MEG signal. Overall, our study sheds light on the neural mechanisms underlying human sensitivity to network structures and highlights the potential role of Hebbian-like mechanisms in supporting learning at various temporal scales.

Neurobiological Causal Models of Language Processing

The language faculty is physically realized in the neurobiological infrastructure of the human brain. Despite significant efforts, an integrated understanding of this system remains a formidable challenge. What is missing from most theoretical accounts is a specification of the neural mechanisms that implement language function. Computational models that have been put forward generally lack an explicit neurobiological foundation. We propose a neurobiologically informed causal modeling approach which offers a framework for how to bridge this gap. A neurobiological causal model is a mechanistic description of language processing that is grounded in, and constrained by, the characteristics of the neurobiological substrate. It intends to model the generators of language behavior at the level of implementational causality. We describe key features and neurobiological component parts from which causal models can be built and provide guidelines on how to implement them in model simulations. Then we outline how this approach can shed new light on the core computational machinery for language, the long-term storage of words in the mental lexicon and combinatorial processing in sentence comprehension. In contrast to cognitive theories of behavior, causal models are formulated in the "machine language" of neurobiology which is universal to human cognition. We argue that neurobiological causal modeling should be pursued in addition to existing approaches. Eventually, this approach will allow us to develop an explicit computational neurobiology of language.

Spontaneously emerging internal models of visual sequences combine abstract and event-specific information in the prefrontal cortex

When exposed to sensory sequences, do macaque monkeys spontaneously form abstract internal models that generalize to novel experiences? Here, we show that neuronal populations in macaque ventrolateral prefrontal cortex jointly encode visual sequences by separate codes for the specific pictures presented and for their abstract sequential structure. We recorded prefrontal neurons while macaque monkeys passively viewed visual sequences and sequence mismatches in the local-global paradigm. Even without any overt task or response requirements, prefrontal populations spontaneously form representations of sequence structure, serial order, and image identity within distinct but superimposed neuronal subspaces. Representations of sequence structure rapidly update following single exposure to a mismatch sequence, while distinct populations represent mismatches for sequences of different complexity. Finally, those representations generalize across sequences following the same repetition structure but comprising different images. These results suggest that prefrontal populations spontaneously encode rich internal models of visual sequences reflecting both content-specific and abstract information.

Tracking the neural codes for words and phrases during semantic composition, working-memory storage, and retrieval

The ability to compose successive words into a meaningful phrase is a characteristic feature of human cognition, yet its neural mechanisms remain incompletely understood. Here, we analyze the cortical mechanisms of semantic composition using magnetoencephalography (MEG) while participants read one-word, two-word, and five-word noun phrases and compared them with a subsequent image. Decoding of MEG signals revealed three processing stages. During phrase comprehension, the representation of individual words was sustained for a variable duration depending on phrasal context. During the delay period, the word code was replaced by a working-memory code whose activation increased with semantic complexity. Finally, the speed and accuracy of retrieval depended on semantic complexity and was faster for surface than for deep semantic properties. In conclusion, we propose that the brain initially encodes phrases using factorized dimensions for successive words but later compresses them in working memory and requires a period of decompression to access them.

Location-specific deviant responses to object sequences in macaque inferior temporal cortex

Many species learn temporal regularities in their visual environment, demonstrating visual statistical learning. In this study, we explored the sensitivity of macaque inferior temporal (IT) cortical neurons to transition probabilities of sequentially presented visual images, presented at different locations in the visual field. We exposed monkeys to sequences of two images, where the first image was presented either foveally or peripherally, and the second image was consistently presented foveally. Following several weeks of exposure, we recorded IT responses to assess differences between the exposed (Fixed) and new, Deviant sequences, where the identity of the first image in a sequence differed from the exposure phase. While enhanced responses to Deviant sequences were observed when both images of a pair were foveally presented during exposure, no such deviant responses were present when the first image was presented peripherally. This finding challenges the notion that mere exposure to image sequences always leads to deviant responses in macaque IT. The results highlight the complexity of the mechanisms underlying statistical learning in primates, particularly in the context of peripheral image presentations, emphasizing the need for further investigation into the origins of these responses in the IT cortex.

Computational insights on asymmetrical D1 and D2 receptor-mediated chunking: implications for OCD and Schizophrenia

Repetitive thoughts and motor programs including perseveration are bridge symptoms characteristic of obsessive compulsive disorder (OCD), schizophrenia and in the co-morbid overlap of these conditions. The above pathologies are sensitive to altered activation and kinetics of dopamine D 1 and D 2 receptors that differently influence sequence learning and recall. Recognizing start and stop elements of motor and cognitive behaviors has crucial importance. During chunking, frequent components of temporal strings are concatenated into single units. We extended a published computational model (Asabuki et al. 2018), where two populations of neurons are connected and simulated in a reservoir computing framework. These neural pools were adopted to represent D1 and D2 striatal neuronal populations. We investigated how specific neural and striatal circuit parameters can influence start/stop signaling and found that asymmetric intra-network connection probabilities, synaptic weights and differential time constants may contribute to signaling of start/stop elements within learned sequences. Asymmetric coupling between the striatal D 1 and D 2 neural populations was also demonstrated to be beneficial. Our modeling results predict that dynamical differences between the two dopaminergic striatal populations and the interaction between them may play complementary roles in chunk boundary signaling. Start and stop dichotomies can arise from the larger circuit dynamics as well, since neural and intra-striatal connections only partially support a clear division of labor.

Resource-rational account of sequential effects in human prediction

An abundant literature reports on 'sequential effects' observed when humans make predictions on the basis of stochastic sequences of stimuli. Such sequential effects represent departures from an optimal, Bayesian process. A prominent explanation posits that humans are adapted to changing environments, and erroneously assume non-stationarity of the environment, even if the latter is static. As a result, their predictions fluctuate over time. We propose a different explanation in which sub-optimal and fluctuating predictions result from cognitive constraints (or costs), under which humans however behave rationally. We devise a framework of costly inference, in which we develop two classes of models that differ by the nature of the constraints at play: in one case the precision of beliefs comes at a cost, resulting in an exponential forgetting of past observations, while in the other beliefs with high predictive power are favored. To compare model predictions to human behavior, we carry out a prediction task that uses binary random stimuli, with probabilities ranging from 0.05 to 0.95. Although in this task the environment is static and the Bayesian belief converges, subjects' predictions fluctuate and are biased toward the recent stimulus history. Both classes of models capture this 'attractive effect', but they depart in their characterization of higher-order effects. Only the precision-cost model reproduces a 'repulsive effect', observed in the data, in which predictions are biased away from stimuli presented in more distant trials. Our experimental results reveal systematic modulations in sequential effects, which our theoretical approach accounts for in terms of rationality under cognitive constraints.

Introducing MEG-MASC a high-quality magneto-encephalography dataset for evaluating natural speech processing

The "MEG-MASC" dataset provides a curated set of raw magnetoencephalography (MEG) recordings of 27 English speakers who listened to two hours of naturalistic stories. Each participant performed two identical sessions, involving listening to four fictional stories from the Manually Annotated Sub-Corpus (MASC) intermixed with random word lists and comprehension questions. We time-stamp the onset and offset of each word and phoneme in the metadata of the recording, and organize the dataset according to the 'Brain Imaging Data Structure' (BIDS). This data collection provides a suitable benchmark to large-scale encoding and decoding analyses of temporally-resolved brain responses to speech. We provide the Python code to replicate several validations analyses of the MEG evoked responses such as the temporal decoding of phonetic features and word frequency. All code and MEG, audio and text data are publicly available to keep with best practices in transparent and reproducible research.

A multimodal cortical network of sensory expectation violation revealed by fMRI

The brain is subjected to multi-modal sensory information in an environment governed by statistical dependencies. Mismatch responses (MMRs), classically recorded with EEG, have provided valuable insights into the brain's processing of regularities and the generation of corresponding sensory predictions. Only few studies allow for comparisons of MMRs across multiple modalities in a simultaneous sensory stream and their corresponding cross-modal context sensitivity remains unknown. Here, we used a tri-modal version of the roving stimulus paradigm in fMRI to elicit MMRs in the auditory, somatosensory and visual modality. Participants (N = 29) were simultaneously presented with sequences of low and high intensity stimuli in each of the three senses while actively observing the tri-modal input stream and occasionally reporting the intensity of the previous stimulus in a prompted modality. The sequences were based on a probabilistic model, defining transition probabilities such that, for each modality, stimuli were more likely to repeat (p = .825) than change (p = .175) and stimulus intensities were equiprobable (p = .5). Moreover, each transition was conditional on the configuration of the other two modalities comprising global (cross-modal) predictive properties of the sequences. We identified a shared mismatch network of modality general inferior frontal and temporo-parietal areas as well as sensory areas, where the connectivity (psychophysiological interaction) between these regions was modulated during mismatch processing. Further, we found deviant responses within the network to be modulated by local stimulus repetition, which suggests highly comparable processing of expectation violation across modalities. Moreover, hierarchically higher regions of the mismatch network in the temporo-parietal area around the intraparietal sulcus were identified to signal cross-modal expectation violation. With the consistency of MMRs across audition, somatosensation and vision, our study provides insights into a shared cortical network of uni- and multi-modal expectation violation in response to sequence regularities.

Brainstem fMRI signaling of surprise across different types of deviant stimuli

Detection of deviant stimuli is crucial to orient and adapt our behavior. Previous work shows that deviant stimuli elicit phasic activation of the locus coeruleus (LC), which releases noradrenaline and controls central arousal. However, it is unclear whether the detection of behaviorally relevant deviant stimuli selectively triggers LC responses or other neuromodulatory systems (dopamine, serotonin, and acetylcholine). We combine human functional MRI (fMRI) recordings optimized for brainstem imaging with pupillometry to perform a mapping of deviant-related responses in subcortical structures. Participants have to detect deviant items in a "local-global" paradigm that distinguishes between deviance based on the stimulus probability and the sequence structure. fMRI responses to deviant stimuli are distributed in many cortical areas. Both types of deviance elicit responses in the pupil, LC, and other neuromodulatory systems. Our results reveal that the detection of task-relevant deviant items recruits the same multiple subcortical systems across computationally different types of deviance.

Brain-imaging evidence for compression of binary sound sequences in human memory

According to the language-of-thought hypothesis, regular sequences are compressed in human memory using recursive loops akin to a mental program that predicts future items. We tested this theory by probing memory for 16-item sequences made of two sounds. We recorded brain activity with functional MRI and magneto-encephalography (MEG) while participants listened to a hierarchy of sequences of variable complexity, whose minimal description required transition probabilities, chunking, or nested structures. Occasional deviant sounds probed the participants' knowledge of the sequence. We predicted that task difficulty and brain activity would be proportional to the complexity derived from the minimal description length in our formal language. Furthermore, activity should increase with complexity for learned sequences, and decrease with complexity for deviants. These predictions were upheld in both fMRI and MEG, indicating that sequence predictions are highly dependent on sequence structure and become weaker and delayed as complexity increases. The proposed language recruited bilateral superior temporal, precentral, anterior intraparietal, and cerebellar cortices. These regions overlapped extensively with a localizer for mathematical calculation, and much less with spoken or written language processing. We propose that these areas collectively encode regular sequences as repetitions with variations and their recursive composition into nested structures.

Generalization of procedural motor sequence learning after a single practice trial

When humans begin learning new motor skills, they typically display early rapid performance improvements. It is not well understood how knowledge acquired during this early skill learning period generalizes to new, related skills. Here, we addressed this question by investigating factors influencing generalization of early learning from a skill A to a different, but related skill B. Early skill generalization was tested over four experiments (N = 2095). Subjects successively learned two related motor sequence skills (skills A and B) over different practice schedules. Skill A and B sequences shared ordinal (i.e., matching keypress locations), transitional (i.e., ordered keypress pairs), parsing rule (i.e., distinct sequence events like repeated keypresses that can be used as a breakpoint for segmenting the sequence into smaller units) structures, or possessed no structure similarities. Results showed generalization for shared parsing rule structure between skills A and B after only a single 10-second practice trial of skill A. Manipulating the initial practice exposure to skill A (1 to 12 trials) and inter-practice rest interval (0-30 s) between skills A and B had no impact on parsing rule structure generalization. Furthermore, this generalization was not explained by stronger sensorimotor mapping between individual keypress actions and their symbolic representations. In contrast, learning from skill A did not generalize to skill B during early learning when the sequences shared only ordinal or transitional structure features. These results document sequence structure that can be very rapidly generalized during initial learning to facilitate generalization of skill.

The effect of abstract inter-chunk relationships on serial-order control

Hierarchical control is often thought to dissect a complex task space into isolated subspaces in order to eliminate interference. Yet, there is also evidence from serial-order control tasks that our cognitive system can make use of abstract relationships between different parts (chunks) of a sequence. Past evidence in this regard was limited to situations with ordered stimuli (e.g., numbers or positions) that may have aided the detection of relationships and allowed gradual learning and hypothesis testing. Therefore, we used a modified task-span paradigm (with no ordered elements between tasks) in which participants performed memorized sequences of tasks that were encoded in terms of separate chunks of three tasks each. To allow examination of learning effects, each sequence was "cycled" through repeatedly. Importantly, we compared sequences whose chunks were governed by a common, abstract grammar with sequences whose chunks were governed by different grammars. Experiment 1 examined the effect of relationships between shared-element chunks (e.g., ABB-BAA vs. ABB-BAB), Experiment 2 and 3 between different-element chunks (e.g., ABA-CDC vs. ABA-CCD), and Experiment 4 examined second-order relationships (e.g., ABA-ABB--CDC-CDD vs. ABA-ABB--CDC-CCD). Robust evidence in favor of beneficial effects of abstract inter-chunk relationships was obtained across all four experiments. Importantly, these effects were at least as strong in initial cycles of performing a given sequence as during later cycles, suggesting that the cognitive system operates with an "expectation of abstract relationships," rather than benefitting from them through gradual learning. We discuss the implications of these results for models of hierarchical control.

Top-Down Number Reading: Language Affects the Visual Identification of Digit Strings

Reading numbers aloud involves visual processes that analyze the digit string and verbal processes that produce the number words. Cognitive models of number reading assume that information flows from the visual input to the verbal production processes-a feed-forward processing mode in which the verbal production depends on the visual input but not vice versa. Here, I show that information flows also in the opposite direction, from verbal production to the visual input processes. Participants read aloud briefly presented multi-digit strings in Hebrew, in which the order of words is congruent with the order of digits (21 = twenty-and-one), and in Arabic, in which the ones word precedes the tens word (one-and-twenty). The error-by-digit-position curve was affected by language: relative to Hebrew, in Arabic the error rate was slightly lower for the unit digit and slightly higher for the decade digit, indicating that in Arabic the unit digit was processed earlier and the decade digit later, in accord with the Arabic word order. This language-dependent processing order originated in the visual level and was not a verbal confound, because it persisted even when I controlled for the serial position of the decade/unit word in the verbal number by using numbers with 0 (two hundred three/two hundred thirty). I conclude that the visual analyzer's digit scanning order, decade-first or unit-first, is not fixed but affected by the language in which the number is produced-a top-down, verbal-to-visual information flow.

"What" and "when" predictions modulate auditory processing in a mutually congruent manner

Introduction

Extracting regularities from ongoing stimulus streams to form predictions is crucial for adaptive behavior. Such regularities exist in terms of the content of the stimuli and their timing, both of which are known to interactively modulate sensory processing. In real-world stimulus streams such as music, regularities can occur at multiple levels, both in terms of contents (e.g., predictions relating to individual notes vs. their more complex groups) and timing (e.g., pertaining to timing between intervals vs. the overall beat of a musical phrase). However, it is unknown whether the brain integrates predictions in a manner that is mutually congruent (e.g., if "beat" timing predictions selectively interact with "what" predictions falling on pulses which define the beat), and whether integrating predictions in different timing conditions relies on dissociable neural correlates.

Methods

To address these questions, our study manipulated "what" and "when" predictions at different levels - (local) interval-defining and (global) beat-defining - within the same stimulus stream, while neural activity was recorded using electroencephalogram (EEG) in participants (N = 20) performing a repetition detection task.

Results

Our results reveal that temporal predictions based on beat or interval timing modulated mismatch responses to violations of "what" predictions happening at the predicted time points, and that these modulations were shared between types of temporal predictions in terms of the spatiotemporal distribution of EEG signals. Effective connectivity analysis using dynamic causal modeling showed that the integration of "what" and "when" predictions selectively increased connectivity at relatively late cortical processing stages, between the superior temporal gyrus and the fronto-parietal network.

Discussion

Taken together, these results suggest that the brain integrates different predictions with a high degree of mutual congruence, but in a shared and distributed cortical network. This finding contrasts with recent studies indicating separable mechanisms for beat-based and memory-based predictive processing.

Combined low-frequency brain oscillatory activity and behavior predict future errors in human motor skill

Human skills are composed of sequences of individual actions performed with utmost precision. When occasional errors occur, they may have serious consequences, for example, when pilots are manually landing a plane. In such cases, the ability to predict an error before it occurs would clearly be advantageous. Here, we asked whether it is possible to predict future errors in a keyboard procedural human motor skill. We report that prolonged keypress transition times (KTTs), reflecting slower speed, and anomalous delta-band oscillatory activity in cingulate-entorhinal-precuneus brain regions precede upcoming errors in skill. Combined anomalous low-frequency activity and prolonged KTTs predicted up to 70% of future errors. Decoding strength (posterior probability of error) increased progressively approaching the errors. We conclude that it is possible to predict future individual errors in skill sequential performance.