Simultaneously active pre-attentive representations of local and global rules for sound sequences in the human brain

Markov chains as a proxy for the predictive memory representations underlying mismatch negativity

Events not conforming to a regularity inherent to a sequence of events elicit prediction error signals of the brain such as the Mismatch Negativity (MMN) and impair behavioral task performance. Events conforming to a regularity lead to attenuation of brain activity such as stimulus-specific adaptation (SSA) and behavioral benefits. Such findings are usually explained by theories stating that the information processing system predicts the forthcoming event of the sequence via detected sequential regularities. A mathematical model that is widely used to describe, to analyze and to generate event sequences are Markov chains: They contain a set of possible events and a set of probabilities for transitions between these events (transition matrix) that allow to predict the next event on the basis of the current event and the transition probabilities. The accuracy of such a prediction depends on the distribution of the transition probabilities. We argue that Markov chains also have useful applications when studying cognitive brain functions. The transition matrix can be regarded as a proxy for generative memory representations that the brain uses to predict the next event. We assume that detected regularities in a sequence of events correspond to (a subset of) the entries in the transition matrix. We apply this idea to the Mismatch Negativity (MMN) research and examine three types of MMN paradigms: classical oddball paradigms emphasizing sound probabilities, between-sound regularity paradigms manipulating transition probabilities between adjacent sounds, and action-sound coupling paradigms in which sounds are associated with actions and their intended effects. We show that the Markovian view on MMN yields theoretically relevant insights into the brain processes underlying MMN and stimulates experimental designs to study the brain's processing of event sequences.

Impact of feeding habits on the development of language-specific processing of phonemes in brain: An event-related potentials study

Introduction

Infancy is a stage characterized by multiple brain and cognitive changes. In a short time, infants must consolidate a new brain network and develop two important properties for speech comprehension: phonemic normalization and categorical perception. Recent studies have described diet as an essential factor in normal language development, reporting that breastfed infants show an earlier brain maturity and thus a faster cognitive development. Few studies have described a long-term effect of diet on phonological perception.

Methods

To explore that effect, we compared the event-related potentials (ERPs) collected during an oddball paradigm (frequent /pa/80%, deviant/ba/20%) of infants fed with breast milk (BF), cow-milk-based formula (MF), and soy-based formula (SF), which were assessed at 3, 6, 9, 12, and 24 months of age [Mean across all age groups: 127 BF infants, Mean (M) 39.6 gestation weeks; 121 MF infants, M = 39.16 gestation weeks; 116 SF infants, M = 39.16 gestation weeks].

Results

Behavioral differences between dietary groups in acoustic comprehension were observed at 24-months of age. The BF group displayed greater scores than the MF and SF groups. In phonological discrimination task, the ERPs analyses showed that SF group had an electrophysiological pattern associated with difficulties in phonological-stimulus awareness [mismatch negativity (MMN)-2 latency in frontal left regions of interest (ROI) and longer MMN-2 latency in temporal right ROI] and less brain maturity than BF and MF groups. The SF group displayed more right-lateralized brain recruitment in phonological processing at 12-months old.

Discussion

We conclude that using soy-based formula in a prolonged and frequent manner might trigger a language development different from that observed in the BF or MF groups. The soy-based formula's composition might affect frontal left-brain area development, which is a nodal brain region in phonological-stimuli awareness.

Syntactic chunking reveals a core syntactic representation of multi-digit numbers, which is generative and automatic

Representing the base-10 structure of numbers is a challenging cognitive ability, unique to humans, but it is yet unknown how precisely this is done. Here, we examined whether and how literate adults represent a number’s full syntactic structure. In 5 experiments, participants repeated number-word sequences and we systematically varied the order of words within each sequence. Repetition on grammatical sequences (e.g., two hundred ninety-seven) was better than on non-grammatical ones (hundred seven two ninety). We conclude that the participants represented the number’s full syntactic structure and used it to merge number words into chunks in short-term memory. Accuracy monotonously improved for sequences with increasingly longer grammatical segments, up to a limit of ~ 4 words per segment, irrespectively of the number of digits, and worsened thereafter. Namely, short chunks improved memorization, whereas oversized chunks disrupted memorization. This chunk size limit suggests that the chunks are not based on predefined structures, whose size limit is not expected to be so low, but are created ad hoc by a generative process, such as the hierarchical syntactic representation hypothesized in Michael McCloskey’s number-processing model. Chunking occurred even when it disrupted performance, as in the oversized chunks, and even when external cues for chunking were controlled for or were removed. We conclude that the above generative process operates automatically rather than voluntarily. To date, this is the most detailed account of the core representation of the syntactic structure of numbers—a critical aspect of numerical literacy and of the ability to read and write numbers.

Predictability-Based Source Segregation and Sensory Deviance Detection in Auditory Aging

When multiple sound sources are present at the same time, auditory perception is often challenged with disentangling the resulting mixture and focusing attention on the target source. It has been repeatedly demonstrated that background (distractor) sound sources are easier to ignore when their spectrotemporal signature is predictable. Prior evidence suggests that this ability to exploit predictability for foreground-background segregation degrades with age. On a theoretical level, this has been related with an impairment in elderly adults’ capabilities to detect certain types of sensory deviance in unattended sound sequences. Yet the link between those two capacities, deviance detection and predictability-based sound source segregation, has not been empirically demonstrated. Here we report on a combined behavioral-EEG study investigating the ability of elderly listeners (60–75 years of age) to use predictability as a cue for sound source segregation, as well as their sensory deviance detection capacities. Listeners performed a detection task on a target stream that can only be solved when a concurrent distractor stream is successfully ignored. We contrast two conditions whose distractor streams differ in their predictability. The ability to benefit from predictability was operationalized as performance difference between the two conditions. Results show that elderly listeners can use predictability for sound source segregation at group level, yet with a high degree of inter-individual variation in this ability. In a further, passive-listening control condition, we measured correlates of deviance detection in the event-related brain potential (ERP) elicited by occasional deviations from the same spectrotemporal pattern as used for the predictable distractor sequence during the behavioral task. ERP results confirmed neural signatures of deviance detection in terms of mismatch negativity (MMN) at group level. Correlation analyses at single-subject level provide no evidence for the hypothesis that deviance detection ability (measured by MMN amplitude) is related to the ability to benefit from predictability for sound source segregation. These results are discussed in the frameworks of sensory deviance detection and predictive coding.

Intention-based and sensory-based predictions

We inhabit a continuously changing world, where the ability to anticipate future states of the environment is critical for adaptation. Anticipation can be achieved by learning about the causal or temporal relationship between sensory events, as well as by learning to act on the environment to produce an intended effect. Together, sensory-based and intention-based predictions provide the flexibility needed to successfully adapt. Yet it is currently unknown whether the two sources of information are processed independently to form separate predictions, or are combined into a common prediction. To investigate this, we ran an experiment in which the final tone of two possible four-tone sequences could be predicted from the preceding tones in the sequence and/or from the participants' intention to trigger that final tone. This tone could be congruent with both sensory-based and intention-based predictions, incongruent with both, or congruent with one while incongruent with the other. Trials where predictions were incongruent with each other yielded similar prediction error responses irrespectively of the violated prediction, indicating that both predictions were formulated and coexisted simultaneously. The violation of intention-based predictions yielded late additional error responses, suggesting that those violations underwent further differential processing which the violations of sensory-based predictions did not receive.

The posterior auditory field is the chief generator of prediction error signals in the auditory cortex

The auditory cortex (AC) encompasses distinct fields subserving partly different aspects of sound processing. One essential function of the AC is the detection of unpredicted sounds, as revealed by differential neural activity to predictable and unpredictable sounds. According to the predictive coding framework, this effect can be explained by repetition suppression and/or prediction error signaling. The present study investigates functional specialization of the rat AC fields in repetition suppression and prediction error by combining a tone frequency oddball paradigm (involving high-probable standard and low-probable deviant tones) with two different control sequences (many-standards and cascade). Tones in the control sequences were comparable to deviant events with respect to neural adaptation but were not violating a regularity. Therefore, a difference in the neural activity between deviant and control tones indicates a prediction error effect, whereas a difference between control and standard tones indicates a repetition suppression effect. Single-unit recordings revealed by far the largest prediction error effects for the posterior auditory field, while the primary auditory cortex, the anterior auditory field, the ventral auditory field, and the suprarhinal auditory field were dominated by repetition suppression effects. Statistically significant repetition suppression effects occurred in all AC fields, whereas prediction error effects were less robust in the primary auditory cortex and the anterior auditory field. Results indicate that the non-lemniscal, posterior auditory field is more engaged in context-dependent processing underlying deviance-detection than the other AC fields, which are more sensitive to stimulus-dependent effects underlying differential degrees of neural adaptation.

Behavioral effect of mismatch negativity neurofeedback on foreign language learning

Listening is critical for foreign language learning. Listening difficulties can occur because of an inability to perceive or recognize sounds while listening to speech, whereas successful listening can boost understanding and improve speaking when learning a foreign language. Previous studies in our laboratory revealed that EEG-neurofeedback (NF) using mismatch negativity event-related brain potential successfully induced unconscious learning in terms of auditory discrimination of speech sounds. Here, we conducted a feasibility study with a small participant group (NF group and control group; six participants each) to examine the practical effects of mismatch negativity NF for improving the perception of speech sounds in a foreign language. Native Japanese speakers completed a task in which they learned to perceive and recognize spoken English words containing the consonants "l" or "r". Participants received neurofeedback training while not explicitly attending to auditory stimuli. The results revealed that NF training significantly improved the proportion of correct in discrimination and recognition trials, even though the training time for each word pair was reduced to 20% of the training time reported in our previous study. The learning effect was not affected by training with three pairs of words with different vowels. The current results indicate that NF resulted in long-term learning that persisted for at least 2 months.

The encoding of stochastic regularities is facilitated by action-effect predictions

Our brains continuously build and update predictive models of the world, sources of prediction being drawn for example from sensory regularities and/or our own actions. Yet, recent results in the auditory system indicate that stochastic regularities may not be easily encoded when a rare medium pitch deviant is presented between frequent high and low pitch standard sounds in random order, as reflected in the lack of sensory prediction error event-related potentials [i.e., mismatch negativity (MMN)]. We wanted to test the implication of the predictive coding theory that predictions based on higher-order generative models-here, based on action intention, are fed top-down in the hierarchy to sensory levels. Participants produced random sequences of high and low pitch sounds by button presses in two conditions: In a "specific" condition, one button produced high and the other low pitch sounds; in an "unspecific" condition, both buttons randomly produced high or low-pitch sounds. Rare medium pitch deviants elicited larger MMN and N2 responses in the "specific" compared to the "unspecific" condition, despite equal sound probabilities. These results thus demonstrate that action-effect predictions can boost stochastic regularity-based predictions and engage higher-order deviance detection processes, extending previous notions on the role of action predictions at sensory levels.

What exactly is missing here? The sensory processing of unpredictable omissions is modulated by the specificity of expected action-effects

We select our actions according to the desired outcomes; for instance, piano players press certain keys to generate specific musical notes. It is well-described that the omission of a predicted action-effect may elicit prediction error signals in the brain, but what happens in the case of simultaneous effector-specific (by contrast to effector-unspecific) predictions? To answer this question, we asked participants to press left and right keys to generate tones A and B; based on the action-effect association, the tones' identity was either predictable or unpredictable, while rarely, the expected input was omitted. Crucially, the data show that omissions following hand-specific associations reliably elicited a late omission N1 (oN1) component, by contrast to the hand-unspecific associations, where the late oN1 was rather weak. An additional condition where both key-presses generated a unique tone was implemented. Here, rare omissions of the expected tone generated both early and late oN1 responses, by contrast to the condition in which two simultaneous action-effect representations had to be maintained, where only late oN1 responses were elicited. Finally, omission P3 (oP3) responses were strongly elicited for all omission types without differences, indicating that a general expectation based on a tone presentation (rather than which tone), is likely indexed at this stage. The present results emphasize the top-down effects of action intention on the sensory processing of omissions, where unspecific (vs. specific) and multiple (vs. single) action-effect representations are associated with processing costs at the early sensory levels.

Encoding of deterministic and stochastic auditory rules in the human brain: The mismatch negativity mechanism does not reflect basic probability

Regularities in a sequence of sounds can be automatically encoded in a predictive model by the auditory system. When a sound deviates from the one predicted by the model, a mismatch negativity (MMN) is elicited, which is taken to reflect a prediction error at a particular level of the model hierarchy. Although there are many studies on deterministic regularities, only a few have investigated the brain's ability to encode non-deterministic regularities. We studied a simple stochastic regularity: two tone pitches (standards, each occurring on 45% of trials); this regularity was occasionally violated by another tone pitch (deviant, occurring on 10% of trials). We found MMN when the deviant's pitch was outside those of the standards, but not when it was between them. Importantly, when we alternated the occurrence of the same two standards, making them deterministic, the deviant elicited MMN, even when its pitch was between those of the standards. Thus, although the MMN system is extremely powerful in establishing even quite complex deterministic regularities, it fails with a simple stochastic regularity. We argue that the MMN system does not know basic probability.

Making Sense of Mismatch Negativity

Evoked potentials provide valuable insight into brain processes that are integral to our ability to interact effectively and efficiently in the world. The mismatch negativity (MMN) component of the evoked potential has proven highly informative on the ways in which sensitivity to regularity contributes to perception and cognition. This review offers a compendium of research on MMN with a view to scaffolding an appreciation for its use as a tool to explore the way regularities contribute to predictions about the sensory environment over many timescales. In compiling this work, interest in MMN as an index of sensory encoding and memory are addressed, as well as attention. Perspectives on the possible underlying computational processes are reviewed as well as recent observations that invite consideration of how MMN relates to how we learn, what we learn, and why.

Cognitive resources are distributed among the entire auditory landscape in auditory scene analysis

Although attention has been shown to enhance neural representations of selected inputs, the fate of unselected background sounds is still debated. The goal of the current study was to understand how processing resources are distributed among attended and unattended sounds during auditory scene analysis. We used a three-stream paradigm with four acoustic features uniquely defining each sound stream (frequency, envelope shape, spatial location, tone quality). We manipulated task load by having participants perform a difficult auditory task and an easy movie-viewing task with the same set of sounds in separate conditions. The mismatch negativity (MMN) component of event-related brain potentials (ERPs) was measured to evaluate sound processing in both conditions. We found no effect of task demands on unattended sound processing: MMNs were elicited by unattended deviants during both low- and high-load task conditions. A key factor of this result was the use of unique tone feature combinations to distinguish each of the three sound streams, strengthening the segregation of streams. In the auditory task, the P3b component demonstrates a two-stage process of target evaluation. Thus, these results, in conjunction with results of previous studies, suggest that stimulus-driven factors that strengthen stream segregation can free up processing capacity for higher-level analyses. The results illustrate the interactive nature of top-down and stimulus-driven processes in stream formation, supporting a distributive theory of attention that balances the strength of the bottom-up input with perceptual goals in analyzing the auditory scene.

Hierarchical modulation of auditory prediction error signaling is independent of attention

The auditory system is tuned to detect rhythmic regularities in the environment which can occur on different timescales. Event-related potentials such as mismatch negativity (MMN) and P3b are thought to index local and global deviance, respectively. However, it is not clear how these hierarchical levels interact and to what extent attention modulates this interaction. In this EEG study with 17 healthy young adults, we used a hierarchical oddball paradigm with local (sequence-level) and global (block-level) violations in attended and unattended conditions. Amplitude of N2 and P3b were analyzed in a 2*2*2 factorial model (local status, global status, attention condition). We found a significant interaction between the local and global status on the N2 amplitude, while there was no significant three-way interaction with attention, together demonstrating that lower-level prediction error is modulated by detection of higher-order regularity but expressed independently of attention. By contrast, higher-level prediction error, indexed by P3b, was sensitive to global regularity violations if the auditory stream was attended. The results demonstrate the capacity of our auditory perception to preattentively resolve conflicts between different levels of predictive hierarchy even across longer time intervals as indexed by MMN modulation, while P3b represents a different, attention-dependent system.

Single-Trial EEG-fMRI Reveals the Generation Process of the Mismatch Negativity

Although research on the mismatch negativity (MMN) has been ongoing for 40 years, the generation process of the MMN remains largely unknown. In this study, we used a single-trial electro-encephalography (EEG)-functional magnetic resonance imaging (fMRI) coupling method which can analyze neural activity with both high temporal and high spatial resolution and thus assess the generation process of the MMN. We elicited the MMN with an auditory oddball paradigm while recording simultaneous EEG and fMRI. We divided the MMN into five equal-durational phases. Utilizing the single-trial variability of the MMN, we analyzed the neural generators of the five phases, thereby determining the spatiotemporal generation process of the MMN. We found two distinct bottom-up prediction error propagations: first from the auditory cortex to the motor areas and then from the auditory cortex to the inferior frontal gyrus (IFG). Our results support the regularity-violation hypothesis of MMN generation.

Involuntary sensory enhancement of gain- and loss-associated tones: A general relevance principle

In a recent event-related potential (ERP) study (Folyi et al., 2016), we have demonstrated that sensory processing of task-irrelevant tones is enhanced when they were previously associated with positive or negative (by the means of monetary gains and losses, respectively) affective meaning relative to tones with neutral meaning, as indexed by the enhancement of the auditory N1-amplitude. In the present study, (1) in line with the hypothesis of affective counter-regulation, we investigated whether positive versus negative tones can receive differential attentional enhancement, depending on motivational context (Experiment 1); and (2) whether the early facilitation of positive and negative tones can operate strictly outside of the focus of voluntary attention (Experiment 2). In Experiment 1, we replicated the basic N1 valence effect, but found no moderation by motivational context. In Experiment 2, we found a small valence effect on the N1. By combining data from the three experiments (i.e., our previous experiment and the present ones; N = 72), we found a clear enhancement of N1-amplitudes for valenced tones without moderation by experiment. This pattern of results suggests comparable early attentional enhancement of valenced tones in general: (a) despite different level of concurrent task-relevant attentional and motivational demands in these experiments; and (b) without prioritizing one valence category over another, supporting our claim that the general relevance of the tones with high motivational value that governs early attentional facilitation.

Assessing the background decomposition of a complex auditory scene with event-related brain potentials

A listener who focusses on a sound source of interest must continuously integrate the sounds emitted by the attended source and ignore the sounds emitted by the remaining sources in the auditory scene. Little is known about how the ignored sound sources in the background are mentally represented after the source of interest has formed the perceptual foreground. This is due to a key methodological challenge: the background representation is by definition not overtly reportable. Here we developed a paradigm based on event-related brain potentials (ERPs) to assess the mental representation of background sounds. Participants listened to sequences of three repeatedly presented tones arranged in an ascending order (low, middle, high frequency). They were instructed to detect intensity deviants in one of the tones, creating the perceptual foreground. The remaining two background tones contained timing and location deviants. Those deviants were set up such that mismatch negativity (MMN) components would be elicited in distinct ways if the background was decomposed into two separate sound streams (background segregation) or if it was not further decomposed (background integration). Results provide MMN-based evidence for background segregation and integration in parallel. This suggests that mental representations of background integration and segregation can be concurrently available, and that collecting empirical evidence for only one of these background organization alternatives might lead to erroneous conclusions.

Neural Correlate of Transition Violation and Deviance Detection in the Songbird Auditory Forebrain

Deviants are stimuli that violate one's prediction about the incoming stimuli. Studying deviance detection helps us understand how nervous system learns temporal patterns between stimuli and forms prediction about the future. Detecting deviant stimuli is also critical for animals' survival in the natural environment filled with complex sounds and patterns. Using natural songbird vocalizations as stimuli, we recorded multi-unit and single-unit activity from the zebra finch auditory forebrain while presenting rare repeated stimuli after regular alternating stimuli (alternating oddball experiment) or rare deviant among multiple different common stimuli (context oddball experiment). The alternating oddball experiment showed that neurons were sensitive to rare repetitions in regular alternations. In the absence of expectation, repetition suppresses neural responses to the 2nd stimulus in the repetition. When repetition violates expectation, neural responses to the 2nd stimulus in the repetition were stronger than expected. The context oddball experiment showed that a stimulus elicits stronger neural responses when it is presented infrequently as a deviant among multiple common stimuli. As the acoustic differences between deviant and common stimuli increase, the response enhancement also increases. These results together showed that neural encoding of a stimulus depends not only on the acoustic features of the stimulus but also on the preceding stimuli and the transition patterns between them. These results also imply that the classical oddball effect may result from a combination of repetition suppression and deviance enhancement. Classification analyses showed that the difficulties in decoding the stimulus responsible for the neural responses differed for deviants in different experimental conditions. These findings suggest that learning transition patterns and detecting deviants in natural sequences may depend on a hierarchy of neural mechanisms, which may be involved in more complex forms of auditory processing that depend on the transition patterns between stimuli, such as speech processing.

Cascade and no-repetition rules are comparable controls for the auditory frequency mismatch negativity in oddball tasks

The mismatch negativity (MMN) has been widely studied with oddball tasks to index processing of unexpected auditory change. The MMN is computed as the difference of deviant minus standard and is used to capture the pattern violation by the deviant. However, this oddball MMN is confounded because the deviant differs physically from the standard and is presented less often. To improve measurement, the same tone as the deviant is presented in a separate condition. This control tone is equiprobable with other tones and is used to compute a corrected MMN (deviant minus control). Typically, the tones are in random order except that consecutive tones are not identical (no-repetition rule). In contrast, a recent study on frequency MMN presented tones in a regular up-and-down sequence (cascade rule). If the cascade rule is detected more easily than the no-repetition rule, there should be a lower risk of a confounding MMN within the cascade condition. However, in previous research, the cascade and no-repetition conditions differed not only in the regularity of the tone sequence but also in number of tones, frequency range, and proportion of tones. We controlled for these differences to isolate effects of regularity in the tone sequence. Results of our preregistered analyses provided moderate evidence (BF₀₁ >6) that the corrected MMN did not differ between cascade and no-repetition conditions. These findings imply that no-repetition and cascade rules are processed similarly and that the no-repetition condition provides an adequate control in frequency MMN.

Automatic change detection and spatial attention: a visual mismatch negativity study

Visual mismatch negativity (vMMN) is the electrophysiological correlate of automatic detection of unattended changes in the visual environment. However, vMMNs' relatedness to spatial attention has not been explicitly tested. Thus, the aim of the study was to investigate the effects of spatial attention on the vMMN event-related potential component. To this end, participants were instructed to fixate and attend to task-related stimuli. In an oddball sequence, offset stimuli were applied, i.e., from time-to time, the two sides of permanently presented objects disappeared. Distance between the task-related and unrelated events resulted in the typical finding of spatial attention; the amplitude of the N1 component was larger at the shorter distance between the two kinds of events. VMMN was elicited by the deviant vanishing parts, with no reliable effect of distance between the task-field and vMMN-related stimuli. In terms of the difference potentials, vMMN was followed by a positive posterior component in the 270-330 ms range. This positivity was much larger when the task-field was close to vMMN-related stimuli. The reappearance of the vanishing parts was also investigated. The reappearance of the whole objects after a deviant offset elicited vMMN but only when the task-field was close to the oddball sequence. We concluded that infrequently vanishing parts of objects are detected automatically. However, these deviant events initiate orientation only if the objects are close to the field of task-relevant events. Similarly, automatic registration of the rare but expected events are registered only in the visual field close to the focus of attention.

Evidence Integration in Natural Acoustic Textures during Active and Passive Listening

Many natural sounds can be well described on a statistical level, for example, wind, rain, or applause. Even though the spectro-temporal profile of these acoustic textures is highly dynamic, changes in their statistics are indicative of relevant changes in the environment. Here, we investigated the neural representation of change detection in natural textures in humans, and specifically addressed whether active task engagement is required for the neural representation of this change in statistics. Subjects listened to natural textures whose spectro-temporal statistics were modified at variable times by a variable amount. Subjects were instructed to either report the detection of changes (active) or to passively listen to the stimuli. A subset of passive subjects had performed the active task before (passive-aware vs passive-naive). Psychophysically, longer exposure to pre-change statistics was correlated with faster reaction times and better discrimination performance. EEG recordings revealed that the build-up rate and size of parieto-occipital (PO) potentials reflected change size and change time. Reduced effects were observed in the passive conditions. While P2 responses were comparable across conditions, slope and height of PO potentials scaled with task involvement. Neural source localization identified a parietal source as the main contributor of change-specific potentials, in addition to more limited contributions from auditory and frontal sources. In summary, the detection of statistical changes in natural acoustic textures is predominantly reflected in parietal locations both on the skull and source level. The scaling in magnitude across different levels of task involvement suggests a context-dependent degree of evidence integration.

Food words distract the hungry: Evidence of involuntary semantic processing of task-irrelevant but biologically-relevant unexpected auditory words

Rare changes in a stream of otherwise repeated task-irrelevant sounds break through selective attention and disrupt performance in an unrelated visual task by triggering shifts of attention to and from the deviant sound (deviance distraction). Evidence indicates that the involuntary orientation of attention to unexpected sounds is followed by their semantic processing. However, past demonstrations relied on tasks in which the meaning of the deviant sounds overlapped with features of the primary task. Here we examine whether such processing is observed when no such overlap is present but sounds carry some relevance to the participants’ biological need to eat when hungry. We report the results of an experiment in which hungry and satiated participants partook in a cross-modal oddball task in which they categorized visual digits (odd/even) while ignoring task-irrelevant sounds. On most trials the irrelevant sound was a sinewave tone (standard sound). On the remaining trials, deviant sounds consisted of spoken words related to food (food deviants) or control words (control deviants). Questionnaire data confirmed state (but not trait) differences between the two groups with respect to food craving, as well as a greater desire to eat the food corresponding to the food-related words in the hungry relative to the satiated participants. The results of the oddball task revealed that food deviants produced greater distraction (longer response times) than control deviants in hungry participants while the reverse effect was observed in satiated participants. This effect was observed in the first block of trials but disappeared thereafter, reflecting semantic saturation. Our results suggest that (1) the semantic content of deviant sounds is involuntarily processed even when sharing no feature with the primary task; and that (2) distraction by deviant sounds can be modulated by the participants’ biological needs.

Transitional Probabilities Are Prioritized over Stimulus/Pattern Probabilities in Auditory Deviance Detection: Memory Basis for Predictive Sound Processing

Representations encoding the probabilities of auditory events do not directly support predictive processing. In contrast, information about the probability with which a given sound follows another (transitional probability) allows predictions of upcoming sounds. We tested whether behavioral and cortical auditory deviance detection (the latter indexed by the mismatch negativity event-related potential) relies on probabilities of sound patterns or on transitional probabilities. We presented healthy adult volunteers with three types of rare tone-triplets among frequent standard triplets of high-low-high (H-L-H) or L-H-L pitch structure: proximity deviant (H-H-H/L-L-L), reversal deviant (L-H-L/H-L-H), and first-tone deviant (L-L-H/H-H-L). If deviance detection was based on pattern probability, reversal and first-tone deviants should be detected with similar latency because both differ from the standard at the first pattern position. If deviance detection was based on transitional probabilities, then reversal deviants should be the most difficult to detect because, unlike the other two deviants, they contain no low-probability pitch transitions. The data clearly showed that both behavioral and cortical auditory deviance detection uses transitional probabilities. Thus, the memory traces underlying cortical deviance detection may provide a link between stimulus probability-based change/novelty detectors operating at lower levels of the auditory system and higher auditory cognitive functions that involve predictive processing.

SIGNIFICANCE STATEMENT

Our research presents the first definite evidence for the auditory system prioritizing transitional probabilities over probabilities of individual sensory events. Forming representations for transitional probabilities paves the way for predictions of upcoming sounds. Several recent theories suggest that predictive processing provides the general basis of human perception, including important auditory functions, such as auditory scene analysis. Our results demonstrate that the memory traces underlying cortical deviance detection form a link between stimulus probability-based change/novelty detectors operating at lower levels of the auditory system and higher auditory cognitive functions that involve predictive processing.

Selective entrainment of brain oscillations drives auditory perceptual organization

Perceptual sound organization supports our ability to make sense of the complex acoustic environment, to understand speech and to enjoy music. However, the neuronal mechanisms underlying the subjective experience of perceiving univocal auditory patterns that can be listened to, despite hearing all sounds in a scene, are poorly understood. We hereby investigated the manner in which competing sound organizations are simultaneously represented by specific brain activity patterns and the way attention and task demands prime the internal model generating the current percept. Using a selective attention task on ambiguous auditory stimulation coupled with EEG recordings, we found that the phase of low-frequency oscillatory activity dynamically tracks multiple sound organizations concurrently. However, whereas the representation of ignored sound patterns is circumscribed to auditory regions, large-scale oscillatory entrainment in auditory, sensory-motor and executive-control network areas reflects the active perceptual organization, thereby giving rise to the subjective experience of a unitary percept.

How regularity representations of short sound patterns that are based on relative or absolute pitch information establish over time: An EEG study

The recognition of sound patterns in speech or music (e.g., a melody that is played in different keys) requires knowledge about pitch relations between successive sounds. We investigated the formation of regularity representations for sound patterns in an event-related potential (ERP) study. A pattern, which consisted of six concatenated 50 ms tone segments differing in fundamental frequency, was presented 1, 2, 3, 6, or 12 times and then replaced by another pattern by randomly changing the pitch of the tonal segments (roving standard paradigm). In an absolute repetition condition, patterns were repeated identically, whereas in a transposed condition, only the pitch relations of the tonal segments of the patterns were repeated, while the entire patterns were shifted up or down in pitch. During ERP measurement participants were not informed about the pattern repetition rule, but were instructed to discriminate rarely occurring targets of lower or higher sound intensity. EPRs for pattern changes (mismatch negativity, MMN; and P3a) and for pattern repetitions (repetition positivity, RP) revealed that the auditory system is able to rapidly extract regularities from unfamiliar complex sound patterns even when absolute pitch varies. Yet, enhanced RP and P3a amplitudes, and improved behavioral performance measured in a post-hoc test, in the absolute as compared with the transposed condition suggest that it is more difficult to encode patterns without absolute pitch information. This is explained by dissociable processing of standards and deviants as well as a back propagation mechanism to early sensory processing stages, which is effective after less repetitions of a standard stimulus for absolute pitch.

Stimulus Phase Locking of Cortical Oscillations for Rhythmic Tone Sequences in Rats

Humans can rapidly detect regular patterns (i.e., within few cycles) without any special attention to the acoustic environment. This suggests that human sensory systems are equipped with a powerful mechanism for automatically predicting forthcoming stimuli to detect regularity. It has recently been hypothesized that the neural basis of sensory predictions exists for not only what happens (predictive coding) but also when a particular stimulus occurs (predictive timing). Here, we hypothesize that the phases of neural oscillations are critical in predictive timing, and these oscillations are modulated in a band-specific manner when acoustic patterns become predictable, i.e., regular. A high-density microelectrode array (10 × 10 within 4 × 4 mm²) was used to characterize spatial patterns of band-specific oscillations when a random-tone sequence was switched to a regular-tone sequence. Increasing the regularity of the tone sequence enhanced phase locking in a band-specific manner, notwithstanding the type of the regular sound pattern. Gamma-band phase locking increased immediately after the transition from random to regular sequences, while beta-band phase locking gradually evolved with time after the transition. The amplitude of the tone-evoked response, in contrast, increased with frequency separation with respect to the prior tone, suggesting that the evoked-response amplitude encodes sequence information on a local scale, i.e., the local order of tones. The phase locking modulation spread widely over the auditory cortex, while the amplitude modulation was confined around the activation foci. Thus, our data suggest that oscillatory phase plays a more important role than amplitude in the neuronal detection of tone sequence regularity, which is closely related to predictive timing. Furthermore, band-specific contributions may support recent theories that gamma oscillations encode bottom-up prediction errors, whereas beta oscillations are involved in top-down prediction.

Initial uncertainty impacts statistical learning in sound sequence processing

This paper features two studies confirming a lasting impact of first learning on how subsequent experience is weighted in early relevance-filtering processes. In both studies participants were exposed to sequences of sound that contained a regular pattern on two different timescales. Regular patterning in sound is readily detected by the auditory system and used to form "prediction models" that define the most likely properties of sound to be encountered in a given context. The presence and strength of these prediction models is inferred from changes in automatically elicited components of auditory evoked potentials. Both studies employed sound sequences that contained both a local and longer-term pattern. The local pattern was defined by a regular repeating pure tone occasionally interrupted by a rare deviating tone (p=0.125) that was physically different (a 30msvs. 60ms duration difference in one condition and a 1000Hz vs. 1500Hz frequency difference in the other). The longer-term pattern was defined by the rate at which the two tones alternated probabilities (i.e., the tone that was first rare became common and the tone that was first common became rare). There was no task related to the tones and participants were asked to ignore them while focussing attention on a movie with subtitles. Auditory-evoked potentials revealed long lasting modulatory influences based on whether the tone was initially encountered as rare and unpredictable or common and predictable. The results are interpreted as evidence that probability (or indeed predictability) assigns a differential information-value to the two tones that in turn affects the extent to which prediction models are updated and imposed. These effects are exposed for both common and rare occurrences of the tones. The studies contribute to a body of work that reveals that probabilistic information is not faithfully represented in these early evoked potentials and instead exposes that predictability (or conversely uncertainty) may trigger value-based learning modulations even in task-irrelevant incidental learning.

Implicit learning of predictable sound sequences modulates human brain responses at different levels of the auditory hierarchy

Deviant stimuli, violating regularities in a sensory environment, elicit the mismatch negativity (MMN), largely described in the Event-Related Potential literature. While it is widely accepted that the MMN reflects more than basic change detection, a comprehensive description of mental processes modulating this response is still lacking. Within the framework of predictive coding, deviance processing is part of an inference process where prediction errors (the mismatch between incoming sensations and predictions established through experience) are minimized. In this view, the MMN is a measure of prediction error, which yields specific expectations regarding its modulations by various experimental factors. In particular, it predicts that the MMN should decrease as the occurrence of a deviance becomes more predictable. We conducted a passive oddball EEG study and manipulated the predictability of sound sequences by means of different temporal structures. Importantly, our design allows comparing mismatch responses elicited by predictable and unpredictable violations of a simple repetition rule and therefore departs from previous studies that investigate violations of different time-scale regularities. We observed a decrease of the MMN with predictability and interestingly, a similar effect at earlier latencies, within 70 ms after deviance onset. Following these pre-attentive responses, a reduced P3a was measured in the case of predictable deviants. We conclude that early and late deviance responses reflect prediction errors, triggering belief updating within the auditory hierarchy. Beside, in this passive study, such perceptual inference appears to be modulated by higher-level implicit learning of sequence statistical structures. Our findings argue for a hierarchical model of auditory processing where predictive coding enables implicit extraction of environmental regularities.

Auditory perceptual objects as generative models: Setting the stage for communication by sound

Communication by sounds requires that the communication channels (i.e. speech/speakers and other sound sources) had been established. This allows to separate concurrently active sound sources, to track their identity, to assess the type of message arriving from them, and to decide whether and when to react (e.g., reply to the message). We propose that these functions rely on a common generative model of the auditory environment. This model predicts upcoming sounds on the basis of representations describing temporal/sequential regularities. Predictions help to identify the continuation of the previously discovered sound sources to detect the emergence of new sources as well as changes in the behavior of the known ones. It produces auditory event representations which provide a full sensory description of the sounds, including their relation to the auditory context and the current goals of the organism. Event representations can be consciously perceived and serve as objects in various cognitive operations.

Absolute Pitch: Evidence for Early Cognitive Facilitation during Passive Listening as Revealed by Reduced P3a Amplitudes

Absolute pitch (AP) is the rare ability to identify or produce different pitches without using reference tones. At least two sequential processing stages are assumed to contribute to this phenomenon. The first recruits a pitch memory mechanism at an early stage of auditory processing, whereas the second is driven by a later cognitive mechanism (pitch labeling). Several investigations have used active tasks, but it is unclear how these two mechanisms contribute to AP during passive listening. The present work investigated the temporal dynamics of tone processing in AP and non-AP (NAP) participants by using EEG. We applied a passive oddball paradigm with between- and within-tone category manipulations and analyzed the MMN reflecting the early stage of auditory processing and the P3a response reflecting the later cognitive mechanism during the second processing stage. Results did not reveal between-group differences in MMN waveforms. By contrast, the P3a response was specifically associated with AP and sensitive to the processing of different pitch types. Specifically, AP participants exhibited smaller P3a amplitudes, especially in between-tone category conditions, and P3a responses correlated significantly with the age of commencement of musical training, suggesting an influence of early musical exposure on AP. Our results reinforce the current opinion that the representation of pitches at the processing level of the auditory-related cortex is comparable among AP and NAP participants, whereas the later processing stage is critical for AP. Results are interpreted as reflecting cognitive facilitation in AP participants, possibly driven by the availability of multiple codes for tones.

Neural correlates of auditory scene analysis and perception

The auditory system is designed to transform acoustic information from low-level sensory representations into perceptual representations. These perceptual representations are the computational result of the auditory system's ability to group and segregate spectral, spatial and temporal regularities in the acoustic environment into stable perceptual units (i.e., sounds or auditory objects). Current evidence suggests that the cortex-specifically, the ventral auditory pathway-is responsible for the computations most closely related to perceptual representations. Here, we discuss how the transformations along the ventral auditory pathway relate to auditory percepts, with special attention paid to the processing of vocalizations and categorization, and explore recent models of how these areas may carry out these computations.

The semantic aftermath of distraction by deviant sounds: Crosstalk interference is mediated by the predictability of semantic congruency

Rare changes in a stream of otherwise repeated task-irrelevant sounds break through selective attention and disrupt performance in an unrelated visual task. This deviance distraction effect emerges because deviant sounds violate the cognitive system's predictions. In this study we sought to examine whether predictability also mediate the so-called semantic effect whereby behavioral performance suffers from the clash between the involuntary semantic evaluation of irrelevant sounds and the voluntary processing of visual targets (e.g., when participants must categorize a right visual arrow following the presentation of the deviant sound "left"). By manipulating the conditional probabilities of the congruent and incongruent deviant sounds in a left/right arrow categorization task, we elicited implicit predictions about the upcoming target and related response. We observed a linear increase of the semantic effect with the proportion of congruent deviant trials (i.e., as deviant sounds increasingly predicted congruent targets). We conclude that deviant sounds affect response times based on a combination of crosstalk interference and two types of prediction violations: stimulus violations (violations of predictions regarding the identity of upcoming irrelevant sounds) and semantic violations (violations of predictions regarding the target afforded by deviant sounds). We report a three-parameter model that captures all key features of the observed RTs. Overall, our results fit with the view that the brain builds forward models of the environment in order to optimize cognitive processing and that control of one's attention and actions is called upon when predictions are violated. This article is part of a Special Issue entitled SI: Prediction and Attention.

Separate and concurrent symbolic predictions of sound features are processed differently

The studies investigated the impact of predictive visual information about the pitch and location of a forthcoming sound on the sound processing. In Symbol-to-Sound matching paradigms, symbols induced predictions of particular sounds. The brain's error signals (IR and N2b components of the event-related potential) were measured in response to occasional violations of the prediction, i.e., when a sound was incongruent to the corresponding symbol. IR and N2b index the detection of prediction violations at different levels, IR at a sensory and N2b at a cognitive level. Participants evaluated the congruency between prediction and actual sound by button press. When the prediction referred to only the pitch or only the location feature (Experiment 1), the violation of each feature elicited IR and N2b. The IRs to pitch and location violations revealed differences in the in time course and topography, suggesting that they were generated in feature-specific sensory areas. When the prediction referred to both features concurrently (Experiment 2), that is, the symbol predicted the sound's pitch and location, either one or both predictions were violated. Unexpectedly, no significant effects in the IR range were obtained. However, N2b was elicited in response to all violations. N2b in response to concurrent violations of pitch and location had a shorter latency. We conclude that associative predictions can be established by arbitrary rule-based symbols and for different sound features, and that concurrent violations are processed in parallel. In complex situations as in Experiment 2, capacity limitations appear to affect processing in a hierarchical manner. While predictions were presumably not reliably established at sensory levels (absence of IR), they were established at more cognitive levels, where sounds are represented categorially (presence of N2b).

Effects of task-switching on neural representations of ambiguous sound input

The ability to perceive discrete sound streams in the presence of competing sound sources relies on multiple mechanisms that organize the mixture of the auditory input entering the ears. Many studies have focused on mechanisms that contribute to integrating sounds that belong together into one perceptual stream (integration) and segregating those that come from different sound sources (segregation). However, little is known about mechanisms that allow us to perceive individual sound sources within a dynamically changing auditory scene, when the input may be ambiguous, and heard as either integrated or segregated. This study tested the question of whether focusing on one of two possible sound organizations suppressed representation of the alternative organization. We presented listeners with ambiguous input and cued them to switch between tasks that used either the integrated or the segregated percept. Electrophysiological measures indicated which organization was currently maintained in memory. If mutual exclusivity at the neural level was the rule, attention to one of two possible organizations would preclude neural representation of the other. However, significant MMNs were elicited to both the target organization and the unattended, alternative organization, along with the target-related P3b component elicited only to the designated target organization. Results thus indicate that both organizations (integrated and segregated) were simultaneously maintained in memory regardless of which task was performed. Focusing attention to one aspect of the sounds did not abolish the alternative, unattended organization when the stimulus input was ambiguous. In noisy environments, such as walking on a city street, rapid and flexible adaptive processes are needed to help facilitate rapid switching to different sound sources in the environment. Having multiple representations available to the attentive system would allow for such flexibility, needed in everyday situations to maintain stable auditory percepts, and to allow rapid scanning of interesting events in a busy environment.

Encoding of nested levels of acoustic regularity in hierarchically organized areas of the human auditory cortex

Our auditory system is able to encode acoustic regularity of growing levels of complexity to model and predict incoming events. Recent evidence suggests that early indices of deviance detection in the time range of the middle-latency responses (MLR) precede the mismatch negativity (MMN), a well-established error response associated with deviance detection. While studies suggest that only the MMN, but not early deviance-related MLR, underlie complex regularity levels, it is not clear whether these two mechanisms interplay during scene analysis by encoding nested levels of acoustic regularity, and whether neuronal sources underlying local and global deviations are hierarchically organized. We registered magnetoencephalographic evoked fields to rapidly presented four-tone local sequences containing a frequency change. Temporally integrated local events, in turn, defined global regularities, which were infrequently violated by a tone repetition. A global magnetic mismatch negativity (MMNm) was obtained at 140-220 ms when breaking the global regularity, but no deviance-related effects were shown in early latencies. Conversely, Nbm (45-55 ms) and Pbm (60-75 ms) deflections of the MLR, and an earlier MMNm response at 120-160 ms, responded to local violations. Distinct neuronal generators in the auditory cortex underlay the processing of local and global regularity violations, suggesting that nested levels of complexity of auditory object representations are represented in separated cortical areas. Our results suggest that the different processing stages and anatomical areas involved in the encoding of auditory representations, and the subsequent detection of its violations, are hierarchically organized in the human auditory cortex.

Behavioral correlates of auditory streaming in rhesus macaques

Perceptual representations of auditory stimuli (i.e., sounds) are derived from the auditory system's ability to segregate and group the spectral, temporal, and spatial features of auditory stimuli-a process called "auditory scene analysis". Psychophysical studies have identified several of the principles and mechanisms that underlie a listener's ability to segregate and group acoustic stimuli. One important psychophysical task that has illuminated many of these principles and mechanisms is the "streaming" task. Despite the wide use of this task to study psychophysical mechanisms of human audition, no studies have explicitly tested the streaming abilities of non-human animals using the standard methodologies employed in human-audition studies. Here, we trained rhesus macaques to participate in the streaming task using methodologies and controls similar to those presented in previous human studies. Overall, we found that the monkeys' behavioral reports were qualitatively consistent with those of human listeners, thus suggesting that this task may be a valuable tool for future neurophysiological studies.

Stable individual characteristics in the perception of multiple embedded patterns in multistable auditory stimuli

The ability of the auditory system to parse complex scenes into component objects in order to extract information from the environment is very robust, yet the processing principles underlying this ability are still not well understood. This study was designed to investigate the proposal that the auditory system constructs multiple interpretations of the acoustic scene in parallel, based on the finding that when listening to a long repetitive sequence listeners report switching between different perceptual organizations. Using the “ABA-” auditory streaming paradigm we trained listeners until they could reliably recognize all possible embedded patterns of length four which could in principle be extracted from the sequence, and in a series of test sessions investigated their spontaneous reports of those patterns. With the training allowing them to identify and mark a wider variety of possible patterns, participants spontaneously reported many more patterns than the ones traditionally assumed (Integrated vs. Segregated). Despite receiving consistent training and despite the apparent randomness of perceptual switching, we found individual switching patterns were idiosyncratic; i.e., the perceptual switching patterns of each participant were more similar to their own switching patterns in different sessions than to those of other participants. These individual differences were found to be preserved even between test sessions held a year after the initial experiment. Our results support the idea that the auditory system attempts to extract an exhaustive set of embedded patterns which can be used to generate expectations of future events and which by competing for dominance give rise to (changing) perceptual awareness, with the characteristics of pattern discovery and perceptual competition having a strong idiosyncratic component. Perceptual multistability thus provides a means for characterizing both general mechanisms and individual differences in human perception.

Attention effects on auditory scene analysis: insights from event-related brain potentials

Sounds emitted by different sources arrive at our ears as a mixture that must be disentangled before meaningful information can be retrieved. It is still a matter of debate whether this decomposition happens automatically or requires the listener's attention. These opposite positions partly stem from different methodological approaches to the problem. We propose an integrative approach that combines the logic of previous measurements targeting either auditory stream segregation (interpreting a mixture as coming from two separate sources) or integration (interpreting a mixture as originating from only one source). By means of combined behavioral and event-related potential (ERP) measures, our paradigm has the potential to measure stream segregation and integration at the same time, providing the opportunity to obtain positive evidence of either one. This reduces the reliance on zero findings (i.e., the occurrence of stream integration in a given condition can be demonstrated directly, rather than indirectly based on the absence of empirical evidence for stream segregation, and vice versa). With this two-way approach, we systematically manipulate attention devoted to the auditory stimuli (by varying their task relevance) and to their underlying structure (by delivering perceptual tasks that require segregated or integrated percepts). ERP results based on the mismatch negativity (MMN) show no evidence for a modulation of stream integration by attention, while stream segregation results were less clear due to overlapping attention-related components in the MMN latency range. We suggest future studies combining the proposed two-way approach with some improvements in the ERP measurement of sequential stream segregation.

Event-related potential measures of a violation of an expected increase and decrease in intensity

Unexpected physical increases in the intensity of a frequently occurring “standard” auditory stimulus are experienced as obtrusive. This could either be because of a physical change, the increase in intensity of the “deviant” stimulus, or a psychological change, the violation of the expectancy for the occurrence of the lower intensity standard stimulus. Two experiments were run in which event-related potentials (ERPs) were recorded to determine whether “psychological” increments (violation of an expectancy for a lower intensity) would be processed differently than psychological decrements (violation of an expectancy for a higher intensity). Event-related potentials (ERPs) were recorded while subjects were presented with auditory tones that alternated between low and high intensity. The subjects ignored the auditory stimuli while watching a video. Deviants were created by repeating the same stimulus. In the first experiment, pairs of stimuli alternating in intensity, were presented in separate increment (H-L…H-L…H-H…H-L, in which H = 80 dB SPL and L = 60 dB SPL) and decrement conditions (L-H…L-H…L-L… L-H, in which H = 90 dB SPL and L = 80 dB SPL). The paradigm employed in the second experiment consisted of an alternating intensity pattern (H-L-H-L-H-H-H-L) or (H-L-H-L-L-L-H-L). Importantly, the stimulus prior to the deviant (the standard) and the actual deviants in both increment and decrement conditions in both experiments were physically identical (80 dB SPL tones). The repetition of the lower intensity tone therefore acted as a psychological rather than a physical decrement (a higher intensity tone was expected) while the repetition of the higher intensity tone acted as a psychological increment (a lower intensity tone was expected). The psychological increments in both experiments elicited a larger amplitude mismatch negativity (MMN) than the decrements. Thus, regardless of whether an acoustic change signals a physical increase in intensity or violates an expected decrease in intensity, a large MMN will be elicited.

Modelling the emergence and dynamics of perceptual organisation in auditory streaming

Many sound sources can only be recognised from the pattern of sounds they emit, and not from the individual sound events that make up their emission sequences. Auditory scene analysis addresses the difficult task of interpreting the sound world in terms of an unknown number of discrete sound sources (causes) with possibly overlapping signals, and therefore of associating each event with the appropriate source. There are potentially many different ways in which incoming events can be assigned to different causes, which means that the auditory system has to choose between them. This problem has been studied for many years using the auditory streaming paradigm, and recently it has become apparent that instead of making one fixed perceptual decision, given sufficient time, auditory perception switches back and forth between the alternatives—a phenomenon known as perceptual bi- or multi-stability. We propose a new model of auditory scene analysis at the core of which is a process that seeks to discover predictable patterns in the ongoing sound sequence. Representations of predictable fragments are created on the fly, and are maintained, strengthened or weakened on the basis of their predictive success, and conflict with other representations. Auditory perceptual organisation emerges spontaneously from the nature of the competition between these representations. We present detailed comparisons between the model simulations and data from an auditory streaming experiment, and show that the model accounts for many important findings, including: the emergence of, and switching between, alternative organisations; the influence of stimulus parameters on perceptual dominance, switching rate and perceptual phase durations; and the build-up of auditory streaming. The principal contribution of the model is to show that a two-stage process of pattern discovery and competition between incompatible patterns can account for both the contents (perceptual organisations) and the dynamics of human perception in auditory streaming.

The sound waves produced by objects in the environment mix together before reaching the ears. Before we can make sense of an auditory scene, our brains must solve the puzzle of how to disassemble the sound waveform into groupings that correspond to the original source signals. How is this feat accomplished? We propose that the auditory system continually scans the structure of incoming signals in search of clues to indicate which pieces belong together. For instance, sound events may belong together if they have similar features, or form part of a clear temporal pattern. However this process is complicated by lack of knowledge of future events and the many possible ways in which even a simple sound sequence can be decomposed. The biological solution is multistability: one possible interpretation of a sound is perceived initially, which then gives way to another interpretation, and so on. We propose a model of auditory multistability, in which fragmental descriptions of the signal compete and cooperate to explain the sound scene. We demonstrate, using simplified experimental stimuli, that the model can account for both the contents (perceptual organisations) and the dynamics of human perception in auditory streaming.

The human brain maintains contradictory and redundant auditory sensory predictions

Computational and experimental research has revealed that auditory sensory predictions are derived from regularities of the current environment by using internal generative models. However, so far, what has not been addressed is how the auditory system handles situations giving rise to redundant or even contradictory predictions derived from different sources of information. To this end, we measured error signals in the event-related brain potentials (ERPs) in response to violations of auditory predictions. Sounds could be predicted on the basis of overall probability, i.e., one sound was presented frequently and another sound rarely. Furthermore, each sound was predicted by an informative visual cue. Participants’ task was to use the cue and to discriminate the two sounds as fast as possible. Violations of the probability based prediction (i.e., a rare sound) as well as violations of the visual-auditory prediction (i.e., an incongruent sound) elicited error signals in the ERPs (Mismatch Negativity [MMN] and Incongruency Response [IR]). Particular error signals were observed even in case the overall probability and the visual symbol predicted different sounds. That is, the auditory system concurrently maintains and tests contradictory predictions. Moreover, if the same sound was predicted, we observed an additive error signal (scalp potential and primary current density) equaling the sum of the specific error signals. Thus, the auditory system maintains and tolerates functionally independently represented redundant and contradictory predictions. We argue that the auditory system exploits all currently active regularities in order to optimally prepare for future events.

Not so primitive: context-sensitive meta-learning about unattended sound sequences

Mismatch negativity (MMN), an evoked response potential elicited when a “deviant” sound violates a regularity in the auditory environment, is integral to auditory scene processing and has been used to demonstrate “primitive intelligence” in auditory short-term memory. Using a new multiple-context and -timescale protocol we show that MMN magnitude displays a context-sensitive modulation depending on changes in the probability of a deviant at multiple temporal scales. We demonstrate a primacy bias causing asymmetric evidence-based modulation of predictions about the environment, and we demonstrate that learning how to learn about deviant probability (meta-learning) induces context-sensitive variation in the accessibility of predictive long-term memory representations that underpin the MMN. The existence of the bias and meta-learning are consistent with automatic attributions of behavioral salience governing relevance-filtering processes operating outside of awareness.