Assessing the information content of ERP signals in schizophrenia using multivariate decoding methods

The Timing of Gaze Direction Perception: ERP Decoding and Task Modulation

Distinguishing the direction of another person's eye gaze is extremely important in everyday social interaction, as it provides critical information about people's attention and, therefore, intentions. The temporal dynamics of gaze processing have been investigated using event-related potentials (ERPs) recorded with electroencephalography (EEG). However, the moment at which our brain distinguishes the gaze direction (GD), irrespectively of other facial cues, remains unclear. To solve this question, the present study aimed to investigate the time course of gaze direction processing, using an ERP decoding approach, based on the combination of a support vector machine and error-correcting output codes. We recorded EEG in young healthy subjects, 32 of them performing GD detection and 34 conducting face orientation tasks. Both tasks presented 3D realistic faces with five different head and gaze orientations each: 30°, 15° to the left or right, and 0°. While the classical ERP analyses did not show clear GD effects, ERP decoding analyses revealed that discrimination of GD, irrespective of head orientation, started at 140 ms in the GD task and at 120 ms in the face orientation task. GD decoding accuracy was higher in the GD task than in the face orientation task and was the highest for the direct gaze in both tasks. These findings suggest that the decoding of brain patterns is modified by task relevance, which changes the latency and the accuracy of GD decoding.

Using multivariate pattern analysis to increase effect sizes for event-related potential analyses

Multivariate pattern analysis (MVPA) approaches can be applied to the topographic distribution of event-related potential (ERP) signals to "decode" subtly different stimulus classes, such as different faces or different orientations. These approaches are extremely sensitive, and it seems possible that they could also be used to increase effect sizes and statistical power in traditional paradigms that ask whether an ERP component differs in amplitude across conditions. To assess this possibility, we leveraged the open-source ERP CORE data set and compared the effect sizes resulting from conventional univariate analyses of mean amplitude with two MVPA approaches (support vector machine decoding and the cross-validated Mahalanobis distance, both of which are easy to compute using open-source software). We assessed these approaches across seven widely studied ERP components (N170, N400, N2pc, P3b, lateral readiness potential, error related negativity, and mismatch negativity). Across all components, we found that multivariate approaches yielded effect sizes that were as large or larger than the effect sizes produced by univariate approaches. These results indicate that researchers could obtain larger effect sizes, and therefore greater statistical power, by using multivariate analysis of topographic voltage patterns instead of traditional univariate analyses in many ERP studies.

Using Multivariate Pattern Analysis to Increase Effect Sizes for Event-Related Potential Analyses

Multivariate pattern analysis approaches can be applied to the topographic distribution of event-related potential (ERP) signals to 'decode' subtly different stimulus classes, such as different faces or different orientations. These approaches are extremely sensitive, and it seems possible that they could also be used to increase effect sizes and statistical power in traditional paradigms that ask whether an ERP component differs in amplitude across conditions. To assess this possibility, we leveraged the open-source ERP CORE dataset and compared the effect sizes resulting from conventional univariate analyses of mean amplitude with two multivariate pattern analysis approaches (support vector machine decoding and the cross-validated Mahalanobis distance, both of which are easy to compute using open-source software). We assessed these approaches across seven widely studied ERP components (N170, N400, N2pc, P3b, lateral readiness potential, error related negativity, and mismatch negativity). Across all components, we found that multivariate approaches yielded effect sizes that were as large or larger than the effect sizes produced by univariate approaches. These results indicate that researchers could obtain larger effect sizes, and therefore greater statistical power, by using multivariate analysis of topographic voltage patterns instead of traditional univariate analyses in many ERP studies.

Prolonged neural encoding of visual information in autism

Autism spectrum disorder (ASD) is associated with a hyper-focused visual attentional style, impacting higher-order social and affective domains. The understanding of such peculiarity can benefit from the use of multivariate pattern analysis (MVPA) of high-resolution electroencephalography (EEG) data, which has proved to be a powerful technique to investigate the hidden neural dynamics orchestrating sensory and cognitive processes. Here, we recorded EEG in typically developing (TD) children and in children with ASD during a visuo-spatial attentional task where attention was exogenously captured by a small (zoom-in) or large (zoom-out) cue in the visual field before the appearance of a target at different eccentricities. MVPA was performed both in the cue-locked period, to reveal potential differences in the modulation of the attentional focus, and in the target-locked period, to reveal potential cascade effects on stimulus processing. Cue-locked MVPA revealed that while in the TD group the pattern of neural activity contained information about the cue mainly before the target appearance, the ASD group showed a temporally sustained and topographically diffuse significant decoding of the cue neural response even after the target onset, suggesting a delayed extinction of cue-related neural activity. Crucially, this delayed extinction positively correlated with behavioral measures of attentional hyperfocusing. Results of target-locked MVPA were coherent with a hyper-focused attentional profile, highlighting an earlier and stronger decoding of target neural responses in small cue trials in the ASD group. The present findings document a spatially and temporally overrepresented encoding of visual information in ASD, which can constitute one of the main reasons behind their peculiar cognitive style.

Classification of age groups and task conditions provides additional evidence for differences in electrophysiological correlates of inhibitory control across the lifespan

The aim of this study was to extend previous findings on selective attention over a lifetime using machine learning procedures. By decoding group membership and stimulus type, we aimed to study differences in the neural representation of inhibitory control across age groups at a single-trial level. We re-analyzed data from 211 subjects from six age groups between 8 and 83 years of age. Based on single-trial EEG recordings during a flanker task, we used support vector machines to predict the age group as well as to determine the presented stimulus type (i.e., congruent, or incongruent stimulus). The classification of group membership was highly above chance level (accuracy: 55%, chance level: 17%). Early EEG responses were found to play an important role, and a grouped pattern of classification performance emerged corresponding to age structure. There was a clear cluster of individuals after retirement, i.e., misclassifications mostly occurred within this cluster. The stimulus type could be classified above chance level in ~ 95% of subjects. We identified time windows relevant for classification performance that are discussed in the context of early visual attention and conflict processing. In children and older adults, a high variability and latency of these time windows were found. We were able to demonstrate differences in neuronal dynamics at the level of individual trials. Our analysis was sensitive to mapping gross changes, e.g., at retirement age, and to differentiating components of visual attention across age groups, adding value for the diagnosis of cognitive status across the lifespan. Overall, the results highlight the use of machine learning in the study of brain activity over a lifetime.

Altered engagement of the speech motor network is associated with reduced phonological working memory in autism

Nonword repetition, a common clinical measure of phonological working memory, involves component processes of speech perception, working memory, and speech production. Autistic children often show behavioral challenges in nonword repetition, as do many individuals with communication disorders. It is unknown which subprocesses of phonological working memory are vulnerable in autistic individuals, and whether the same brain processes underlie the transdiagnostic difficulty with nonword repetition. We used functional magnetic resonance imaging (fMRI) to investigate the brain bases for nonword repetition challenges in autism. We compared activation during nonword repetition in functional brain networks subserving speech perception, working memory, and speech production between neurotypical and autistic children. Autistic children performed worse than neurotypical children on nonword repetition and had reduced activation in response to increasing phonological working memory load in the supplementary motor area. Multivoxel pattern analysis within the speech production network classified shorter vs longer nonword-repetition trials less accurately for autistic than neurotypical children. These speech production motor-specific differences were not observed in a group of children with reading disability who had similarly reduced nonword repetition behavior. These findings suggest that atypical function in speech production brain regions may contribute to nonword repetition difficulties in autism.

Information-based multivariate decoding reveals imprecise neural encoding in children with attention deficit hyperactivity disorder during visual selective attention

Attention deficit hyperactivity disorder (ADHD) is a common neurodevelopmental disorder in school-age children. Attentional orientation is a potential clinical diagnostic marker to aid in the early diagnosis of ADHD. However, the underlying pathophysiological substrates of impaired attentional orienting in childhood ADHD remain unclear. Electroencephalography (EEG) was measured in 135 school-age children (70 with childhood ADHD and 65 matched typically developing children) to directly investigate target localization during spatial selective attention through univariate ERP analysis and information-based multivariate pattern machine learning analysis. Compared with children with typical development, a smaller N2pc was found in the ADHD group through univariate ERP analysis. Children with ADHD showed a lower parieto-occipital multivariate decoding accuracy approximately 240-340 ms after visual search onset, which predicts a slower reaction time and larger standard deviation of reaction time. Furthermore, a significant correlation was found between N2pc and decoding accuracy in typically developing children but not in children with ADHD. These observations reveal that impaired attentional orienting in ADHD may be due to inefficient neural encoding responses. By using a personalized information-based multivariate machine learning approach, we have advanced the understanding of cognitive deficits in neurodevelopmental disorders. Our study provides potential research directions for the early diagnosis and optimization of personalized intervention in children with ADHD.

Decoding violated sensory expectations from the auditory cortex of anaesthetised mice: Hierarchical recurrent neural network depicts separate 'danger' and 'safety' units

The ability to respond appropriately to sensory information received from the external environment is among the most fundamental capabilities of central nervous systems. In the auditory domain, processes underlying this behaviour are studied by measuring auditory‐evoked electrophysiology during sequences of sounds with predetermined regularities. Identifying neural correlates of ensuing auditory novelty responses is supported by research in experimental animals. In the present study, we reanalysed epidural field potential recordings from the auditory cortex of anaesthetised mice during frequency and intensity oddball stimulation. Multivariate pattern analysis (MVPA) and hierarchical recurrent neural network (RNN) modelling were adopted to explore these data with greater resolution than previously considered using conventional methods. Time‐wise and generalised temporal decoding MVPA approaches revealed previously underestimated asymmetry between responses to sound‐level transitions in the intensity oddball paradigm, in contrast with tone frequency changes. After training, the cross‐validated RNN model architecture with four hidden layers produced output waveforms in response to simulated auditory inputs that were strongly correlated with grand‐average auditory‐evoked potential waveforms (r² > .9). Units in hidden layers were classified based on their temporal response properties and characterised using principal component analysis and sample entropy. These demonstrated spontaneous alpha rhythms, sound onset and offset responses and putative ‘safety’ and ‘danger’ units activated by relatively inconspicuous and salient changes in auditory inputs, respectively. The hypothesised existence of corresponding biological neural sources is naturally derived from this model. If proven, this could have significant implications for prevailing theories of auditory processing.

Early evoked brain activity underlies auditory and audiovisual speech recognition deficits in schizophrenia

•

Reduced N1 amplitudes reflect speech processing deficits in schizophrenia (SZ).

•

Crossmodal N1 amplitude suppression in audiovisual speech is preserved in SZ.

•

N1 amplitudes correlate with speech recognition performance in controls but not in SZ.

Objectives

People with Schizophrenia (SZ) show deficits in auditory and audiovisual speech recognition. It is possible that these deficits are related to aberrant early sensory processing, combined with an impaired ability to utilize visual cues to improve speech recognition. In this electroencephalography study we tested this by having SZ and healthy controls (HC) identify different unisensory auditory and bisensory audiovisual syllables at different auditory noise levels.

Methods

SZ (N = 24) and HC (N = 21) identified one of three different syllables (/da/, /ga/, /ta/) at three different noise levels (no, low, high). Half the trials were unisensory auditory and the other half provided additional visual input of moving lips. Task-evoked mediofrontal N1 and P2 brain potentials triggered to the onset of the auditory syllables were derived and related to behavioral performance.

Results

In comparison to HC, SZ showed speech recognition deficits for unisensory and bisensory stimuli. These deficits were primarily found in the no noise condition. Paralleling these observations, reduced N1 amplitudes to unisensory and bisensory stimuli in SZ were found in the no noise condition. In HC the N1 amplitudes were positively related to the speech recognition performance, whereas no such relationships were found in SZ. Moreover, no group differences in multisensory speech recognition benefits and N1 suppression effects for bisensory stimuli were observed.

Conclusion

Our study suggests that reduced N1 amplitudes reflect early auditory and audiovisual speech processing deficits in SZ. The findings that the amplitude effects were confined to salient speech stimuli and the attenuated relationship with behavioral performance in patients compared to HC, indicates a diminished decoding of the auditory speech signals in SZs. Our study also revealed relatively intact multisensory benefits in SZs, which implies that the observed auditory and audiovisual speech recognition deficits were primarily related to aberrant processing of the auditory syllables.

Does Creativity Influence Visual Perception? - An Event-Related Potential Study With Younger and Older Adults

We do not know enough about the cognitive background of creativity despite its significance. Using an active oddball paradigm with unambiguous and ambiguous portrait paintings as the standard stimuli, our aim was to examine whether: creativity in the figural domain influences the perception of visual stimuli; any stages of visual processing; or if healthy aging has an effect on these processes. We investigated event related potentials (ERPs) and applied ERP decoding analyses in four groups: younger less creative; younger creative; older less creative; and older creative adults. The early visual processing did not differ between creativity groups. In the later ERP stages the amplitude for the creative compared with the less creative groups was larger between 300 and 500 ms. The stimuli types were clearly distinguishable: within the 300–500 ms range the amplitude was larger for ambiguous rather than unambiguous paintings, but this difference in the traditional ERP analysis was only observable in the younger, not elderly groups, who also had this difference when using decoding analysis. Our results could not prove that visual creativity influences the early stage of perception, but showed creativity had an effect on stimulus processing in the 300–500 ms range, in indexing differences in top-down control, and having more flexible cognitive control in the younger creative group.

The Time Course of Face Representations during Perception and Working Memory Maintenance

Successful social communication requires accurate perception and maintenance of invariant (face identity) and variant (facial expression) aspects of faces. While numerous studies investigated how face identity and expression information is extracted from faces during perception, less is known about the temporal aspects of the face information during perception and working memory (WM) maintenance. To investigate how face identity and expression information evolve over time, I recorded electroencephalography (EEG) while participants were performing a face WM task where they remembered a face image and reported either the identity or the expression of the face image after a short delay. Using multivariate event-related potential (ERP) decoding analyses, I found that the two types of information exhibited dissociable temporal dynamics: Although face identity was decoded better than facial expression during perception, facial expression was decoded better than face identity during WM maintenance. Follow-up analyses suggested that this temporal dissociation was driven by differential maintenance mechanisms: Face identity information was maintained in a more “activity-silent” manner compared to facial expression information, presumably because invariant face information does not need to be actively tracked in the task. Together, these results provide important insights into the temporal evolution of face information during perception and WM maintenance.

Neural measures of working memory in a bilateral change detection task

The change detection task is a widely used paradigm to examine visual working memory processes. Participants memorize a set of items and then, try to detect changes in the set after a retention period. The negative slow wave (NSW) and contralateral delay activity (CDA) are event-related potentials in the EEG signal that are commonly used in change detection tasks to track working memory load, as both increase with the number of items maintained in working memory (set size). While the CDA was argued to more purely reflect the memory-specific neural activity than the NSW, it also requires a lateralized design and attention shifts prior to memoranda onset, imposing more restrictions on the task than the NSW. The present study proposes a novel change detection task in which both CDA and NSW can be measured at the same time. Memory items were presented bilaterally, but their distribution in the left and right hemifield varied, inducing a target imbalance or "net load." NSW increased with set size, whereas CDA increased with net load. In addition, a multivariate linear classifier was able to decode the set size and net load from the EEG signal. CDA, NSW, and decoding accuracy predicted an individual's working memory capacity. In line with the notion of a bilateral advantage in working memory, accuracy, and CDA data suggest that participants tended to encode items relatively balanced. In sum, this novel change detection task offers a basis to make use of converging neural measures of working memory in a comprehensive paradigm.