The discovery of processing stages: Extension of Sternberg's method

Word Type and Frequency Effects on Lexical Decisions Are Process-dependent and Start Early

When encountering letter strings, we rapidly determine whether they are words. The speed of such lexical decisions (LDs) is affected by word frequency. Apart from influencing late, decision-related, processing stages, frequency has also been shown to affect very early stages, and even the processing of nonwords. We developed a detailed account of the different frequency effects involved in LDs by (1) dividing LDs into processing stages using a combination of hidden semi-Markov models and multivariate pattern analysis applied to EEG data and (2) using generalized additive mixed models to investigate how the effect of continuous word and nonword frequency differs between these stages. We discovered six stages shared between word types, with the fifth stage consisting of two substages for pseudowords only. In the earliest stages, visual processing was completed faster for frequent words, but took longer for word-like nonwords. Later stages involved an orthographic familiarity assessment followed by an elaborate decision process, both affected differently by frequency. We therefore conclude that frequency indeed affects all processes involved in LDs and that the magnitude and direction of these effects differ both by process and word type.

Tracking the Cognitive Band in an Open-Ended Task

Open-ended tasks can be decomposed into the three levels of Newell's Cognitive Band: the Unit-Task level, the Operation level, and the Deliberate-Act level. We analyzed the video game Co-op Space Fortress at these levels, reporting both the match of a cognitive model to subject behavior and the use of electroencephalogram (EEG) to track subject cognition. The Unit Task level in this game involves coordinating with a partner to kill a fortress. At this highest level of the Cognitive Band, there is a good match between subject behavior and the model. The EEG signals were also strong enough to track when Unit Tasks succeeded or failed. The intermediate Operation level in this task involves legs of flight to achieve a kill. The EEG signals associated with these operations are much weaker than the signals associated with the Unit Tasks. Still, it was possible to reconstruct subject play with much better than chance success. There were significant differences in the leg behavior of subjects and models. Model behavior did not provide a good basis for interpreting a subject's behavior at this level. At the lowest Deliberate-Act level, we observed overlapping key actions, which the model did not display. Such overlapping key actions also frustrated efforts to identify EEG signals of motor actions. We conclude that the Unit-task level is the appropriate level both for understanding open-ended tasks and for using EEG to track the performance of open-ended tasks.

Testing two-step models of negative quantification using a novel machine learning analysis of EEG

The sentences "More than half of the students passed the exam" and "Fewer than half of the students failed the exam" describe the same set of situations, and yet the former results in shorter reaction times in verification tasks. The two-step model explains this result by postulating that negative quantifiers contain hidden negation, which involves an extra processing stage. To test this theory, we applied a novel EEG analysis technique focused on detecting cognitive stages (HsMM-MVPA) to data from a picture-sentence verification task. We estimated the number of processing stages during reading and verification of quantified sentences (e.g. "Fewer than half of the dots are blue") that followed the presentation of pictures containing coloured geometric shapes. We did not find evidence for an extra step during the verification of sentences with fewer than half. We provide an alternative interpretation of our results in line with an expectation-based pragmatic account.

A whole-task brain model of associative recognition that accounts for human behavior and neuroimaging data

Brain models typically focus either on low-level biological detail or on qualitative behavioral effects. In contrast, we present a biologically-plausible spiking-neuron model of associative learning and recognition that accounts for both human behavior and low-level brain activity across the whole task. Based on cognitive theories and insights from machine-learning analyses of M/EEG data, the model proceeds through five processing stages: stimulus encoding, familiarity judgement, associative retrieval, decision making, and motor response. The results matched human response times and source-localized MEG data in occipital, temporal, prefrontal, and precentral brain regions; as well as a classic fMRI effect in prefrontal cortex. This required two main conceptual advances: a basal-ganglia-thalamus action-selection system that relies on brief thalamic pulses to change the functional connectivity of the cortex, and a new unsupervised learning rule that causes very strong pattern separation in the hippocampus. The resulting model shows how low-level brain activity can result in goal-directed cognitive behavior in humans.

Thalamic bursts modulate cortical synchrony locally to switch between states of global functional connectivity in a cognitive task

Performing a cognitive task requires going through a sequence of functionally diverse stages. Although it is typically assumed that these stages are characterized by distinct states of cortical synchrony that are triggered by sub-cortical events, little reported evidence supports this hypothesis. To test this hypothesis, we first identified cognitive stages in single-trial MEG data of an associative recognition task, showing with a novel method that each stage begins with local modulations of synchrony followed by a state of directed functional connectivity. Second, we developed the first whole-brain model that can simulate cortical synchrony throughout a task. The model suggests that the observed synchrony is caused by thalamocortical bursts at the onset of each stage, targeted at cortical synapses and interacting with the structural anatomical connectivity. These findings confirm that cognitive stages are defined by distinct states of cortical synchrony and explains the network-level mechanisms necessary for reaching stage-dependent synchrony states.

Discovering the brain stages of lexical decision: Behavioral effects originate from a single neural decision process

Lexical decision (LD) - judging whether a sequence of letters constitutes a word - has been widely investigated. In a typical lexical decision task (LDT), participants are asked to respond whether a sequence of letters is an actual word or a nonword. Although behavioral differences between types of words/nonwords have been robustly detected in LDT, there is an ongoing discussion about the exact cognitive processes that underlie the word identification process in this task. To obtain data-driven evidence on the underlying processes, we recorded electroencephalographic (EEG) data and applied a novel machine-learning method, hidden semi-Markov model multivariate pattern analysis (HsMM-MVPA). In the current study, participants performed an LDT in which we varied the frequency of words (high, low frequency) and "wordlikeness" of non-words (pseudowords, random non-words). The results revealed that models with six processing stages accounted best for the data in all conditions. While most stages were shared, Stage 5 differed between conditions. Together, these results indicate that the differences in word frequency and lexicality effects are driven by a single cognitive processing stage. Based on its latency and topology, we interpret this stage as a Decision process during which participants discriminate between words and nonwords using activated lexical information.

The Discovery and Interpretation of Evidence Accumulation Stages

To improve the understanding of cognitive processing stages, we combined two prominent traditions in cognitive science: evidence accumulation models and stage discovery methods. While evidence accumulation models have been applied to a wide variety of tasks, they are limited to tasks in which decision-making effects can be attributed to a single processing stage. Here, we propose a new method that first uses machine learning to discover processing stages in EEG data and then applies evidence accumulation models to characterize the duration effects in the identified stages. To evaluate this method, we applied it to a previously published associative recognition task (Application 1) and a previously published random dot motion task with a speed-accuracy trade-off manipulation (Application 2). In both applications, the evidence accumulation models accounted better for the data when we first applied the stage-discovery method, and the resulting parameter estimates where generally in line with psychological theories. In addition, in Application 1 the results shed new light on target-foil effects in associative recognition, while in Application 2 the stage discovery method identified an additional stage in the accuracy-focused condition — challenging standard evidence accumulation accounts. We conclude that the new framework provides a powerful new tool to investigate processing stages.

Brain-behavior relationships in the perceptual decision-making process through cognitive processing stages

Perceptual decision making - the process of detecting and categorizing information - has been studied extensively over the last two decades. In this study, we aim to bridge the gap between neural and behavioral representations of the perceptual decision-making process. The neural characterization of decision-making was investigated by evaluating the duration and neural signature of the information processing stages. We further evaluated the processing stages of the decision-making process at the behavioral level by estimating the drift rate and non-decision time parameters. We asked whether the neural and behavioral characterizations of the decision-making process provided consistent results under different stimulus coherency levels and spatial attention. Our statistical analysis revealed that, at both representational levels, decision-making was affected more by the coherency factor. We further found that among different information processing stages, the decision stage had the highest role in the performance of the decision-making process. Such that, the shorter decision stage duration at the neural level and higher drift rate at the behavioral level lead to faster decision-making. Through our consistent neural and behavioral results, we have shown that the decision-making components at these two representational levels were significantly associated. Moreover, the neural signature of the processing stages gave information about the regions that contributed more to the decision-making process. Our overall results demonstrate that uncovering the cognitive processing stages provided more insights into the decision-making process.

EEG-based Identification of Evidence Accumulation Stages in Decision-Making

Dating back to the 19th century, the discovery of processing stages has been of great interest to researchers in cognitive science. The goal of this paper is to demonstrate the validity of a recently developed method, hidden semi-Markov model multivariate pattern analysis (HsMM-MVPA), for discovering stages directly from EEG data, in contrast to classical reaction-time-based methods. To test the validity of stages discovered with the HsMM-MVPA method, we applied it to two relatively simple tasks where the interpretation of processing stages is straightforward. In these visual discrimination EEG data experiments, perceptual processing and decision difficulty were manipulated. The HsMM-MVPA revealed that participants progressed through five cognitive processing stages while performing these tasks. The brain activation of one of those stages was dependent on perceptual processing, whereas the brain activation and the duration of two other stages were dependent on decision difficulty. In addition, evidence accumulation models (EAMs) were used to assess to what extent the results of HsMM-MVPA are comparable to standard reaction-time-based methods. Consistent with the HsMM-MVPA results, EAMs showed that nondecision time varied with perceptual difficulty and drift rate varied with decision difficulty. Moreover, nondecision and decision time of the EAMs correlated highly with the first two and last three stages of the HsMM-MVPA, respectively, indicating that the HsMM-MVPA gives a more detailed description of stages discovered with this more classical method. The results demonstrate that cognitive stages can be robustly inferred with the HsMM-MVPA.

Neural habituation enhances novelty detection: an EEG study of rapidly presented words

Huber and O'Reilly (2003) proposed that neural habituation aids perceptual processing, separating neural responses to currently viewed objects from recently viewed objects. However, synaptic depression has costs, producing repetition deficits. Prior work confirmed the transition from repetition benefits to deficits with increasing duration of a prime object, but the prediction of enhanced novelty detection was not tested. The current study examined this prediction with a same/different word priming task, using support vector machine (SVM) classification of EEG data, ERP analyses focused on the N400, and dynamic neural network simulations fit to behavioral data to provide a priori predictions of the ERP effects. Subjects made same/different judgements to a response word in relation to an immediately preceding brief target word; prime durations were short (50ms) or long (400ms), and long durations decreased P100/N170 responses to the target word, suggesting that this manipulation increased habituation. Following long duration primes, correct "different" judgments of primed response words increased, evidencing enhanced novelty detection. An SVM classifier predicted trial-by-trial behavior with 66.34% accuracy on held-out data, with greatest predictive power at a time pattern consistent with the N400. The habituation model was augmented with a maintained semantics layer (i.e., working memory) to generate behavior and N400 predictions. A second experiment used response-locked ERPs, confirming the model's assumption that residual activation in working memory is the basis of novelty decisions. These results support the theory that neural habituation enhances novelty detection, and the model assumption that the N400 reflects updating of semantic information in working memory.

Spatiotemporal analysis of event-related fMRI to reveal cognitive states

Cognitive science has a rich history of developing theories of processing that characterize the mental steps involved in performance of many tasks. Recent work in neuroimaging and machine learning has greatly improved our ability to link cognitive processes with what is happening in the brain. This article analyzes a hidden semi-Markov model-multivoxel pattern-analysis (HSMM-MVPA) methodology that we have developed for inferring the sequence of brain states one traverses in the performance of a cognitive task. The method is applied to a functional magnetic resonance imaging (fMRI) experiment where task boundaries are known that should separate states. The method is able to accurately identify those boundaries. Then, applying the method to synthetic data, we explore more fully those factors that influence performance of the method: signal-to-noise ratio, numbers of states, state sojourn times, and numbers of underlying experimental conditions. The results indicate the types of experimental tasks where applications of the HSMM-MVPA method are likely to yield accurate and insightful results.

Beyond mean reaction times: Combining distributional analyses with processing stage manipulations in the Simon task

We combined analyses of reaction time (RT) distributions with experimental manipulations of different processing stages (perception, decision, motor execution) in a Simon task to investigate which changes in Simon effects could be explained entirely by fading irrelevant response activation. Consistent with fading activation accounts, the Simon effect on mean RT was usually smaller for conditions with slower responses (Expts. 1-3 but not Expt. 4), and delta plot analyses revealed that it was always smaller for the slower responses within each condition. Critically, however, these analyses also revealed that some experimental manipulations produced upward or downward shifts in the RT delta plots, thus altering the Simon effect on mean RT in ways that could not be explained by fading activation. The results demonstrate the power of combining RT distributional analyses with experimental manipulations to reveal mechanisms contributing to the Simon effect that would not be revealed using only mean RT. We consider alternatives to fading activation accounts of decreasing delta plots and discuss the contribution of different cognitive stages in modulating Simon effects.

Uncovering cognitive constraints is the bottleneck in resource-rational analysis

A major constraint in resource-rational analysis is cognitive resources. Yet, uncovering the nature of individual components of the human mind has progressed slowly, because even the simplest behavior is a function of most (if not all) of the mind. Accelerating our understanding of the mind's structure requires more efforts in developing cognitive architectures.

The Common Time Course of Memory Processes Revealed

Magnetoencephalography (MEG) was used to compare memory processes in two experiments, one involving recognition of word pairs and the other involving recall of newly learned arithmetic facts. A combination of hidden semi-Markov models and multivariate pattern analysis was used to locate brief "bumps" in the sensor data that marked the onset of different stages of cognitive processing. These bumps identified a separation between a retrieval stage that identified relevant information in memory and a decision stage that determined what response was implied by that information. The encoding, retrieval, decision, and response stages displayed striking similarities across the two experiments in their duration and brain activation patterns. Retrieval and decision processes involve distinct brain activation patterns. We conclude that memory processes for two different tasks, associative recognition versus arithmetic retrieval, follow a common spatiotemporal neural pattern and that both tasks have distinct retrieval and decision stages.

Dynamics of brain activity reveal a unitary recognition signal

Dual-process models of recognition memory typically assume that independent familiarity and recollection signals with distinct temporal profiles can each lead to recognition (enabling 2 routes to recognition), whereas single-process models posit a unitary "memory strength" signal. Using multivariate classifiers trained on spectral electroencephalogram (EEG) features, we quantified neural evidence for recognition decisions as a function of time. Classifiers trained on a small portion of the decision period performed similarly to those also incorporating information from previous time points indicating that neural activity reflects an integrated evidence signal. We propose a single-route account of recognition memory that is compatible with contributions from familiarity and recollection signals, but relies on a unitary evidence signal that integrates all available evidence. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Single-Trial EEG Analysis Predicts Memory Retrieval and Reveals Source-Dependent Differences

We used pattern classifiers to extract features related to recognition memory retrieval from the temporal information in single-trial electroencephalography (EEG) data during attempted memory retrieval. Two-class classification was conducted on correctly remembered trials with accurate context (or source) judgments vs. correctly rejected trials. The average accuracy for datasets recorded in a single session was 61% while the average accuracy for datasets recorded in two separate sessions was 56%. To further understand the basis of the classifier’s performance, two other pattern classifiers were trained on different pairs of behavioral conditions. The first of these was designed to use information related to remembering the item and the second to use information related to remembering the contextual information (or source) about the item. Mollison and Curran (2012) had earlier shown that subjects’ familiarity judgments contributed to improved memory of spatial contextual information but not of extrinsic associated color information. These behavioral results were similarly reflected in the event-related potential (ERP) known as the FN400 (an early frontal effect relating to familiarity) which revealed differences between correct and incorrect context memories in the spatial but not color conditions. In our analyses we show that a classifier designed to distinguish between correct and incorrect context memories, more strongly involves early activity (400–500 ms) over the frontal channels for the location distinctions, than for the extrinsic color associations. In contrast, the classifier designed to classify memory for the item (without memory for the context), had more frontal channel involvement for the color associated experiments than for the spatial experiments. Taken together these results argue that location may be bound more tightly with the item than an extrinsic color association. The multivariate classification approach also showed that trial-by-trial variation in EEG corresponding to these ERP components were predictive of subjects’ behavioral responses. Additionally, the multivariate classification approach enabled analysis of error conditions that did not have sufficient trials for standard ERP analyses. These results suggested that false alarms were primarily attributable to item memory (as opposed to memory of associated context), as commonly predicted, but with little previous corroborating EEG evidence.

Characterizing synchrony patterns across cognitive task stages of associative recognition memory

Numerous studies seek to understand the role of oscillatory synchronization in cognition. This problem is particularly challenging in the context of complex cognitive behavior, which consists of a sequence of processing steps with uncertain duration. In this study, we analyzed oscillatory connectivity measures in time windows that previous computational models had associated with a specific sequence of processing steps in an associative memory recognition task (visual encoding, familiarity, memory retrieval, decision making, and motor response). The timing of these processing steps was estimated on a single‐trial basis with a novel hidden semi‐Markov model multivariate pattern analysis (HSMM‐MVPA) method. We show that different processing stages are associated with specific patterns of oscillatory connectivity. Visual encoding is characterized by a dense network connecting frontal, posterior, and temporal areas as well as frontal and occipital phase locking in the 4–9 Hz theta band. Familiarity is associated with frontal phase locking in the 9–14 Hz alpha band. Decision making is associated with frontal and temporo‐central interhemispheric connections in the alpha band. During decision making, a second network in the theta band that connects left‐temporal, central, and occipital areas bears similarity to the neural signature for preparing a motor response. A similar theta band network is also present during the motor response, with additionally alpha band connectivity between right‐temporal and posterior areas. This demonstrates that the processing stages discovered with the HSMM‐MVPA method are indeed linked to distinct synchronization patterns, leading to a closer understanding of the functional role of oscillations in cognition.

Inter-subject alignment of MEG datasets in a common representational space

Pooling neural imaging data across subjects requires aligning recordings from different subjects. In magnetoencephalography (MEG) recordings, sensors across subjects are poorly correlated both because of differences in the exact location of the sensors, and structural and functional differences in the brains. It is possible to achieve alignment by assuming that the same regions of different brains correspond across subjects. However, this relies on both the assumption that brain anatomy and function are well correlated, and the strong assumptions that go into solving the under-determined inverse problem given the high-dimensional source space. In this article, we investigated an alternative method that bypasses source-localization. Instead, it analyzes the sensor recordings themselves and aligns their temporal signatures across subjects. We used a multivariate approach, multiset canonical correlation analysis (M-CCA), to transform individual subject data to a low-dimensional common representational space. We evaluated the robustness of this approach over a synthetic dataset, by examining the effect of different factors that add to the noise and individual differences in the data. On an MEG dataset, we demonstrated that M-CCA performs better than a method that assumes perfect sensor correspondence and a method that applies source localization. Last, we described how the standard M-CCA algorithm could be further improved with a regularization term that incorporates spatial sensor information. Hum Brain Mapp 38:4287-4301, 2017. © 2017 Wiley Periodicals, Inc.

The Effects of Probe Similarity on Retrieval and Comparison Processes in Associative Recognition

In this study, we investigated the information processing stages underlying associative recognition. We recorded EEG data while participants performed a task that involved deciding whether a probe word triple matched any previously studied triple. We varied the similarity between probes and studied triples. According to a model of associative recognition developed in the Adaptive Control of Thought-Rational cognitive architecture, probe similarity affects the duration of the retrieval stage: Retrieval is fastest when the probe is similar to a studied triple. This effect may be obscured, however, by the duration of the comparison stage, which is fastest when the probe is not similar to the retrieved triple. Owing to the opposing effects of probe similarity on retrieval and comparison, overall RTs provide little information about each stage's duration. As such, we evaluated the model using a novel approach that decomposes the EEG signal into a sequence of latent states and provides information about the durations of the underlying information processing stages. The approach uses a hidden semi-Markov model to identify brief sinusoidal peaks (called bumps) that mark the onsets of distinct cognitive stages. The analysis confirmed that probe type has opposite effects on retrieval and comparison stages.