Tracking the continuity of language comprehension: computer mouse trajectories suggest parallel syntactic processing

Category Locality Theory: A unified account of locality effects in sentence comprehension

In real-time sentence comprehension, the comprehender is often required to establish syntactic dependencies between words that are linearly distant. Major models of sentence comprehension assume that longer dependencies are more difficult to process because of working memory limitations. While the expected effect of distance on reading times (locality effect) has been robustly observed in certain constructions, such as relative clauses in English, its generalizability to a wider range of constructions has been empirically questioned. The current study proposes a new metric of syntactic distance that capitalizes on the flexible constituency of Combinatory Categorial Grammar (CCG), and argues that it offers a unified account of the locality effects. It is shown that this metric correctly predicts both the presence of the locality effect in English relative clauses and its absence in verb-final languages, without assuming language- or dependency-specific differences in the sensitivity to the locality effect. It is further shown that the CCG-based distance is a significant predictor of the self-paced reading times from an English corpus, even when other known predictors such as dependency-based locality and surprisal are taken into account. These results suggest that human sentence comprehension involves rapid integration of input words into efficiently compressed syntactic representations, and CCG is a plausible theory of the grammar that subserves this process.

Cognitive Science Progresses Toward Interactive Frameworks

Despite its many twists and turns, the arc of cognitive science generally bends toward progress, thanks to its interdisciplinary nature. By glancing at the last few decades of experimental and computational advances, it can be argued that-far from failing to converge on a shared set of conceptual assumptions-the field is indeed making steady consensual progress toward what can broadly be referred to as interactive frameworks. This inclination is apparent in the subfields of psycholinguistics, visual perception, embodied cognition, extended cognition, neural networks, dynamical systems theory, and more. This pictorial essay briefly documents this steady progress both from a bird's eye view and from the trenches. The conclusion is one of optimism that cognitive science is getting there, albeit slowly and arduously, like any good science should.

Predictive Sentence Processing at Speed: Evidence from Online Mouse Cursor Tracking

Three online mouse cursor-tracking experiments investigated predictive sentence processing at speed. Participants viewed visual arrays with objects like a bike and kite while hearing predictive sentences like, "What the man will ride, which is shown on this page, is the bike," or non-predictive sentences like, "What the man will spot, which is shown on this page, is the bike." Based on the selectional restrictions of "ride" (i.e., vs. "spot"), participants made mouse cursor movements to the bike before hearing the noun "bike." Compellingly, this effect was observed at speech rates of ∼3 (Experiment 1), ∼6 (Experiment 2), and ∼9 (Experiment 3) syllables/s. While prior research suggests striking limits on prediction, these results highlight temporal dynamics that may impact comprehenders' ability to preactivate information when hearing impressively rapid speech. Implications for theories of sentence processing are discussed.

Continuous cursor-captured conceptual competition: Investigating the spatiotemporal dynamics of spoken word comprehension

Semantically related concepts are coactivated during spoken word comprehension. Two internet-mediated cursor-tracking experiments examined the spatiotemporal dynamics of this coactivation. Participants viewed visual arrays containing images of a target (e.g., accordion) and a semantically related (e.g., banjo) or unrelated (e.g., plum) distractor whilst hearing the target word (e.g., "accordion"). Participants were tasked with moving their cursor from the bottom of the visual array to the target in one of the upper corners. In contrast to Experiment 1, the onset of stimulus presentation was triggered by cursor movement in Experiment 2. Across both experiments, temporal (e.g., RT) and spatial (e.g., AUC) measures revealed significantly greater attraction to images of semantically related compared with unrelated distractors. These results reveal that online cursor-tracking methods are sensitive to semantic competition and suitable for studying the activation of semantic knowledge during language comprehension.

A method for estimating the time of initiating correct categorization in mouse-tracking

Mouse-tracking facilitates exploration of the mental processes underlying decision-making. As the cognitive system works to settle on a decision, response competition manifests in the motor movements of the hand, bringing the mouse relatively closer to one alternative versus the other. Many metrics provide insight into decision-making processes by indexing the shape or complexity of the mouse trajectory. Lacking, however, is a metric that estimates the point in time when a participant begins to correctly categorize a stimulus. We rectify this absence by introducing a metric we refer to as time of initiating correct categorization (TICC), which is the point in time when people began moving relatively closer to the selected target relative to the distractor. We briefly review existing approaches to measuring time in mouse-tracking before describing the TICC and demonstrating its utility in three data sets.

Temporal dynamics of a perceptual decision

Previous research suggests that cognitive factors acting in a top-down manner influence the perceptual interpretation of ambiguous stimuli. To examine the temporal unfolding of these influences as a perceptual decision evolves, we have implemented a modified version of the stream-bounce display. Our novel approach allows us to track responses to stream-bounce stimuli dynamically over the entire course of the motion sequence rather than collecting a subjective report after the fact. Using a trackpad, we had participants control a cursor to track a stream-bounce target actively from start to end and measured tracking speed throughout as the dependent variable. Our paradigm replicated the typical effect of visual-only displays being associated with a streaming bias and audiovisual displays with a bouncing bias. Our main finding is a significant behavioral change preceding a perceptual decision that then predicts that decision. Specifically, for trials in which the sound was presented, tracking speeds were significantly slower starting 500 ms before the point of coincidence and presentation of the sound for bounce compared to stream responses. We suggest that behavioral response may reflect a cognitive expectation of a perceptual outcome that then biases action and the interpretation of sensory input to favor that forthcoming percept in a manner consistent with both the predictive-coding and common-coding theoretical frameworks. Our approach provides a novel behavioral corroboration of recent imaging studies that are suggestive of early brain activity in perception and action.

Social Judgments of Digitally Manipulated Stuttered Speech: An Evaluation of Self-Disclosure on Cognition

Purpose Persons who stutter (PWS) may be susceptible to discrimination because of negative judgments made by listeners. The current study sought to determine how the cognitive system's explicit (i.e., conscious) and implicit (i.e., nonconscious) biases about PWS are impacted by self-disclosure. Method A computer mouse-tracking paradigm was used to evaluate categorical social judgments about PWS. Computer mouse trajectories, which have been shown to reveal underlying cognitive pull or competition between opposing concepts, were used to measure implicit bias (i.e., nonconscious stereotypes). Participants were asked to explicitly categorize the speaker as either intelligent or unintelligent before and after listening to a speaker self-disclose. Mouse cursor trajectories during the explicit response categorization were used to evaluate implicit bias associated with the decision-making process. Results Results indicated that participants chose "intelligent" for a higher proportion of the trials in the disclosure condition compared to baseline, showing that listeners' explicit biases changed after listening to a self-disclosure that the speaker stutters. Results also indicated listeners exhibited a more negative implicit bias, based on computer mouse trajectories, when rating the PWS relative to the "persons who do not stutter" talker, but this negative implicit bias did seem to reduce over time after the disclosure was made. Conclusions These findings indicate that, even though explicit and implicit biases were evident when listeners heard stuttering, both explicit and implicit biases seemed to extinguish over time after a self-disclosure. Although the bias was not completely extinguished, these results provide promising evidence toward developing methods to reduce negative beliefs and reactions toward PWS. Supplemental Material http://osf.io/mwrp7/.

Real-time social reasoning: the effect of disfluency on the meaning of some

The scalar quantifier some is locally ambiguous between pragmatic (some-but-not-all) and literal (some-and-possibly-all) meanings. Although comprehenders typically favour an eventual pragmatic interpretation, debate persists regarding what factors influence interpretation, the time course of comprehension, and whether literal meaning takes precedence. We investigate how the interpretation of some depends on social reasoning derived from a speaker’s manner of delivery. Specifically, we test the effect of disfluency on the derivation of meaning in a context where hesitation may signal speaker embarrassment due to potential face-loss associated with the literal meaning of “some”. Participants $$(n=24)$$(n=24) viewed displays comprising two different snack quantities while hearing a recorded utterance describing how much a speaker had eaten. Critical utterances $$(n=16)$$(n=16) contained the quantifier some, half with a filled pause disfluency (“I ate <uh>, some oreos”). Participants’ eye and mouse movements showed (via empirical logit regressions) that fluent utterances yielded a bias toward a pragmatic interpretation, while disfluency attenuated this bias in favour of the literal meaning (where the speaker ate all the oreos). Crucially, this difference emerged rapidly post-onset of some. Taken together, our findings do not support a literal-first account of scalar comprehension, but rather, suggest that some is interpreted rapidly in a context-dependent manner.

Computer mouse tracking reveals motor signatures in a cognitive task of spatial language grounding

In a novel computer mouse tracking paradigm, participants read a spatial phrase such as "The blue item to the left of the red one" and then see a scene composed of 12 visual items. The task is to move the mouse cursor to the target item (here, blue), which requires perceptually grounding the spatial phrase. This entails visually identifying the reference item (here, red) and other relevant items through attentional selection. Response trajectories are attracted toward distractors that share the target color but match the spatial relation less well. Trajectories are also attracted toward items that share the reference color. A competing pair of items that match the specified colors but are in the inverse spatial relation increases attraction over-additively compared to individual items. Trajectories are also influenced by the spatial term itself. While the distractor effect resembles deviation toward potential targets in previous studies, the reference effect suggests that the relevance of the reference item for the relational task, not its role as a potential target, was critical. This account is supported by the strengthened effect of a competing pair. We conclude, therefore, that the attraction effects in the mouse trajectories reflect the neural processes that operate on sensorimotor representations to solve the relational task. The paradigm thus provides an experimental window through motor behavior into higher cognitive function and the evolution of activation in modal substrates, a longstanding topic in the area of embodied cognition.

Mouse tracking as a window into decision making

Mouse tracking promises to be an efficient method to investigate the dynamics of cognitive processes: It is easier to deploy than eyetracking, yet in principle it is much more fine-grained than looking at response times. We investigated these claimed benefits directly, asking how the features of decision processes-notably, decision changes-might be captured in mouse movements. We ran two experiments, one in which we explicitly manipulated whether our stimuli triggered a flip in decision, and one in which we replicated more ecological, classical mouse-tracking results on linguistic negation (Dale & Duran, Cognitive Science, 35, 983-996, 2011). We concluded, first, that spatial information (mouse path) is more important than temporal information (speed and acceleration) for detecting decision changes, and we offer a comparison of the sensitivities of various typical measures used in analyses of mouse tracking (area under the trajectory curve, direction flips, etc.). We do so using an "optimal" analysis of our data (a linear discriminant analysis explicitly trained to classify trajectories) and see what type of data (position, speed, or acceleration) it capitalizes on. We also quantify how its results compare with those based on more standard measures.

Doing psychological science by hand

Over the past decade, mouse-tracking in choice tasks has become a popular method across psychological science. This method exploits hand movements as a measure of multiple response activations that can be tracked continuously over hundreds of milliseconds. Whereas early mouse-tracking research focused on specific debates, researchers have realized the methodology has far broader theoretical value. This more recent work demonstrates that mouse-tracking is a widely applicable measure across the field, capable of exposing the micro-structure of real-time decisions including their component processes and millisecond-resolution time-course in ways that inform theory. In the article, recent advances in the mouse-tracking approach are described, and comparisons with the gold standard measure of reaction time and other temporally-sensitive methodologies are provided. Future directions, including mapping to neural representations with brain-imaging and ways to improve our theoretical understanding of mouse-tracking methodology, are discussed.

Deconstructing the Gratton effect: Targeting dissociable trial sequence effects in children, pre-adolescents, and adults

The Gratton effect refers to the observation that performance on congruency tasks is often enhanced when the congruency of the current trial matches that of the previous trial. This effect has been at the center of recent debates in the literature on cognitive control as researchers have sought to identify the cognitive and neural underpinnings of the effect. Here, we use a technique known as reach tracking to demonstrate that the Gratton effect originally observed in the flanker task is not a singular effect but the result of two separate trial sequence effects that impact dissociable processes underlying cognitive control. Further, our results indicate that these dissociable processes follow divergent developmental trajectories across childhood, pre-adolescence, and adulthood. Taken together, these findings suggest that manual dynamics can be used to disentangle how key processes underlying cognitive control contribute to the response time effects observed across a wide range of cognitive tasks and age groups.

Minding One's Reach (To Eat): The Promise of Computer Mouse-Tracking to Study Self-Regulation of Eating

In this review, we present the case for using computer mouse-tracking techniques to examine psychological processes that support (and hinder) self-regulation of eating. We first argue that computer mouse-tracking is suitable for studying the simultaneous engagement of-and dynamic interactions between-multiple perceptual and cognitive processes as they unfold and interact over a fine temporal scale (i.e., hundreds of milliseconds). Next, we review recent work that implemented mouse-tracking techniques by measuring mouse movements as participants chose between various food items (of varying nutritional content). Lastly, we propose next steps for future investigations to link behavioral features from mouse-tracking paradigms, corresponding neural correlates, and downstream eating behaviors.

Augmented Go/No-Go Task: Mouse Cursor Motion Measures Improve ADHD Symptom Assessment in Healthy College Students

Attention deficit/hyperactivity disorder (ADHD) is frequently characterized as a disorder of executive function (EF). However, behavioral tests of EF, such as go/No-go tasks, often fail to grasp the deficiency in EF revealed by questionnaire-based measures. This inability is usually attributed to questionnaires and behavioral tasks assessing different constructs of EFs. We propose an additional explanation for this discrepancy. We hypothesize that this problem stems from the lack of dynamic assessment of decision-making (e.g., continuous monitoring of motor behavior such as velocity and acceleration in choice reaching) in classical versions of behavioral tasks. We test this hypothesis by introducing dynamic assessment in the form of mouse motion in a go/No-go task. Our results indicate that, among healthy college students, self-report measures of ADHD symptoms become strongly associated with performance in behavioral tasks when continuous assessment (e.g., acceleration in the mouse-cursor motion) is introduced.

Decision landscapes: visualizing mouse-tracking data

Computerized paradigms have enabled gathering rich data on human behaviour, including information on motor execution of a decision, e.g. by tracking mouse cursor trajectories. These trajectories can reveal novel information about ongoing decision processes. As the number and complexity of mouse-tracking studies increase, more sophisticated methods are needed to analyse the decision trajectories. Here, we present a new computational approach to generating decision landscape visualizations based on mouse-tracking data. A decision landscape is an analogue of an energy potential field mathematically derived from the velocity of mouse movement during a decision. Visualized as a three-dimensional surface, it provides a comprehensive overview of decision dynamics. Employing the dynamical systems theory framework, we develop a new method for generating decision landscapes based on arbitrary number of trajectories. This approach not only generates three-dimensional illustration of decision landscapes, but also describes mouse trajectories by a number of interpretable parameters. These parameters characterize dynamics of decisions in more detail compared with conventional measures, and can be compared across experimental conditions, and even across individuals. The decision landscape visualization approach is a novel tool for analysing mouse trajectories during decision execution, which can provide new insights into individual differences in the dynamics of decision making.

Cognitive control in action: Tracking the dynamics of rule switching in 5- to 8-year-olds and adults

Recent studies have suggested that dissociable processes featuring distinct types of inhibition support cognitive control in tasks requiring participants to override a prepotent response with a control-demanding alternative response. An open question concerns how these processes support cognitive flexibility in rule-switching tasks. We used a technique known as reach tracking to investigate how 5- to 8-year-olds (Experiment 1) and adults (Experiment 2) select, maintain, and switch between incompatible rule sets in a computerized version of the Dimensional Change Card Sort (DCCS). Our results indicate that rule switching differentially impacts two key processes underlying cognitive control in children and adults. Adult performance also revealed a strong response bias not observed in children, which complicated a direct comparison of switching between the age groups and reopens questions concerning the relation between child and adult performance on the task. We discuss these findings in the context of a contemporary model of cognitive control.

When expectancies collide: Action dynamics reveal the interaction between stimulus plausibility and congruency

The cognitive architecture routinely relies on expectancy mechanisms to process the plausibility of stimuli and establish their sequential congruency. In two computer mouse-tracking experiments, we use a cross-modal verification task to uncover the interaction between plausibility and congruency by examining their temporal signatures of activation competition as expressed in a computer- mouse movement decision response. In this task, participants verified the content congruency of sentence and scene pairs that varied in plausibility. The order of presentation (sentence-scene, scene-sentence) was varied between participants to uncover any differential processing. Our results show that implausible but congruent stimuli triggered less accurate and slower responses than implausible and incongruent stimuli, and were associated with more complex angular mouse trajectories independent of the order of presentation. This study provides novel evidence of a disassociation between the temporal signatures of plausibility and congruency detection on decision responses.

Good-enough linguistic representations and online cognitive equilibrium in language processing

We review previous research showing that representations formed during language processing are sometimes just “good enough” for the task at hand and propose the “online cognitive equilibrium” hypothesis as the driving force behind the formation of good-enough representations in language processing. Based on this view, we assume that the language comprehension system by default prefers to achieve as early as possible and remain as long as possible in a state of cognitive equilibrium where linguistic representations are successfully incorporated with existing knowledge structures (i.e., schemata) so that a meaningful and coherent overall representation is formed, and uncertainty is resolved or at least minimized. We also argue that the online equilibrium hypothesis is consistent with current theories of language processing, which maintain that linguistic representations are formed through a complex interplay between simple heuristics and deep syntactic algorithms and also theories that hold that linguistic representations are often incomplete and lacking in detail. We also propose a model of language processing that makes use of both heuristic and algorithmic processing, is sensitive to online cognitive equilibrium, and, we argue, is capable of explaining the formation of underspecified representations. We review previous findings providing evidence for underspecification in relation to this hypothesis and the associated language processing model and argue that most of these findings are compatible with them.

Reach tracking reveals dissociable processes underlying cognitive control

The current study uses reach tracking to investigate how cognitive control is implemented during online performance of the Stroop task (Experiment 1) and the Eriksen flanker task (Experiment 2). We demonstrate that two of the measures afforded by reach tracking, initiation time and reach curvature, capture distinct patterns of effects that have been linked to dissociable processes underlying cognitive control in electrophysiology and functional neuroimaging research. Our results suggest that initiation time reflects a response threshold adjustment process involving the inhibition of motor output, while reach curvature reflects the degree of co-activation between response alternatives registered by a monitoring process over the course of a trial. In addition to shedding new light on fundamental questions concerning how these processes contribute to the cognitive control of behavior, these results present a framework for future research to investigate how these processes function across different tasks, develop across the lifespan, and differ among individuals.

Semantic support and parallel parsing in Chinese

Two eye-tracking experiments were conducted using written Chinese sentences that contained a multi-word ambiguous region. The goal was to determine whether readers maintained multiple interpretations throughout the ambiguous region or selected a single interpretation at the point of ambiguity. Within the ambiguous region, we manipulated the strength of support for the complement clause (CC) analysis and the relative clause (RC) analysis of the ambiguous construction Verb NP1 de NP2. In Experiment 1, the critical sentences were disambiguated to the dispreferred CC interpretation; in Experiment 2, the sentences were disambiguated as the preferred RC interpretation. Unsurprisingly, processing difficulty at the point of disambiguation was observed only in Experiment 1. As predicted by a parallel mechanism, greater processing difficulty arose at disambiguation when the RC interpretation was much more strongly supported by semantic cues relative to the CC alternative, than when the two analyses were semantically supported to a similar degree. Regression analyses confirmed that the degree of semantic support predicted processing difficulty at disambiguation. The findings provide evidence for a parallel constraint-based parsing mechanism.

Advanced mouse-tracking analytic techniques for enhancing psychological science

Computer mouse-tracking is a relatively recently developed behavioral methodology that can contribute unique insight into a wide variety of psychological phenomena. By recording mouse movements en route to specific responses on a screen, researchers glean continuous information about tentative commitments to multiple response alternatives over time. This approach yields a richness of data that can be fully explored with a variety of sophisticated analytic techniques, but these approaches are relatively underutilized and can be difficult to adopt. Here we describe several techniques for researchers to examine the onset and timing of evolving decision processes; test the degree of response competition at different time points; assess trajectory complexity with spatial disorder analyses; identify qualitatively distinct psychological processes during response generation; and finally to distill unique and meaningful components from mouse-tracking data for subsequent analysis. With this guide, we hope researchers can address novel hypotheses otherwise inaccessible with more traditional methods.

Local dynamics in decision making: The evolution of preference within and across decisions

Within decisions, perceived alternatives compete until one is preferred. Across decisions, the playing field on which these alternatives compete evolves to favor certain alternatives. Mouse cursor trajectories provide rich continuous information related to such cognitive processes during decision making. In three experiments, participants learned to choose symbols to earn points in a discrimination learning paradigm and the cursor trajectories of their responses were recorded. Decisions between two choices that earned equally high-point rewards exhibited far less competition than decisions between choices that earned equally low-point rewards. Using positional coordinates in the trajectories, it was possible to infer a potential field in which the choice locations occupied areas of minimal potential. These decision spaces evolved through the experiments, as participants learned which options to choose. This visualisation approach provides a potential framework for the analysis of local dynamics in decision-making that could help mitigate both theoretical disputes and disparate empirical results.

Processing temporary syntactic ambiguity: the effect of contextual bias

This paper reports two experiments using sentences with a temporary ambiguity between a direct object and a sentence complement analysis that is resolved toward the normally preferred direct object analysis. Postverbal noun phrases in these sentences could be ambiguously attached as either a direct object or the subject of a sentence complement, whereas in unambiguous versions of the sentences the subcategorization of the verb forced the direct object interpretation. Participants read these sentences in relatively long paragraph contexts, where the context supported the direct object analysis ("preferred"), supported the sentence complement analysis ("unpreferred"), or provided conflicting evidence about both analyses ("conflicting"). Self-paced reading times for ambiguous postverbal noun phrases were almost equivalent to the reading times of their unambiguous counterparts, even in unpreferred and conflicted context conditions. However, time to read a following region, which forced the direct object interpretation, was affected by the interaction of verb subcategorization ambiguity and contextual support. The full pattern of results do not fit well with either an unelaborated single-analysis ("garden path") model or a competitive constraint-satisfaction model, but are consistent with a race model in which multiple factors affect the speed of constructing a single initial analysis.

Hand in motion reveals mind in motion

Recently, researchers have measured hand movements en route to choices on a screen to understand the dynamics of a broad range of psychological processes. We review this growing body of research and explain how manual action exposes the real-time unfolding of underlying cognitive processing. We describe how simple hand motions may be used to continuously index participants’ tentative commitments to different choice alternatives during the evolution of a behavioral response. As such, hand-tracking can provide unusually high-fidelity, real-time motor traces of the mind. These motor traces cast novel theoretical and empirical light onto a wide range of phenomena and serve as a potential bridge between far-reaching areas of psychological science – from language, to high-level cognition and learning, to social cognitive processes.

Pupillometry reveals processing load during spoken language comprehension

This study investigated processing effort by measuring peoples' pupil diameter as they listened to sentences containing a temporary syntactic ambiguity. In the first experiment, we manipulated prosody. The results showed that when prosodic structure conflicted with syntactic structure, pupil diameter reliably increased. In the second experiment, we manipulated both prosody and visual context. The results showed that when visual context was consistent with the correct interpretation, prosody had very little effect on processing effort. However, when visual context was inconsistent with the correct interpretation, prosody had a large effect on processing effort. The interaction between visual context and prosody shows that visual context has an effect on online processing and that it can modulate the influence of linguistic sources of information, such as prosody. Pupillometry is a sensitive measure of processing effort during spoken language comprehension.

Computer-mouse tracking reveals TMS disruptions of prefrontal function during semantic retrieval

Converging evidence from neuroimaging and neuropsychological studies is essential for understanding human frontal cortical function. We introduce a new method for studying the effects of transient disruptions of frontal activity during transcranial magnetic stimulation (TMS). Using a novel combination of TMS and computer-mouse tracking, through two experiments we tested process models of semantic competition in left ventrolateral prefrontal cortex (VLPFC). On TMS stimulation of left mid-VLPFC just after presentation of an ambiguous stimulus, participants' mouse-movement trajectories deviated more toward the incorrect target for weak associate trials than for any other trial type. This effect was extinguished when participants were simultaneously shown both target and cue stimuli. Results suggest that left mid-VLPFC is necessary to resolve semantic competition when a response is underdetermined by the stimulus and the interpretive context of the stimulus is ambiguous. Computer-mouse movements reveal the dynamics of competitive interactions as they resolve, making this technique ideally suited for studying cognitive control processes and a more sensitive index of TMS disruption than reaction time and accuracy alone.