Time-domain analysis for extracting fast-paced pupil responses

From pre-processing to advanced dynamic modeling of pupil data

The pupil of the eye provides a rich source of information for cognitive scientists, as it can index a variety of bodily states (e.g., arousal, fatigue) and cognitive processes (e.g., attention, decision-making). As pupillometry becomes a more accessible and popular methodology, researchers have proposed a variety of techniques for analyzing pupil data. Here, we focus on time series-based, signal-to-signal approaches that enable one to relate dynamic changes in pupil size over time with dynamic changes in a stimulus time series, continuous behavioral outcome measures, or other participants' pupil traces. We first introduce pupillometry, its neural underpinnings, and the relation between pupil measurements and other oculomotor behaviors (e.g., blinks, saccades), to stress the importance of understanding what is being measured and what can be inferred from changes in pupillary activity. Next, we discuss possible pre-processing steps, and the contexts in which they may be necessary. Finally, we turn to signal-to-signal analytic techniques, including regression-based approaches, dynamic time-warping, phase clustering, detrended fluctuation analysis, and recurrence quantification analysis. Assumptions of these techniques, and examples of the scientific questions each can address, are outlined, with references to key papers and software packages. Additionally, we provide a detailed code tutorial that steps through the key examples and figures in this paper. Ultimately, we contend that the insights gained from pupillometry are constrained by the analysis techniques used, and that signal-to-signal approaches offer a means to generate novel scientific insights by taking into account understudied spectro-temporal relationships between the pupil signal and other signals of interest.

Pupil dilation reflects the dynamic integration of audiovisual emotional speech

Emotional speech perception is a multisensory process. When speaking with an individual we concurrently integrate the information from their voice and face to decode e.g., their feelings, moods, and emotions. However, the physiological reactions-such as the reflexive dilation of the pupil-associated to these processes remain mostly unknown. That is the aim of the current article, to investigate whether pupillary reactions can index the processes underlying the audiovisual integration of emotional signals. To investigate this question, we used an algorithm able to increase or decrease the smiles seen in a person's face or heard in their voice, while preserving the temporal synchrony between visual and auditory channels. Using this algorithm, we created congruent and incongruent audiovisual smiles, and investigated participants' gaze and pupillary reactions to manipulated stimuli. We found that pupil reactions can reflect emotional information mismatch in audiovisual speech. In our data, when participants were explicitly asked to extract emotional information from stimuli, the first fixation within emotionally mismatching areas (i.e., the mouth) triggered pupil dilation. These results reveal that pupil dilation can reflect the dynamic integration of audiovisual emotional speech and provide insights on how these reactions are triggered during stimulus perception.

Inconclusive evidence that breathing shapes pupil dynamics in humans: a systematic review

More than 50 years ago, it was proposed that breathing shapes pupil dynamics. This widespread idea is also the general understanding currently. However, there has been no attempt at synthesizing the progress on this topic since. We therefore conducted a systematic review of the literature on how breathing affects pupil dynamics in humans. We assessed the effect of breathing phase, depth, rate, and route (nose/mouth). We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, and conducted a systematic search of the scientific literature databases MEDLINE, Web of Science, and PsycInfo in November 2021. Thirty-one studies were included in the final analyses, and their quality was assessed with QualSyst. The study findings were summarized in a descriptive manner, and the strength of the evidence for each parameter was estimated following the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach. The effect of breathing phase on pupil dynamics was rated as "low" (6 studies). The effect of breathing depth and breathing rate (6 and 20 studies respectively) were rated as "very low". Breathing route was not investigated by any of the included studies. Overall, we show that there is, at best, inconclusive evidence for an effect of breathing on pupil dynamics in humans. Finally, we suggest some possible confounders to be considered, and outstanding questions that need to be addressed, to answer this fundamental question. Trial registration: This systematic review has been registered in the international prospective register of systematic reviews (PROSPERO) under the registration number: CRD42022285044.

Emotion schema effects on associative memory differ across emotion categories at the behavioural, physiological and neural level: Emotion schema effects on associative memory differs for disgust and fear

Previous behavioural and neuroimaging studies have consistently reported that memory is enhanced for associations congruent or incongruent with the structure of prior knowledge, termed as schemas. However, it remains unclear if similar effects arise with emotion-related associations, and whether they depend on the type of emotions. Here, we addressed this question using a novel face-word pair association paradigm combined with fMRI and eye-tracking techniques. In two independent studies, we demonstrated and replicated that both congruency with emotion schemas and emotion category interact to affect associative memory. Overall, memory retrieval was higher for faces from pairs congruent vs. incongruent with emotion schemas, paralleled by a greater recruitment of left inferior frontal gyrus (IFG) during successful encoding. However, emotion schema effects differed across two negative emotion categories. Disgust was remembered better than fear, and only disgust activated left IFG stronger during encoding of congruent vs. incongruent pairs, suggestive of deeper semantic processing for the associations. On the contrary, encoding of congruent fear vs. disgust-related pairs was accompanied with greater activity in right fusiform gyrus (FG), suggesting a stronger sensory processing of faces. In addition, successful memory formation for congruent disgust pairs was associated with a higher pupil dilation index related to sympathetic activation, longer gaze time on words compared to faces, and more gaze switches between paired words and faces. This was reversed for fear-related congruent pairs where the faces attracted longer gaze time (compared to words). Overall, our results provide converging evidence from behavioural, physiological, and neural measures to suggest that congruency with available emotion schemas influence memory associations in a similar manner to semantic schemas. However, these effects vary across distinct emotion categories, pointing to a differential role of semantic processing and visual attention processes in the modulation of memory by disgust and fear, respectively.

Eye Movement and Pupil Measures: A Review

Our subjective visual experiences involve complex interaction between our eyes, our brain, and the surrounding world. It gives us the sense of sight, color, stereopsis, distance, pattern recognition, motor coordination, and more. The increasing ubiquity of gaze-aware technology brings with it the ability to track gaze and pupil measures with varying degrees of fidelity. With this in mind, a review that considers the various gaze measures becomes increasingly relevant, especially considering our ability to make sense of these signals given different spatio-temporal sampling capacities. In this paper, we selectively review prior work on eye movements and pupil measures. We first describe the main oculomotor events studied in the literature, and their characteristics exploited by different measures. Next, we review various eye movement and pupil measures from prior literature. Finally, we discuss our observations based on applications of these measures, the benefits and practical challenges involving these measures, and our recommendations on future eye-tracking research directions.

Conscious awareness of motor fluidity improves performance and decreases cognitive effort in sequence learning

Motor skill learning is improved when participants are instructed to judge after each trial whether their performed movements have reached maximal fluidity. Consequently, the conscious awareness of this maximal fluidity can be classified as a genuine learning factor for motor sequences. However, it is unknown whether this effect of conscious awareness on motor learning could be mediated by the increased cognitive effort that may accompany such judgment making. The main aim of this study was to test this hypothesis in comparing two groups with, and without, the conscious awareness of the maximal fluidity. To assess the possible involvement of cognitive effort, we have recorded the pupillary dilation to the task, which is well-known to increase in proportion to cognitive effort. Results confirmed that conscious awareness indeed improved motor sequence learning of the trained sequence specifically. Pupil dilation was smaller during trained than during novel sequence performance, indicating that sequence learning decreased the cognitive cost of sequence execution. However, we found that in the group that had to judge on their maximal fluidity, pupil dilation during sequence production was smaller than in the control group, indicating that the motor improvement induced by the fluidity judgment does not involve additional cognitive effort. We discuss these results in the context of motor learning and cognitive effort theories.

Confidence controls perceptual evidence accumulation

Perceptual decisions are accompanied by feelings of confidence that reflect the likelihood that the decision was correct. Here we aim to clarify the relationship between perception and confidence by studying the same perceptual task across three different confidence contexts. Human observers were asked to categorize the source of sequentially presented visual stimuli. Each additional stimulus provided evidence for making more accurate perceptual decisions, and better confidence judgements. We show that observers' ability to set appropriate evidence accumulation bounds for perceptual decisions is strongly predictive of their ability to make accurate confidence judgements. When observers were not permitted to control their exposure to evidence, they imposed covert bounds on their perceptual decisions but not on their confidence decisions. This partial dissociation between decision processes is reflected in behaviour and pupil dilation. Together, these findings suggest a confidence-regulated accumulation-to-bound process that controls perceptual decision-making even in the absence of explicit speed-accuracy trade-offs.

Dissociable mappings of tonic and phasic pupillary features onto cognitive processes involved in mental arithmetic

Pupil size modulations have been used for decades as a window into the mind, and several pupillary features have been implicated in a variety of cognitive processes. Thus, a general challenge facing the field of pupillometry has been understanding which pupil features should be most relevant for explaining behavior in a given task domain. In the present study, a longitudinal design was employed where participants completed 8 biweekly sessions of a classic mental arithmetic task for the purposes of teasing apart the relationships between tonic/phasic pupil features (baseline, peak amplitude, peak latency) and two task-related cognitive processes including mental processing load (indexed by math question difficulty) and decision making (indexed by response times). We used multi-level modeling to account for individual variation while identifying pupil-to-behavior relationships at the single-trial and between-session levels. We show a dissociation between phasic and tonic features with peak amplitude and latency (but not baseline) driven by ongoing task-related processing, whereas baseline was driven by state-level effects that changed over a longer time period (i.e. weeks). Finally, we report a dissociation between peak amplitude and latency whereby amplitude reflected surprise and processing load, and latency reflected decision making times.

Multiple-Time-Scale Analysis of Attention as Revealed by EEG, NIRS, and Pupil Diameter Signals During a Free Recall Task: A Multimodal Measurement Approach

Attention plays a fundamental role in acquiring and understanding information. Therefore, it is useful to evaluate attention objectively in such fields as education and mental health. Aimed at extracting objective indicators of attention from physiological signals, this study examined the characteristics of electroencephalography (EEG), near-infrared spectroscopy (NIRS), and pupil diameter signals during a free recall task. The objective was to clarify the temporal characteristics of these signals in relation to attention. We used a free recall task as a cognitive task with an attentional load. The participants attempted to memorize and then recall 13 serially presented words. Our hypothesis was that the significant physiological responses should differ depending on the time scale of the attention evaluation. The physiological responses were compared on the basis of differences between success and failure to recall a word on a short time scale, in terms of the attentional state among five serial position groups on a middle time scale, and on the basis of differences between trials with many and few words recalled on a long time scale. We found that the response of each physiological signal depended on the attention in the different time-scale comparisons. (1) The P300 amplitudes of the EEG signals for the words that were recalled were significantly higher than those for the words that were not recalled. (2) Pupillary dilation differed significantly depending on the serial position group. (3) Functional connectivity in the right hemisphere revealed by NIRS was significantly stronger in trials with many words recalled than in those with few words recalled. Different temporal characteristics of physiological signals with respect to attention were suggested by multimodal measurement and multiple-time-scale analysis. Consideration of these characteristics should help in the development of applications requiring objective attention evaluation.

Pupil-Linked Arousal Responds to Unconscious Surprisal

Pupil size under constant illumination reflects brain arousal state, and dilates in response to novel information, or surprisal. Whether this response can be observed regardless of conscious perception is still unknown. In the present study, male and female adult humans performed an implicit learning task across a series of three experiments. We measured pupil and brain-evoked potentials to stimuli that violated transition statistics but were not relevant to the task. We found that pupil size dilated following these surprising events, in the absence of awareness of transition statistics, and only when attention was allocated to the stimulus. These pupil responses correlated with central potentials, evoking an anterior cingulate origin. Arousal response to surprisal outside the scope of conscious perception points to the fundamental relationship between arousal and information processing and indicates that pupil size can be used to track the progression of implicit learning.SIGNIFICANCE STATEMENT Pupil size dilates following increase in mental effort, surprise, or more generally global arousal. However, whether this response arises as a conscious response or reflects a more fundamental mechanism outside the scrutiny of awareness is still unknown. Here, we demonstrate that unexpected changes in the environment, even when processed unconsciously and without being relevant to the task, lead to an increase in arousal levels as reflected by the pupillary response. Further, we show that the concurrent electrophysiological response shares similarities with mismatch negativity, suggesting the involvement of anterior cingulate cortex. All in all, our results establish novel insights about the mechanisms driving global arousal levels, and it provides new possibilities for reliably measuring unconscious processes.

A Cross-Recurrence Analysis of the Pupil Size Fluctuations in Steady Scotopic Conditions

Pupil size fluctuations during stationary scotopic conditions may convey information about the cortical state activity at rest. An important link between neuronal network state modulation and pupil fluctuations is the cholinergic and noradrenergic neuromodulatory tone, which is active at cortical level and in the peripheral terminals of the autonomic nervous system (ANS). This work aimed at studying the low- and high-frequency coupled oscillators in the autonomic spectrum (0–0.45 Hz) which, reportedly, drive the spontaneous pupillary fluctuations. To assess the interaction between the oscillators, we focused on the patterns of their trajectories in the phase-space. Firstly, the frequency spectrum of the pupil signal was determined by empirical mode decomposition. Secondly, cross-recurrence quantification analysis was used to unfold the non-linear dynamics. The global and local patterns of recurrence of the trajectories were estimated by two parameters: determinism and entropy. An elliptic region in the entropy-determinism plane (95% prediction area) yielded health-related values of entropy and determinism. We hypothesize that the data points inside the ellipse would likely represent balanced activity in the ANS. Interestingly, the Epworth Sleepiness Scale scores scaled up along with the entropy and determinism parameters. Although other non-linear methods like Short Time Fourier Transform and wavelets are usually applied for analyzing the pupillary oscillations, they rely on strong assumptions like the stationarity of the signal or the a priori knowledge of the shape of the single basis wave. Instead, the cross-recurrence analysis of the non-linear dynamics of the pupil size oscillations is an adaptable diagnostic tool for identifying the different weight of the autonomic nervous system components in the modulation of pupil size changes at rest in non-luminance conditions.

Locus Coeruleus atrophy doesn't relate to fatigue in Parkinson's disease

Fatigue is a frequent complaint among healthy population and one of the earliest and most debilitating symptoms in Parkinson's disease (PD). Earlier studies have examined the role of dopamine and serotonin in pathogenesis of fatigue, but the plausible role of noradrenalin (NA) remains underexplored. We investigated the relationship between fatigue in Parkinsonian patients and the extent of degeneration of Locus Coeruleus (LC), the main source of NA in the brain. We quantified LC and Substantia Nigra (SN) atrophy using neuromelanin-sensitive imaging, analyzed with a novel, fully automated algorithm. We also assessed patients' fatigue, depression, sleep disturbance and vigilance. We found that LC degeneration correlated with the levels of depression and vigilance but not with fatigue, while fatigue correlated weakly with atrophy of SN. These results indicate that LC degeneration in Parkinson's disease is unlikely to cause fatigue, but may be involved in mood and vigilance alterations.