1
|
Spiering L, Dimigen O. (Micro)saccade-related potentials during face recognition: A study combining EEG, eye-tracking, and deconvolution modeling. Atten Percept Psychophys 2024:10.3758/s13414-024-02846-1. [PMID: 38296873 DOI: 10.3758/s13414-024-02846-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/02/2024] [Indexed: 02/02/2024]
Abstract
Under natural viewing conditions, complex stimuli such as human faces are typically looked at several times in succession, implying that their recognition may unfold across multiple eye fixations. Although electrophysiological (EEG) experiments on face recognition typically prohibit eye movements, participants still execute frequent (micro)saccades on the face, each of which generates its own visuocortical response. This finding raises the question of whether the fixation-related potentials (FRPs) evoked by these tiny gaze shifts also contain psychologically valuable information about face processing. Here, we investigated this question by corecording EEG and eye movements in an experiment with emotional faces (happy, angry, neutral). Deconvolution modeling was used to separate the stimulus ERPs to face onset from the FRPs generated by subsequent microsaccades-induced refixations on the face. As expected, stimulus ERPs exhibited typical emotion effects, with a larger early posterior negativity (EPN) for happy/angry compared with neutral faces. Eye tracking confirmed that participants made small saccades in 98% of the trials, which were often aimed at the left eye of the stimulus face. However, while each saccade produced a strong response over visual areas, this response was unaffected by the face's emotional expression, both for the first and for subsequent (micro)saccades. This finding suggests that the face's affective content is rapidly evaluated after stimulus onset, leading to only a short-lived sensory enhancement by arousing stimuli that does not repeat itself during immediate refixations. Methodologically, our work demonstrates how eye tracking and deconvolution modeling can be used to extract several brain responses from each EEG trial, providing insights into neural processing at different latencies after stimulus onset.
Collapse
Affiliation(s)
- Lisa Spiering
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Olaf Dimigen
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany.
- Department of Psychology, University of Groningen, Grote Kruisstraat 2/1, 9712 TS, Groningen, The Netherlands.
| |
Collapse
|
2
|
Nikolaev AR, Meghanathan RN, van Leeuwen C. Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates. Atten Percept Psychophys 2024:10.3758/s13414-023-02836-9. [PMID: 38169029 DOI: 10.3758/s13414-023-02836-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/17/2023] [Indexed: 01/05/2024]
Abstract
When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.
Collapse
Affiliation(s)
- Andrey R Nikolaev
- Department of Psychology, Lund University, Box 213, 22100, Lund, Sweden.
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium.
| | | | - Cees van Leeuwen
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium
- Center for Cognitive Science, Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau, Kaiserslautern, Germany
| |
Collapse
|
3
|
Wascher E, Reiser J, Rinkenauer G, Larrá M, Dreger FA, Schneider D, Karthaus M, Getzmann S, Gutberlet M, Arnau S. Neuroergonomics on the Go: An Evaluation of the Potential of Mobile EEG for Workplace Assessment and Design. HUMAN FACTORS 2023; 65:86-106. [PMID: 33861182 PMCID: PMC9846382 DOI: 10.1177/00187208211007707] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/15/2021] [Accepted: 03/13/2021] [Indexed: 06/12/2023]
Abstract
OBJECTIVE We demonstrate and discuss the use of mobile electroencephalogram (EEG) for neuroergonomics. Both technical state of the art as well as measures and cognitive concepts are systematically addressed. BACKGROUND Modern work is increasingly characterized by information processing. Therefore, the examination of mental states, mental load, or cognitive processing during work is becoming increasingly important for ergonomics. RESULTS Mobile EEG allows to measure mental states and processes under real live conditions. It can be used for various research questions in cognitive neuroergonomics. Besides measures in the frequency domain that have a long tradition in the investigation of mental fatigue, task load, and task engagement, new approaches-like blink-evoked potentials-render event-related analyses of the EEG possible also during unrestricted behavior. CONCLUSION Mobile EEG has become a valuable tool for evaluating mental states and mental processes on a highly objective level during work. The main advantage of this technique is that working environments don't have to be changed while systematically measuring brain functions at work. Moreover, the workflow is unaffected by such neuroergonomic approaches.
Collapse
Affiliation(s)
- Edmund Wascher
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Julian Reiser
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Gerhard Rinkenauer
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Mauro Larrá
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Felix A. Dreger
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Daniel Schneider
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Melanie Karthaus
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | - Stephan Getzmann
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| | | | - Stefan Arnau
- IfADo – Leibniz Research Centre for Working Environment and
Human Factors, Dortmund, Germany
| |
Collapse
|
4
|
Synchronization of acquisition devices in neuroimaging: An application using co-registration of eye movements and electroencephalography. Behav Res Methods 2021; 54:2545-2564. [PMID: 34918232 DOI: 10.3758/s13428-021-01756-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/22/2021] [Indexed: 11/08/2022]
Abstract
Interest in applications for the simultaneous acquisition of data from different devices is growing. In neuroscience for example, co-registration complements and overcomes some of the shortcomings of individual methods. However, precise synchronization of the different data streams involved is required before joint data analysis. Our article presents and evaluates a synchronization method which maximizes the alignment of information across time. Synchronization through common triggers is widely used in all existing methods, because it is very simple and effective. However, this solution has been found to fail in certain practical situations, namely for the spurious detection of triggers and/or when the timestamps of triggers sampled by each acquisition device are not jointly distributed linearly for the entire duration of an experiment. We propose two additional mechanisms, the "Longest Common Subsequence" algorithm and a piecewise linear regression, in order to overcome the limitations of the classical method of synchronizing common triggers. The proposed synchronization method was evaluated using both real and artificial data. Co-registrations of electroencephalographic signals (EEG) and eye movements were used for real data. We compared the effectiveness of our method to another open source method implemented using EYE-EEG toolbox. Overall, we show that our method, implemented in C++ as a DOS application, is very fast, robust and fully automatic.
Collapse
|
5
|
Stankov AD, Touryan J, Gordon S, Ries AJ, Ki J, Parra LC. During natural viewing, neural processing of visual targets continues throughout saccades. J Vis 2021; 21:7. [PMID: 34491271 PMCID: PMC8431980 DOI: 10.1167/jov.21.10.7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Relatively little is known about visual processing during free-viewing visual search in realistic dynamic environments. Free-viewing is characterized by frequent saccades. During saccades, visual processing is thought to be suppressed, yet we know that the presaccadic visual content can modulate postsaccadic processing. To better understand these processes in a realistic setting, we study here saccades and neural responses elicited by the appearance of visual targets in a realistic virtual environment. While subjects were being driven through a 3D virtual town, they were asked to discriminate between targets that appear on the road. Using a system identification approach, we separated overlapping and correlated activity evoked by visual targets, saccades, and button presses. We found that the presence of a target enhances early occipital as well as late frontocentral saccade-related responses. The earlier potential, shortly after 125 ms post-saccade onset, was enhanced for targets that appeared in the peripheral vision as compared to the central vision, suggesting that fast peripheral processing initiated before saccade onset. The later potential, at 195 ms post-saccade onset, was strongly modulated by the visibility of the target. Together these results suggest that, during natural viewing, neural processing of the presaccadic visual stimulus continues throughout the saccade, apparently unencumbered by saccadic suppression.
Collapse
Affiliation(s)
- Atanas D Stankov
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| | - Jonathan Touryan
- U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA.,
| | | | - Anthony J Ries
- U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA.,
| | - Jason Ki
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| | - Lucas C Parra
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| |
Collapse
|
6
|
Characteristics of the Deconvolved Transient AEP from 80 Hz Steady-State Responses to Amplitude Modulation Stimulation. J Assoc Res Otolaryngol 2021; 22:741-753. [PMID: 34415469 DOI: 10.1007/s10162-021-00806-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Accepted: 07/02/2021] [Indexed: 10/20/2022] Open
Abstract
This study aimed to validate the existence and investigate the characteristics of the transient responses from conventional auditory steady-state responses (ASSRs) using deconvolution methods capable of dealing with amplitude modulated (AM) stimulation. Conventional ASSRs to seven stimulus rates were recorded from 17 participants. A deconvolution method was selected and modified to accommodate the AM stimulation. The calculated responses were examined in terms of temporal features with respect to different combinations of stimulus rates. Stable transient responses consisting of early stage brainstem responses and middle latency responses were reconstructed consistently for all rate combinations, which indicates that the superposition hypothesis is applicable to the generation of approximately 80 Hz ASSRs evoked by AM tones (AM-ASSRs). The new transient responses are characterized by three pairs of peak-troughs named as n0p0, n1p1, and n2p2 within 40 ms. Compared with conventional ABR-MLRs, the n0p0 indicates the first neural activity where p0 might represent the main ABR components; the n1 is the counterpart of N10; the p2 is corresponding to the robust Pa at about 30 ms; the p1 and n2 are absent of real counterparts. The peak-peak amplitudes show a slight decrease with increasing stimulation rate from 75 to 95 Hz whereas the peak latencies change differently, which is consistent with the known rate-effect on AEPs. This is direct evidence for a transient response derived from AM-ASSRs for the first time. The characteristic components offer insight into the constitution of AM-ASSRs and may be promising in clinical applications and fundamental studies.
Collapse
|
7
|
Dimigen O, Ehinger BV. Regression-based analysis of combined EEG and eye-tracking data: Theory and applications. J Vis 2021; 21:3. [PMID: 33410892 PMCID: PMC7804566 DOI: 10.1167/jov.21.1.3] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2019] [Accepted: 08/14/2020] [Indexed: 12/27/2022] Open
Abstract
Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.
Collapse
Affiliation(s)
- Olaf Dimigen
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany
| | - Benedikt V Ehinger
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
8
|
Carter BT, Luke SG. Best practices in eye tracking research. Int J Psychophysiol 2020; 155:49-62. [PMID: 32504653 DOI: 10.1016/j.ijpsycho.2020.05.010] [Citation(s) in RCA: 61] [Impact Index Per Article: 15.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2019] [Revised: 05/26/2020] [Accepted: 05/27/2020] [Indexed: 12/14/2022]
Abstract
This guide describes best practices in using eye tracking technology for research in a variety of disciplines. A basic outline of the anatomy and physiology of the eyes and of eye movements is provided, along with a description of the sorts of research questions eye tracking can address. We then explain how eye tracking technology works and what sorts of data it generates, and provide guidance on how to select and use an eye tracker as well as selecting appropriate eye tracking measures. Challenges to the validity of eye tracking studies are described, along with recommendations for overcoming these challenges. We then outline correct reporting standards for eye tracking studies.
Collapse
|
9
|
Alexander RG, Macknik SL, Martinez-Conde S. Microsaccades in Applied Environments: Real-World Applications of Fixational Eye Movement Measurements. J Eye Mov Res 2020; 12:10.16910/jemr.12.6.15. [PMID: 33828760 PMCID: PMC7962687 DOI: 10.16910/jemr.12.6.15] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022] Open
Abstract
Across a wide variety of research environments, the recording of microsaccades and other fixational eye movements has provided insight and solutions into practical problems. Here we review the literature on fixational eye movements-especially microsaccades-in applied and ecologically-valid scenarios. Recent technical advances allow noninvasive fixational eye movement recordings in real-world contexts, while observers perform a variety of tasks. Thus, fixational eye movement measures have been obtained in a host of real-world scenarios, such as in connection with driver fatigue, vestibular sensory deprivation in astronauts, and elite athletic training, among others. Here we present the state of the art in the practical applications of fixational eye movement research, examine its potential future uses, and discuss the benefits of including microsaccade measures in existing eye movement detection technologies. Current evidence supports the inclusion of fixational eye movement measures in real-world contexts, as part of the development of new or improved oculomotor assessment tools. The real-world applications of fixational eye movement measurements will only grow larger and wider as affordable high-speed and high-spatial resolution eye trackers become increasingly prevalent.
Collapse
|
10
|
Pfeiffer C, Hollenstein N, Zhang C, Langer N. Neural dynamics of sentiment processing during naturalistic sentence reading. Neuroimage 2020; 218:116934. [PMID: 32416227 DOI: 10.1016/j.neuroimage.2020.116934] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2019] [Revised: 04/24/2020] [Accepted: 05/07/2020] [Indexed: 12/15/2022] Open
Abstract
When we read, our eyes move through the text in a series of fixations and high-velocity saccades to extract visual information. This process allows the brain to obtain meaning, e.g., about sentiment, or the emotional valence, expressed in the written text. How exactly the brain extracts the sentiment of single words during naturalistic reading is largely unknown. This is due to the challenges of naturalistic imaging, which has previously led researchers to employ highly controlled, timed word-by-word presentations of custom reading materials that lack ecological validity. Here, we aimed to assess the electrical neural correlates of word sentiment processing during naturalistic reading of English sentences. We used a publicly available dataset of simultaneous electroencephalography (EEG), eye-tracking recordings, and word-level semantic annotations from 7129 words in 400 sentences (Zurich Cognitive Language Processing Corpus; Hollenstein et al., 2018). We computed fixation-related potentials (FRPs), which are evoked electrical responses time-locked to the onset of fixations. A general linear mixed model analysis of FRPs cleaned from visual- and motor-evoked activity showed a topographical difference between the positive and negative sentiment condition in the 224-304 ms interval after fixation onset in left-central and right-posterior electrode clusters. An additional analysis that included word-, phrase-, and sentence-level sentiment predictors showed the same FRP differences for the word-level sentiment, but no additional FRP differences for phrase- and sentence-level sentiment. Furthermore, decoding analysis that classified word sentiment (positive or negative) from sentiment-matched 40-trial average FRPs showed a 0.60 average accuracy (95% confidence interval: [0.58, 0.61]). Control analyses ruled out that these results were based on differences in eye movements or linguistic features other than word sentiment. Our results extend previous research by showing that the emotional valence of lexico-semantic stimuli evoke a fast electrical neural response upon word fixation during naturalistic reading. These results provide an important step to identify the neural processes of lexico-semantic processing in ecologically valid conditions and can serve to improve computer algorithms for natural language processing.
Collapse
Affiliation(s)
- Christian Pfeiffer
- Methods of Plasticity Research Laboratory, Department of Psychology, University of Zurich, Switzerland; University Research Priority Program (URPP) Dynamics of Healthy Aging, Zurich, Switzerland.
| | | | - Ce Zhang
- Department of Computer Science, ETH, Zurich, Switzerland
| | - Nicolas Langer
- Methods of Plasticity Research Laboratory, Department of Psychology, University of Zurich, Switzerland; University Research Priority Program (URPP) Dynamics of Healthy Aging, Zurich, Switzerland; Neuroscience Center Zurich (ZNZ), Zurich, Switzerland
| |
Collapse
|
11
|
de Lissa P, McArthur G, Hawelka S, Palermo R, Mahajan Y, Degno F, Hutzler F. Peripheral preview abolishes N170 face-sensitivity at fixation: Using fixation-related potentials to investigate dynamic face processing. VISUAL COGNITION 2019. [DOI: 10.1080/13506285.2019.1676855] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Affiliation(s)
- Peter de Lissa
- iBMLab, Department of Psychology, University of Fribourg, Fribourg, Switzerland
- Department of Cognitive Science, Macquarie University, Sydney, Australia
| | - Genevieve McArthur
- Department of Cognitive Science, Macquarie University, Sydney, Australia
| | - Stefan Hawelka
- Centre for Cognitive Neuroscience, Salzburg University, Salzburg, Austria
| | - Romina Palermo
- School of Psychological Science, University of Western Australia, Perth, Australia
| | - Yatin Mahajan
- The MARCS Institute, University of Western Sydney, Australia
| | - Federica Degno
- School of Psychology, University of Central Lancashire, Preston, UK
| | - Florian Hutzler
- Centre for Cognitive Neuroscience, Salzburg University, Salzburg, Austria
| |
Collapse
|