Cortical potentials associated with the detection of visual events

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

The fixation and saccade P3

Although most instances of object recognition during natural viewing occur in the presence of saccades, the neural correlates of objection recognition have almost exclusively been examined during fixation. Recent studies have indicated that there are post-saccadic modulations of neural activity immediately following eye movement landing; however, whether post-saccadic modulations affect relatively late occurring cognitive components such as the P3 has not been explored. The P3 as conventionally measured at fixation is commonly used in brain computer interfaces, hence characterizing the post-saccadic P3 could aid in the development of improved brain computer interfaces that allow for eye movements. In this study, the P3 observed after saccadic landing was compared to the P3 measured at fixation. No significant differences in P3 start time, temporal persistence, or amplitude were found between fixation and saccade trials. Importantly, sensory neural responses canceled in the target minus distracter comparisons used to identify the P3. Our results indicate that relatively late occurring cognitive neural components such as the P3 are likely less sensitive to post saccadic modulations than sensory neural components and other neural activity occurring shortly after eye movement landing. Furthermore, due to the similarity of the fixation and saccade P3, we conclude that the P3 following saccadic landing could possibly be used as a viable signal in brain computer interfaces allowing for eye movements.

Neural saccadic response estimation during natural viewing

Studying neural activity during natural viewing conditions is not often attempted. Isolating the neural response of a single saccade is necessary to study neural activity during natural viewing; however, the close temporal spacing of saccades that occurs during natural viewing makes it difficult to determine the response to a single saccade. Herein, a general linear model (GLM) approach is applied to estimate the EEG neural saccadic response for different segments of the saccadic main sequence separately. It is determined that, in visual search conditions, neural responses estimated by conventional event-related averaging are significantly and systematically distorted relative to GLM estimates due to the close temporal spacing of saccades during visual search. Before the GLM is applied, analyses are applied that demonstrate that saccades during visual search with intersaccadic spacings as low as 100-150 ms do not exhibit significant refractory effects. Therefore, saccades displaying different intersaccadic spacings during visual search can be modeled using the same regressor in a GLM. With the use of the GLM approach, neural responses were separately estimated for five different ranges of saccade amplitudes during visual search. Occipital responses time locked to the onsets of saccades during visual search were found to account for, on average, 79 percent of the variance of EEG activity in a window 90-200 ms after the onsets of saccades for all five saccade amplitude ranges that spanned a range of 0.2-6.0 degrees. A GLM approach was also used to examine the lateralized ocular artifacts associated with saccades. Possible extensions of the methods presented here to account for the superposition of microsaccades in event-related EEG studies conducted in nominal fixation conditions are discussed.

P300 and traffic scenes: the effect of temazepam

The present research investigated the effects of a minor tranquillizer (temazepam) on P300 in a paradigm that may be relevant for traffic behaviour. Because accident scenes have not been used previously in P300 research, Experiment 1 (n = 8) examined whether the P300 elicited by safe traffic scenes and scenes of imminent road accidents were sensitive to the probability of occurrence. Event-related potentials were recorded from C3, Cz, C4, P3, Pz and P4 within an oddball paradigm. The type of stimulus to which subjects responded (pictures of imminent accidents or safe road scenes) was crossed with the probability (0.1 or 0.5) of the relevant (to which a response was required) event. The results indicated that P300 amplitude increased with decreasing probability of the relevant stimulus and that P300 was most pronounced at Pz. Experiment 2 (n = 12) employed a drug treatment (10 mg temazepam) and a placebo treatment (100 mg Vitamin E). An oddball paradigm with a probability of the relevant stimulus of 0.1 was used and P300 was recorded from Cz, C3, C4, Pz, P3 and P4. Generally, the ingestion of temazepam decreased P300 amplitude and increased P300 latency at all sites. Reaction time, on the other hand, was not influenced by drug administration. The data demonstrate the clear effect of minor tranquillizers on the psychological processes associated with P300.

P300 and psychophysiological analysis of the structure of behavior

P300 wave was analyzed on the basis of comparison between brain potentials, the structure of behavior and single unit activity within the signal detection task through experiments with humans and animals. The similarity of the late positive components following detected light flashes in humans and rabbits, identified as P300, permitted the use of data concerning single unit activity. The analysis of single unit activity in the visual and motor cortex of rabbits demonstrated that the sequence of behavioral stages in signal detection is based on different sets of units. There is a strong correspondence between the change of set and the P300 development. Since the activity of certain units in behavior reflects the realization of certain functional systems, the modification of the sets of activated units indicates the transformation of the composition of functional systems of different hierarchical levels during the change of behavioral stages. This is the basis for our suggestion that P300 reflects the process of reorganization of the structure of the behavior which is responsible for the transition from one stage of behavior to another.

Saccade-related brain potentials in semantic categorization tasks

Saccade-related brain potentials (SRPs) were recorded in word categorization tasks in which subjects had to perform a saccade in order to perceive the stimulus. For all three conditions representing different degrees of complexity of semantic categorization, the stimuli belonged to one of two categories which appeared with the respective probabilities of either 0.20 or 0.80. The late positivity (P4) of the SRPs to infrequent stimuli appeared systematically later as the complexity of stimulus evaluation increased: The easiest categorization was accompanied by a P4 at 400 msec, in the more complex condition it peaked at 600 msec, and in the most difficult semantic categorization the P4 peaked even later, at 680 msec. This shift in peak latency with increasing complexity of categorization is in agreement with the results for traditional ERPs (e.g. Kutas and Donchin, 1978). The possible overlap of the late components was investigated by applying Principal Component-Varimax Analysis to the SRPs.

Late components of lambda responses in cognitive tasks

The effects of stimulus probability were investigated on the late components of saccade-related brain potentials (lambda responses) in two paradigms, 'counting' and 'guessing.' Subjects performed saccadic eye movements from a middle light to the target area where one of three kinds of target light stimuli appeared in a randomized sequence, each with a different probability (0.17, 0.33, and 0.50). Brain activity was recorded from six scalp areas. In the counting task subjects were instructed to count the prespecified low-probability stimuli. In the guessing experiment they had to make a guess before each trial as to which kind of target stimulus would appear. In both experiments the late positive components showed significantly greater amplitude in the lambda responses associated with infrequent stimuli than in those elicited by frequent stimuli. Principal component-Varimax analysis of the lambda responses revealed factors corresponding to the P300 and the slow wave components of traditional evoked potential studies.

ERPs associated with decisions based on slowly increasing warm and painful stimuli

The event-related potential (ERP), produced by subjects who were experiencing a gradually increasing heat stimulus, was recorded 3.5 sec before and 1 sec after a motor response. Subjects responded when the stimulus was 'just warm,' 'just at prickling pain' or 'above the temperature of just prickling pain.' Two control procedures mimicked the readiness potential (RP) and motor potential (MP) experiments. The results showed that the steadily rising negative MP was modified to a positive wave before the decision of 'warm.' During the 'pain' and 'above pain' conditions, the pre-decision slow wave was less negative than the MP but less positive than that of the 'warm' condition. A temperature control condition confirmed that rising temperature alone did not produce these results. Principal components analysis confirmed these differences by producing factors which were interpreted as representing the ERPs of the conditions of 'warm,' 'pain,' 'above pain,' RP and MP. It was evident from pilot studies that uncontrolled electrical activity from the skin potential response (SPR) was contaminating the results. SPR forced all ERP curves in the positive direction and several attempts to eliminate the SPR from the ERP data failed. While the SPR was produced differentially across conditions and across the experimental session, the smaller the variability in the electro-oculogram (EOG) recorded from above and below the orbits, the smaller was the SPR contamination that we recorded in the ERP. Therefore, we selected subjects for this experiment who showed a stable orbital EOG.

Late components of saccade-related brain potentials in guessing tasks

The late positive components of lambda responses were studied in a guessing task modified in such a way that subjects had to perform a saccadic eye movement in order to perceive task-relevant information. Responses from 6 scalp areas were investigated in 9 subjects and in 2 conditions: control and guessing. In both conditions subjects performed two consecutive saccades in a given trial: the first to the middle light, the second to the target area. In the control condition the subjects knew beforehand what the 'target' would be. In the guessing condition they had to make a guess before each trial as to which of the three target stimuli would appear. The target stimuli occurred with unequal probability and were presented in a randomized sequence. Unlike the control condition, the guessing task led to the appearance of a late positive component in the lambda response. Similar to traditional ERP findings, this late positivity showed an amplitude maximum at the parietal area and a peak latency at 375 msec from saccade onset. Furthermore, Principal Component-Varimax Analysis (PCVA) of the lambda responses revealed a first factor giving the strongest loading in the latency range of the P300, and a second factor which was identified as the slow wave. These factors are quite similar to the factors found in the PCVA of ERPs. Our results suggest that the late components of lambda responses reflect the effects of information processing in cognitive tasks similarly to the way the late positive components of ERPs do.

A no-stimulus, no-response, event-related potential of the human cortex

Brain macropotentials related to self-paced uni- or bimanual, unskilled or skilled movements were recorded from the motosensory cortical areas from 26 subjects. EMG of the active muscles, respiration, GSR, ECG and potentials generated by eye movements were also recorded. Subjects triggered the sweep of an oscilloscope with self-paced thumb movements in the following, unskilled, conditions: left-hand press (LHP), right-hand press (RHP), pressing with both hands simultaneously. They then performed a skilled performance task (SPT) during which the sweep was initiated by an LHP and was terminated within 40--60 msec by an RHP. Ten of the tested subjects participated in a variation of the skilled performance task during which no visual feedback was provided about their performance. All movements were preceded by sustained negativities known as N, component or Bereitschaftspotential (BP). All actions were followed by the motor cortex potential (MCP) with a latency of 32 msec from the EMG peak and a positivity with a latency of 200 msec from movement execution and initiation of the oscilloscope trace. Only the movements during SPT were followed by an additional positivity with a latency of 460 msec from the movement, having a centroparietal distribution and a mean amplitude at the vertex of 12.7 muV and S.D. of 5.0 muV. This skilled performance positivity (SPP) was independent of the amplitude of the previous BP and MCP and it was absent when the subject executed a variation of the SPT test during which visual feedback was not provided. Thus the SPP potential seems to be independent of stimulus or response characteristics and develops under experimental circumstances in which the subject expects and receives information about the outcome of coordinated self-placed movements.