Extended testing with the dot-probe task increases test-retest reliability and validity

Reliability (or lack thereof) of smartphone ecological momentary assessment of visual dot probe attention bias toward threat indices

BACKGROUND AND OBJECTIVES

Cognitive bias theories posit that generalized anxiety disorder (GAD) and social anxiety disorder (SAD) are entwined with attention bias toward threats, commonly indexed by faster response time (RT) on threat-congruent (vs. threat-incongruent) trials on the visual dot probe. Moreover, although smartphone ecological momentary assessment (EMA) of the visual dot probe has been developed, their psychometric properties are understudied. This study thus aimed to assess the reliability of 8 smartphone-delivered visual dot probe attention bias and related indices in persons with and without GAD and SAD.

METHODS

Community-dwelling adults (n = 819; GAD: 64%; SAD: 49%; Mixed GAD and SAD: 37%; Non-GAD/SAD Controls: 24%) completed a five-trial smartphone-delivered visual dot probe for a median of 60 trials (12 sessions x 5 trials/session) and an average of 100 trials (20 sessions x 5 trials/session).

RESULTS

As hypothesized, Global Attention Bias Index, Disengagement Effect, and Facilitation Bias had low-reliability estimates. However, retest-reliability and internal reliability were good for Trial-Level Bias Scores (TLBS) (Bias Toward Treat: intra-class correlation coefficients (ICCs) = 0.626-0.644; split-half r = 0.640-0.670; Attention Bias Variability: ICCs = 0.507-0.567; split-half r = 0.520-0.580) and (In)congruent RTs. Poor retest-reliability and internal reliability estimates were consistently observed for all traditional attention bias and related indices but not TLBS.

LIMITATIONS

Our visual dot probe EMA should have administered ≥320 trials to match best-practice guidelines based on similar laboratory studies.

CONCLUSIONS

Future research should strive to examine attention bias paradigms beyond the dot-probe task that evidenced meaningful test-retest reliability properties in laboratory and real-world naturalistic settings.

Negative affective priming: Reliability and associations with depression symptoms in three samples

The negative affective priming (NAP) task is a behavioral measure of inhibition of emotional stimuli. Previous studies using the NAP task have found that individuals with depression show reduced inhibition of negative stimuli, suggesting that inhibition biases may play a role in the etiology and maintenance of depression. However, the psychometric properties of the NAP task have not been evaluated or reported. In the present study, we report data on the association between NAP task performance and depression symptoms in three independent samples, and we evaluate the internal consistency and test-retest reliability of the NAP effect indices. The NAP effect for both negative and positive target words had poor internal consistency in all three samples, as well as poor 2-week (Study 2) and 6-month (Study 3) test-retest reliability. The internal consistency and test-retest reliability of response times (RT) for the individual trial types were moderate to high, as were the intercorrelations between trial types. This pattern of results indicates that overall RT is reliable but variance in RTs for the different trial types in the NAP task is indistinguishable from variance in overall RT. Depression symptom severity was not associated with the NAP effect for negative or positive target words in any of the samples, which could be due to the poor reliability of the NAP effect. Based on these findings, we do not recommend that researchers use the NAP task as a measure of individual differences in the inhibition of emotional stimuli.

Improving the Reliability of Cognitive Task Measures: A Narrative Review

Cognitive tasks are capable of providing researchers with crucial insights into the relationship between cognitive processing and psychiatric phenomena. However, many recent studies have found that task measures exhibit poor reliability, which hampers their usefulness for individual differences research. Here, we provide a narrative review of approaches to improve the reliability of cognitive task measures. Specifically, we introduce a taxonomy of experiment design and analysis strategies for improving task reliability. Where appropriate, we highlight studies that are exemplary for improving the reliability of specific task measures. We hope that this article can serve as a helpful guide for experimenters who wish to design a new task, or improve an existing one, to achieve sufficient reliability for use in individual differences research.

Investigating the Reliability of Pupillometry as a Measure of Individualized Listening Effort

Recordings of the pupillary response have been used in numerous studies to assess listening effort during a speech-in-noise task. Most studies focused on averaged responses across listeners, whereas less is known about pupil dilation as an indicator of the individuals’ listening effort. The present study investigated the reliability of several pupil features as potential indicators of individual listening effort and the impact of different normalization procedures on the reliability. The pupil diameters of 31 normal-hearing listeners were recorded during multiple visits while performing a speech-in-noise task. The signal-to-noise ratios (SNRs) of the stimuli ranged from −12 dB to +4 dB. All listeners were measured twice at separate visits, and 11 were re-tested at a third visit. To examine the reliability of the pupil responses across visits, the intraclass correlation coefficient was applied to the peak and mean pupil dilation and to the temporal features of the pupil response, extracted using growth curve analysis. The reliability of the pupillary response was assessed in relation to SNR and different normalization procedures over multiple visits. The most reliable pupil features were the traditional mean and peak pupil dilation. The highest reliability results were obtained when the data were baseline-corrected and normalized to the individual pupil response range across all visits. Moreover, the present study results showed only a minor impact of the SNR and the number of visits on the reliability of the pupil response. Overall, the results may provide an important basis for developing a standardized test for pupillometry in the clinic.

Cognitive biases in military personnel with and without PTSD: a systematic review

BACKGROUND

Some cognitive biases, such as excessive attention to threat, are associated with PTSD. However, they may be adaptive for military personnel; attending to threat may improve safety for deployed personnel.

AIMS

The extent to which military personnel with vs. without PTSD differ with respect to specific cognitive biases is currently unclear. This systematic review aimed to address this question.

METHODS

PRISMA guidelines were followed. Articles were identified using a comprehensive literature search; 21 studies (with 1977 participants) were reviewed.

RESULTS

All studies were of "moderate" or "strong" quality. Military personnel with vs. without PTSD used overgeneralised language when describing autobiographical memories and demonstrated impaired performance on a modified Stroop task. Studies using dot-probe paradigms conceptualised attentional response as a dynamic process, fluctuating between bias towards and away from threat; military personnel with vs. without PTSD demonstrated greater fluctuation. Studies using visual search tasks concluded that attentional bias in PTSD involves interference (difficulty disengaging from threat) rather than facilitation (enhanced threat detection). Finally, personnel with vs. without PTSD demonstrated interpretation bias, completing ambiguous sentences with negative rather than neutral endings.

CONCLUSION

The implications for military populations and recommendations for further research and clinical practice are considered.

PROSPERO REGISTRATION

PROSPERO 2018 CRD42018092235.

Reliability of attention bias and attention bias variability to climate change images in the dot-probe task

Climate change is one of the most pressing issues of the 21st century, which is perhaps why information about climate change has been found to capture observers' attention. One of the most common ways of assessing individual differences in attentional processing of climate change information is through the use of reaction time difference scores. However, reaction time-based difference scores have come under scrutiny for their low reliability. Given that a primary goal of the field is to link individual differences in attention processing to participant variables (e.g., environmental attitudes), we assessed the reliability of reaction time-based measures of attention processing of climate change information utilizing an existing dataset with three variations of the dot-probe task. Across all three samples, difference score-based measures of attentional bias were generally uncorrelated across task blocks (r = -0.25 to 0.31). We also assessed the reliability of newer attention bias variability measures that are thought to capture dynamic shifts in attention toward and away from salient information. Although these measures were initially found to be correlated across task blocks (r = 0.17-0.67), they also tended to be highly correlated with general reaction time variability (r = 0.49-0.83). When controlling for general reaction time variability, the correlations across task blocks for attention bias variability were much weaker and generally nonsignificant (r = -0.25 to 0.33). Furthermore, these measures were unrelated to pro-environmental disposition indicating poor predictive validity. In short, reaction time-based measures of attentional processing (including difference score and variability-based approaches) have unacceptably low levels of reliability and are therefore unsuitable for capturing individual differences in attentional bias to climate change information.

Eye-size effects in the dot-probe task: Greater sclera exposure predicts delayed disengagement from fearful faces

Fearful facial expressions are nonverbal and biologically salient signals of potential threat that automatically hold, capture, and direct observers' attention. They are characterized by enlarged eye whites and dilated pupils, and fearful eyes alone are sufficient to capture attention. The morphological properties of the eye region, such as sclera exposure, are thought to play an important role in nonverbal communication. Specifically, increased sclera exposure associated with fearful expressions has been shown to moderate how observers' shift their attention toward the direction of another's gaze. Yet, the extent to which variability in sclera exposure possibly impacts the capture and hold of attention by fearful faces is untested. To address this, a sample of 249 adults completed a dot-probe task of selective attention with fearful and neutral faces. The results suggested that (1) fearful faces were prioritized over neutral faces (i.e., they captured and held attention), (2) greater sclera exposure at target locations facilitated reaction times, and (3) attention was held by greater sclera exposure of fearful faces at task irrelevant locations resulting in delayed disengagement. Collectively, the results indicate that fearful facial expressions and sclera exposure modulate spatial attention through independent and interactive mechanisms. Sclera exposure appears to be an important facilitator of nonverbal communication and perhaps represents an understudied variable in social cognition more broadly.

Reliability of an Interpretation Bias Task of Ambiguous Faces and Its Relationship with Social Anxiety, Depression, and Looming Maladaptive Style

Social anxiety (SA) and depression have been associated with negative interpretation biases of social stimuli. Studies often assess these biases with ambiguous faces, as people with SA and depression tend to interpret such faces negatively. However, the test–retest reliability of this type of task is unknown. Our objectives were to develop a new interpretation bias task with ambiguous faces and analyse its properties in terms of test–retest reliability and in relation to SA, depression, and looming maladaptive style (LMS). Eight hundred sixty-four participants completed a task in which they had to interpret morphed faces as negative or positive on a continuum between happy and angry facial expressions. In addition, they filled out scales on SA, depressive symptoms, and LMS. Eighty-four participants completed the task again after 1–2 months. The test–retest reliability was moderate (r = .57–.69). The data revealed a significant tendency to interpret faces as negative for people with higher SA and depressive symptoms and with higher LMS. Longer response times to interpret the happy faces were positively associated with a higher level of depressive symptoms. The reliability of the present task was moderate. The results highlight associations between the bias interpretation task and SA, depression, and LMS.

The questionable validity of attention bias variability: Evidence from two conceptually unrelated cognitive tasks

Background

Attention bias variability is thought to measure fluctuations in attention towards and away from threat-related information and is elevated in affective disorders. However, recent evidence suggests that attention bias variability may quantify general reaction time variability rather than attention bias behavior per se.

Methods

The current study calculated "attention bias variability" from two conceptually unrelated cognitive tasks: the dot-probe task (measuring attentional bias) and the arrow flanker task (measuring cognitive control).

Results

Attention bias variability measures were correlated across these unrelated tasks. Yet, when general reaction time variability was controlled, attention bias variability across tasks was no longer correlated. In addition, the reliability of attention bias variability measures decreased when controlling for general reaction time variability. Finally, although attention bias variability calculated from the dot-probe task initially correlated with anxious symptoms, this association was no longer significant when controlling for general reaction time variability.

Limitations

Our sample was comprised of high trait anxious individuals. Replication in clinical samples is warranted.

Conclusions

These findings collectively provide strong empirical evidence that attention bias variability is not a valid measure of attention-related behavior, but reflective of general reaction time variability more broadly.

Mindfulness-Based Cognitive Therapy: A Preliminary Examination of the (Event-Related) Potential for Modifying Threat-Related Attentional Bias in Anxiety

Objectives

Mindfulness-based cognitive therapy (MBCT) can reduce anxiety and depression symptoms in adults with anxiety disorders, and changes in threat-related attentional bias may be a key mechanism driving the intervention's effects on anxiety symptoms. Event-related potentials (ERPs) can illuminate the physiological mechanism through which MBCT targets threat bias and reduces symptoms of anxiety. This preliminary study examined whether P1 ERP threat-related attentional bias markers in anxious adults change from pre- to post-MBCT delivered in-person or virtually (via Zoom) and investigated the relationship between P1 threat-related attentional bias markers and treatment response.

Methods

Pre- and post-MBCT, participants with moderate to high levels of anxiety (N = 50) completed a dot-probe task with simultaneous EEG recording. Analyses focused on pre- and post-MBCT P1 amplitudes elicited by angry-neutral and happy-neutral face pair cues, probes, and reaction times in the dot-probe task and anxiety and depression symptoms.

Results

Pre- to post-MBCT, there was a significant reduction in P1-Probe amplitudes (d = .23), anxiety (d = .41) and depression (d = .80) symptoms, and reaction times (d = .10). Larger P1-Angry Cue amplitudes, indexing hypervigilance to angry faces, were associated with higher levels of anxiety both pre- and post-MBCT (d = .20). Post-MBCT, anxiety symptoms were lower in the in-person versus virtual group (d = .80).

Conclusions

MBCT may increase processing efficiency and decreases anxiety and depression symptoms in anxious adults. However, changes in threat bias specifically were generally not supported. Replication with a comparison group is needed to clarify whether changes were MBCT-specific.

Clinical Trials Registration

NCT03571386, June 18, 2018.

Supplementary Information

The online version contains supplementary material available at 10.1007/s12671-022-01910-x.

Measuring Food-Related Attentional Bias

Objective: Food-related attentional bias has been defined as the tendency to give preferential attention to food-related stimuli. Attentional bias is of interest as studies have found that increased attentional bias is associated with obesity; others, however, have not. A possible reason for mixed results may be that there is no agreed upon measure of attentional bias: studies differ in both measurement and scoring of attentional bias. Additionally, little is known about the stability of attentional bias over time. The present study aims to compare attentional bias measures generated from commonly used attentional bias tasks and scoring protocols, and to test re-test reliability.

Methods: As part of a larger study, 69 participants (67% female) completed two food-related visual probe tasks at baseline: lexical (words as stimuli), and pictorial (pictures as stimuli). Reaction time bias scores (attentional bias scores) for each task were calculated in three different ways: by subtracting the reaction times for the trials where probes replaced (1) neutral stimuli from the trials where the probes replaced all food stimuli, (2) neutral stimuli from the trials where probes replaced high caloric food stimuli, and (3) neutral stimuli from low caloric food stimuli. This resulted in three separate attentional bias scores for each task. These reaction time results were then correlated. The pictorial visual probe task was administered a second time 14-days later to assess test-retest reliability.

Results: Regardless of the scoring use, lexical attentional bias scores were minimal, suggesting minimal attentional bias. Pictorial task attentional bias scores were larger, suggesting greater attentional bias. The correlation between the various scores was relatively small (r = 0.13–0.20). Similarly, test-retest reliability for the pictorial task was poor regardless of how the test was scored (r = 0.20–0.41).

Conclusion: These results suggest that at least some of the variation in findings across attentional bias studies could be due to differences in the way that attentional bias is measured. Future research may benefit from either combining eye-tracking measurements in addition to reaction times.

The Effect of Empathy on the Attentional Processing of Painful and Emotional Stimuli

Background

Empathy is shown to affect the attentional processing of painful stimuli and emotional stimuli. However, whether the attentional effects on emotional stimuli depend on emotional valence and the nature of the relationship between the attentional effects on different stimuli are still unknown.

Methods

In the present study, 25 high-empathy (HE) participants and 25 low-empathy (LE) participants were recruited to perform dot-probe tasks on painful stimuli and emotional stimuli.

Results

The results showed that HE individuals had weak attentional disengagement to painful pictures. More importantly, regarding emotional pictures, HE individuals showed attentional avoidance to negative emotion pictures, while LE individuals showed attentional bias to positive emotion pictures. Correlation analysis showed that the attentional bias score and attentional disengagement score were only associated with each other within the same category of stimuli (painful, positive or negative stimuli).

Conclusion

These results revealed that HE individuals mainly showed attentional avoidance to negative stimuli, while LE individuals mainly showed attentional bias to positive stimuli.

Sex Differences in Anxiety: An Investigation of the Moderating Role of Sex in Performance Monitoring and Attentional Bias to Threat in High Trait Anxious Individuals

Anxiety disorders are more predominant in women than men, however there is a lack of understanding as to what neurocognitive mechanisms drive this sex difference. Recent investigation has found a potential moderating role of sex in the relationship between anxiety and the error related negativity (ERN)—a component of error-monitoring that is prevalent in high anxiety individuals—such that females display a positive relationship between anxiety/worry and ERN amplitude. We strove to further explore the influence of sex on the relationship between trait anxiety and performance monitoring, specifically with ERN, as well as extend this work to include another hallmark of anxiety, attentional bias to threat. To meet this end, participants performed the flanker and dot-probe tasks, respectively. We did not find a significant difference in the relationship between attention bias scores and anxiety for female vs. males participants. Furthermore, ΔERN amplitudes were greater in males compared to females, and males had more positive CRN amplitudes than females. There were no significant associations between ERN or ΔERN with anxiety in both male and female participants. However, there was a significant relationship between CRN amplitudes and trait anxiety in male but not female participants. Given these results, the effect of sex on the relationship between components of performance monitoring—namely the CRN and ERN—and anxiety may be more nuanced than the current understanding. Our study was limited to detecting medium to large sized moderation effects. Our findings may be important for future meta-analysis on sex differences in anxiety.

A Pictorial Dot Probe Task to Assess Food-Related Attentional Bias in Youth With and Without Obesity: Overview of Indices and Evaluation of Their Reliability

Several versions of the dot probe detection task are frequently used to assess maladaptive attentional processes associated with a broad range of psychopathology and health behavior, including eating behavior and weight. However, there are serious concerns about the reliability of the indices derived from the paradigm as measurement of attentional bias toward or away from salient stimuli. The present paper gives an overview of different attentional bias indices used in psychopathology research and scrutinizes three types of indices (the traditional attentional bias score, the dynamic trial-level base scores, and the probability index) calculated from a pictorial version of the dot probe task to assess food-related attentional biases in children and youngsters with and without obesity. Correlational analyses reveal that dynamic scores (but not the traditional and probability indices) are dependent on general response speed. Reliability estimates are low for the traditional and probability indices. The higher reliability for the dynamic indices is at least partially explained by general response speed. No significant group differences between youth with and without obesity are found, and correlations with weight are also non-significant. Taken together, results cast doubt on the applicability of this specific task for both experimental and individual differences research on food-related attentional biases in youth. However, researchers are encouraged to make and test adaptations to the procedure or computational algorithm in an effort to increase psychometric quality of the task and to report psychometric characteristics of their version of the task for their specific sample.

A systematic review of event-related potentials as outcome measures of attention bias modification

Attention bias modification (ABM) was initially developed with the goal of reducing attentional bias to threat-and subsequently anxious symptoms-in individuals with heightened anxiety. Although controversial, ABM appears to be generally effective in achieving this goal. Yet, the primary outcome measure of ABM (i.e., the reaction time-based differences score) has poor reliability and temporal resolution, which limits the inferences that can be drawn. In contrast, event-related potentials (ERPs) have superior reliability as well as temporal resolution and may therefore be better outcome measures of ABM. In this review, I systematically assess the research using ERPs as outcome measures in ABM protocols. I focus on the extent to which the ERPs modified by ABM represent earlier or later stages of information processing. In addition, I explore the extent to which ABM produces near and/or far transfer of learning effects on ERP measures. The reviewed literature suggests that ERPs are promising outcome measures of ABM. ABM modulates the effects of affective stimuli on posterior visually evoked ERPs (i.e., P1) as well as ERPs at anterior electrodes (i.e., P2, N2, and ERN). Based on the state of the field, several directions for future research are identified.

Visual cortical regions show sufficient test-retest reliability while salience regions are unreliable during emotional face processing

Functional magnetic resonance imaging studies frequently use emotional face processing tasks to probe neural circuitry related to psychiatric disorders and treatments with an emphasis on regions within the salience network (e.g., amygdala). Findings across previous test-retest reliability studies of emotional face processing have shown high variability, potentially due to differences in data analytic approaches. The present study comprehensively examined the test-retest reliability of an emotional faces task utilizing multiple approaches to region of interest (ROI) analysis and by examining voxel-wise reliability across the entire brain for both neural activation and functional connectivity. Analyses included 42 healthy adult participants who completed an fMRI scan concurrent with an emotional faces task on two separate days with an average of 25.52 days between scans. Intraclass correlation coefficients (ICCs) were calculated for the 'FACES-SHAPES' and 'FACES' (compared to implicit baseline) contrasts across the following: anatomical ROIs identified from a publicly available brain atlas (i.e., Brainnetome), functional ROIs consisting of 5-mm spheres centered on peak voxels from a publicly available meta-analytic database (i.e., Neurosynth), and whole-brain, voxel-wise analysis. Whole-brain, voxel-wise analyses of functional connectivity were also conducted using both anatomical and functional seed ROIs. While group-averaged neural activation maps were consistent across time, only one anatomical ROI and two functional ROIs showed good or excellent individual-level reliability for neural activation. The anatomical ROI was the right medioventral fusiform gyrus for the FACES contrast (ICC ​= ​0.60). The functional ROIs were the left and the right fusiform face area (FFA) for both FACES-SHAPES and FACES (Left FFA ICCs ​= ​0.69 & 0.79; Right FFA ICCs ​= ​0.68 & 0.66). Poor reliability (ICCs ​< ​0.4) was identified for almost all other anatomical and functional ROIs, with some exceptions showing fair reliability (ICCs ​= ​0.4-0.59). Whole-brain voxel-wise analysis of neural activation identified voxels with good (ICCs ​= ​0.6-0.74) to excellent reliability (ICCs ​> ​0.75) that were primarily located in visual cortex, with several clusters in bilateral dorsal lateral prefrontal cortex (DLPFC). Whole-brain voxel-wise analyses of functional connectivity for amygdala and fusiform gyrus identified very few voxels with good to excellent reliability using both anatomical and functional seed ROIs. Exceptions included clusters in right cerebellum and right DLPFC that showed reliable connectivity with left amygdala (ICCs ​> ​0.6). In conclusion, results indicate that visual cortical regions demonstrate good reliability at the individual level for neural activation, but reliability is generally poor for salience regions often focused on within psychiatric research (e.g., amygdala). Given these findings, future clinical neuroimaging studies using emotional faces tasks to examine individual differences might instead focus on visual regions and their role in psychiatric disorders.

The stability and reliability of attentional bias measures in the dot-probe task: Evidence from both traditional mean bias scores and trial-level bias scores

A new method of calculating attentional bias from the dot-probe task measures fluctuations in bias towards and away from emotional stimuli over time using trial level bias score metrics. We assessed the stability and reliability of traditional attentional bias scores and trial level bias score measures of attentional bias across time in two five-block dot-probe task experiments in non-clinical samples. In experiments 1 and 2, both traditional attentional bias scores and trial level bias score measures of attentional bias did not habituate/decrease across time. In general, trial level bias score metrics (i.e., attention bias variability as well as the mean biases toward and away from threat) were more reliable than the traditional attention bias measure. This pattern was observed across both experiments. The traditional bias score, however, did improve in reliability in the later blocks of the fearful face dot-probe task. Although trial level bias score measures did not habituate and were more reliable across blocks, these measures did not correlate with state or trait anxiety. On the other hand, trial level bias score measures were strongly correlated with general reaction time variability-and after controlling for this effect no longer superior in reliability in comparison to the traditional attention bias measure. We conclude that general response variability should be removed from trial level bias score measures to ensure that they truly reflect attention bias variability.

Climate change images produce an attentional bias associated with pro-environmental disposition

Humans have developed mechanisms to prioritize certain sensory input(s). Emotionally salient stimuli automatically capture observers' attention at the cost of less salient information. This prioritized processing is called attentional bias. Images of climate change have been found to elicit emotional responses. Yet, to date, there is no research assessing the extent to which climate change-relevant images produce an attentional bias. In a sample of college students (N = 39), we found that (1) climate change-related images capture attention and that (2) this attentional bias is related to individual differences in environmental disposition. Thus, images of climate change are salient-attention grabbing-signals related to pro-environmental orientation.

Temporal dynamics in attention bias: effects of sex differences, task timing parameters, and stimulus valence

New methods of calculating indices from the dot-probe task measure temporal dynamics in attention bias or fluctuations in attention bias towards and away from emotional stimuli over time. However, it is unclear how task-specific parameters such as stimulus valence and stimulus onset asynchrony (SOA) as well as participant sex affect temporal dynamics in attention bias. To address this, male and female participants (N = 106) completed either a fearful or happy face dot-probe task with four levels of SOA (84 ms, 168 ms, 336 ms, and 672 ms). Results suggest that temporal dynamics in attention bias are (1) greatest at the longest SOA, (2) greater in females, and (3) insensitive to stimulus valence. These findings indicate that participant sex and task timing, but not stimulus valence, are related to temporal dynamics in attention bias and should be considered in future studies utilizing this approach.